Breaking Agents to Build Better Ones

01 / Field Brief

Breaking Agents to Build Better Ones

A hands-on lab for the OWASP Top 10 for LLM Applications.

Shawn Campbell / June 23, 2026

02 / Claim

AI security gets easier when the risks are executable.

The OWASP Top 10 is the threat map. The lab is how we turn that map into engineering practice.

03 / Attack Surface

Agents are not just chat boxes.

• Instructions
• Retrieved text
• Tools
• Memory
• Automation
• Downstream output

04 / Untrusted Terrain

Every string the agent reads can become an attack path.

• Support notes
• Logs
• Dependency READMEs
• Jira comments
• Retrieved policy docs
• MCP tool output
• Saved memory

05 / Threat Map

OWASP Top 10 as engineering questions

06 / Threat Map

The rest of the map

07 / Lab Rationale

A lab turns vague risk into a measured result.

• Reproduce the failure
• Define attack success
• Measure a baseline
• Add one defense
• Rerun the same harness
• Compare the delta

08 / Method

The lab loop

target -> payloads -> attack harness -> result
   ^                                      |
   |                                      v
defense toggle <- compare delta <- rerun harness

09 / Components

What a useful AI security lab needs

• Vulnerable local target
• Synthetic fixtures
• Payload library
• Attack runner
• Result file
• Defense toggle
• Writeup
• Safety boundary

10 / Demo

Concrete demo: LLM01 prompt injection

Target: vulnerable RAG assistant.

Attack: indirect prompt injection through a retrieved document.

Goal: make the assistant follow hostile document instructions instead of the intended task boundary.

11 / Attack Path

The user does not have to type the attack.

user question
  -> retriever
  -> trusted docs + attacker-controlled support note
  -> model context
  -> answer or unsafe action

12 / Metric

Define success before showing results.

An attack succeeds when the assistant follows the injected instruction instead of the intended task boundary.

attack success rate = successful attacks / total attempts

13 / Baseline

Defense off gives every future claim something to beat.

• Same payload set
• Every response recorded
• Attack objective scored
• Structured result saved

14 / Defense

Spotlighting marks the trust boundary.

• Preserve retrieved content
• Label untrusted text
• Tell the model how to treat quoted material
• Measure whether behavior changes

15 / Comparison

Defense on changes one variable.

• Same target
• Same payloads
• Same scoring
• Same result format
• Only the defense changes

16 / Evidence

What this proves and does not prove

17 / Scale Out

The same pattern extends across the Top 10.

• Supply-chain prompt injection
• Data and model poisoning
• Excessive agency
• Improper output handling
• Sensitive information disclosure
• System prompt leakage
• Vector retrieval weaknesses
• Misinformation
• Unbounded consumption

18 / Next Labs

Two high-value next labs

19 / Checklist

Design-review questions

• What untrusted text enters context?
• What tools can the agent call?
• What secrets or sensitive data can it see?
• What consumes model output next?
• What is logged, budgeted, and cancellable?
• Can we reproduce the attack locally?
• Can we measure the defense against a baseline?

20 / Demo Window

Live demo or recorded walkthrough

• Show the poisoned fixture
• Run the in-process comparison harness
• Display defense-off and defense-on summary fields
• Compare the attack-success-rate delta

21 / Roadmap

The lab grows one measured slice at a time.

• First slice for each OWASP category
• Richer payload variants
• Stronger eval metadata
• Clearer writeups
• Public lesson workflow after the lab, blog, and presentation are complete

22 / Close

Build small broken agents. Attack them honestly. Measure what changed.

Then carry the evidence back into real engineering decisions.

23 / Q&A

Questions, objections, and lab ideas.

• Which agent input path worries you most?
• Which tool permission would you remove first?
• What would make this lab useful in your review process?

24 / Appendix A

Safety boundary

• Local targets only
• Synthetic data only
• No real credentials
• No customer data
• No third-party probing
• Vulnerable behavior stays inside owned lab services

25 / Appendix B

Demo commands

Measured comparison from the repo root. The runbook expands this into a timed path.

.venv/bin/python lab/attacker/custom/run_v0_rag_attacks.py --mode compare > /tmp/llm01-compare.json
.venv/bin/python -c 'import json; r=json.load(open("lab/evals/results/v0-rag-latest.json")); print(r["defense_off"]["attack_success_rate"], r["defense_on"]["attack_success_rate"], r["delta"]["absolute_reduction"])'

# Optional HTTP smoke only
.venv/bin/python lab/attacker/custom/run_v0_rag_attacks.py --target http --mode off > /tmp/llm01-http.json

26 / Appendix C

Full module map

27 / Appendix D

Metric notes

• Attack success rate is successful attacks divided by total attempts
• Compare defense-off and defense-on runs with the same payload set
• Treat the delta as a lab signal, not a universal security claim
• Keep result JSON with runner metadata, target type, payload path, and cases

28 / Appendix E

References