Static analysers are good at the things you can express as a rule. They are weak at the things that need judgement: deciding whether a string that looks like an API key is actually a secret or a test fixture, whether verify=False is a deliberate localhost convenience or a production time-bomb, whether a pinned dependency is merely old or actively exploitable. Those are exactly the calls a competent reviewer makes by reading code in context, cross-referencing, and exercising a little scepticism. So in this series I build a reviewer: an autonomous agent that audits a repository for secrets, weak cryptography, TLS and certificate misconfiguration, and risky dependencies, then hands back a triaged report.
This is the first of six build-along articles. By the end of the series the agent will have custom scanning tools, structured machine-readable output, a verification pass to cull false positives, a prioritisation model, a CI integration, and a hardening review. This part lays the foundation: why an agent is the right shape for this problem, the architecture we will grow into, the deliberately-vulnerable target we will audit throughout, and a first minimal auditor you can run today using nothing but the built-in tools.
We will build everything on the Claude Agent SDK for Python. One caveat up front: SDK surfaces evolve, so treat the specific names here as a snapshot and check the current documentation before you ship. The architectural reasoning will last longer than any single function signature.
Why an agent, not a single API call
The instinct, when you reach for a large language model, is to stuff the whole repository into a prompt and ask for a list of vulnerabilities. For a five-file toy that works. For anything real it falls apart, and understanding why tells you what shape the solution needs to be.
A security audit is open-ended: you do not know in advance which files matter, how many there are, or what you will need to read next once you have found a lead. It is multi-file and stateful: a hardcoded key in config.py is only interesting if you can establish it is actually used, which means following it into db.py. And it benefits from iteration: a good reviewer forms a hypothesis ("this service disables TLS verification"), gathers evidence, and either confirms or discards it. That loop (gather context, reason, act, observe, repeat) is precisely what an agent is. A single completion call cannot decide to go and read another file; an agent can.
The test for "do I need an agent?" is simple: if you cannot fully specify the steps in advance because the next step depends on what the last one found, you need a loop that lets the model drive. Auditing a codebase is the textbook case.
Architecture overview
The agent loop is the heart of the design, and the SDK gives most of it to you. Conceptually the agent gathers context, reasons about what it has, calls a tool, observes the result, and decides whether to continue. The SDK runs that loop, ships a set of built-in tools (a filesystem Read, a Grep for content search, Glob for file discovery, and a Bash tool for everything else), and enforces a permission model so the agent cannot, say, execute shell commands unless you have allowed it to. What we add, starting in Part 2, are domain-specific scanning tools that encode security knowledge the model would otherwise have to reconstruct from scratch on every run.
┌──────────────────────────────────────────────┐
│ Claude Agent SDK │
│ │
│ ┌────────────────────────────────────┐ │
│ │ AGENT LOOP │ │
│ │ gather context → reason → act │ │
│ │ ▲ │ │ │
│ │ └──── observe ◀─────┘ │ │
│ └────────────────────────────────────┘ │
│ │ tool calls │
│ ┌──────────────┼───────────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────┐ ┌──────────┐ ┌────────────┐ │
│ │ Read │ │ Grep / │ │ Bash │ │
│ │ │ │ Glob │ │ (gated) │ │ built-in
│ └───────┘ └──────────┘ └────────────┘ │
│ ┌───────────────────────────────────────┐ │
│ │ secret-scan · dep-audit · tls-check │ │ we add (Part 2+)
│ └───────────────────────────────────────┘ │
│ │ │
│ permission layer │
└────────────────────┼─────────────────────────┘
▼
┌──────────────────────────┐
│ target repo: vault-svc │
└──────────────────────────┘The division of labour is the point. The SDK owns orchestration: running the loop, streaming the model's intermediate steps, mediating tool access. We own the security expertise: which tools exist, what the system prompt tells the model to look for, how findings are structured, and how we decide a finding is real. Keeping that boundary clean is what lets the later parts (verification, prioritisation, CI) slot in without rewriting the core.
The demo target: vault-svc
You cannot evaluate an auditor without something to audit, and you cannot measure it without knowing the right answers in advance. So throughout the series we will point the agent at vault-svc, a small, deliberately-vulnerable Python service. It is a controlled specimen rather than a real product, seeded with known issues across each category we care about, plus at least one benign look-alike so we can later prove the agent rejects false positives rather than flagging everything that glitters.
Here is the offending part of vault-svc/config.py. Read it the way the agent will have to:
# vault-svc/config.py
import os
import hashlib
import requests
# SEEDED ISSUE 1: hardcoded credentials committed to source
DB_PASSWORD = "pr0d-Vau1t-9f3a-LIVE" # <-- real-looking prod secret
STRIPE_API_KEY = "sk_live_51Hdon3verBxR2aQkZ" # <-- live API key in VCS
def password_hash(password: str) -> str:
# SEEDED ISSUE 2: MD5 used for password hashing (fast, broken, unsalted)
return hashlib.md5(password.encode()).hexdigest()
def fetch_entitlements(user_id: str) -> dict:
# SEEDED ISSUE 3: TLS certificate verification disabled
resp = requests.get(
f"https://billing.internal/api/v1/entitlements/{user_id}",
verify=False,
)
return resp.json()Three real problems sit in nine lines: secrets in version control, MD5 standing in for a password-hashing function, and verify=False silently turning a TLS connection into an unauthenticated one. Now the benign look-alike, in the test suite:
# vault-svc/tests/fixtures.py
# NOT a finding: clearly-labelled, non-functional test value.
TEST_ONLY_API_KEY = "sk_test_TEST_ONLY_do_not_use_000000" # nosec
SAMPLE_USER = {"id": "u_demo", "plan": "free"}A naive regex for sk_-prefixed strings flags both. A good reviewer reads the name (TEST_ONLY), the sk_test_ prefix, the directory (tests/), and the nosec annotation, and concludes it is a fixture. Distinguishing the two is the whole game, and it is why we want a model in the loop rather than a rule.
Finally, vault-svc/requirements.txt pins a dependency that is years past its last security release:
# vault-svc/requirements.txt
flask==2.0.1
requests==2.25.1
# SEEDED ISSUE 4: ancient, known-vulnerable release
PyYAML==5.3.1
cryptography==3.4.7The ground-truth set
Below is the canonical list of seeded findings, with category and the severity we expect a competent auditor to assign. This list does double duty: it is the answer key for this article, and it is the evaluation set we will score the agent against in Part 6. Anything the agent reports that is not on this list (e.g. the TEST_ONLY fixture) is a false positive; anything on this list it misses is a false negative.
- F1: Hardcoded database password (
config.py,DB_PASSWORD) · Secrets · expected severity: Critical - F2: Hardcoded live Stripe API key (
config.py,STRIPE_API_KEY) · Secrets · expected severity: Critical - F3: MD5 used for password hashing (
config.py,password_hash) · Weak crypto · expected severity: High - F4: TLS verification disabled (
config.py,verify=False) · TLS/cert misconfiguration · expected severity: High - F5: Outdated PyYAML 5.3.1 (
requirements.txt) · Risky dependency · expected severity: High - (decoy)
TEST_ONLY_API_KEY(tests/fixtures.py) · Not a finding · expected verdict: rejected as benign
A first, minimal auditor
With the target defined we can write the simplest thing that could possibly work: a one-shot run that gives the model a security-reviewer persona and access only to the read-only built-in tools, pointed at the vault-svc directory. No custom tools, no structured output, no verification. Just the agent loop and good instructions. This is the skeleton every later part builds on.
The Claude Agent SDK is installed with pip install claude-agent-sdk (Python 3.10+) and is fully asynchronous, so we wrap everything in asyncio.run. The query() helper runs a single task to completion, streaming messages as it goes; the final message of the stream is a ResultMessage carrying the agent's summary in .result.
import asyncio
from claude_agent_sdk import (
query,
ClaudeAgentOptions,
AssistantMessage,
TextBlock,
ToolUseBlock,
ResultMessage,
)
SECURITY_REVIEWER = """You are a senior application-security reviewer.
Audit the code in the working directory for:
- hardcoded secrets (API keys, passwords, tokens) committed to source
- weak or broken cryptography (e.g. MD5/SHA1 for passwords, ECB mode)
- TLS / certificate misconfiguration (disabled verification, ignored errors)
- risky or outdated dependencies
Use Read, Grep and Glob to investigate. Do NOT flag values that are clearly
test fixtures or examples (names like TEST_ONLY, sk_test_ prefixes, files under
tests/). For each real finding, report: file and line, category, a severity
(Critical / High / Medium / Low), and a one-line remediation.
Be precise. A false positive is a real cost to the team reading your report."""
async def main() -> None:
options = ClaudeAgentOptions(
system_prompt=SECURITY_REVIEWER,
allowed_tools=["Read", "Grep", "Glob"], # read-only; no Bash yet
model="claude-opus-4-8",
max_turns=24,
cwd="./vault-svc",
)
async for message in query(prompt="Audit this repository.", options=options):
if isinstance(message, AssistantMessage):
for block in message.content:
if isinstance(block, TextBlock):
print(block.text)
elif isinstance(block, ToolUseBlock):
print(f" [tool] {block.name} {block.input}")
elif isinstance(message, ResultMessage):
print("\n===== FINAL REPORT =====")
print(message.result)
if __name__ == "__main__":
asyncio.run(main())Three design choices in ClaudeAgentOptions deserve a mention. The allowed_tools list is deliberately restricted to Read, Grep and Glob. An auditor should observe, not act, so we withhold Bash entirely at this stage. The cwd scopes the agent to the target directory so its file operations stay inside vault-svc. And max_turns bounds the loop, so a confused run terminates rather than burning tokens indefinitely. We default to claude-opus-4-8 because the judgement calls (fixture versus real secret) reward the strongest model.
Iterating the message stream rather than waiting only for the final result matters more than it looks. Each AssistantMessage carries a list of content blocks: TextBlocks (the model's reasoning and prose) and ToolUseBlocks (each Grep or Read the agent decides to run, with its .name and .input). Printing the tool calls as they happen turns the audit into something you can watch and trust: you see the agent grep for verify=False, open config.py, then reason about what it found. That visibility is the difference between an auditor you believe and a black box that emits a verdict.
Want a conversation instead of one shot?
For later parts we will sometimes need to keep the session alive, to ask a follow-up or feed back the result of a verification step. The SDK's stateful client supports exactly that, using the same options object:
from claude_agent_sdk import ClaudeSDKClient
async def interactive_audit() -> None:
async with ClaudeSDKClient(options=options) as client:
await client.query("Audit this repository for secrets only.")
async for msg in client.receive_response():
... # same block handling as above
# the session retains context, so we can drill in
await client.query("For each secret, confirm it is actually referenced.")
async for msg in client.receive_response():
...For a one-shot audit, query() is the right tool; we reach for ClaudeSDKClient when multi-step verification arrives in Part 4.
What this naive version gets wrong, and the series arc
Run the code above and you will get something genuinely useful: it will find the hardcoded secrets, spot the MD5 hashing, catch verify=False, and (if you are lucky) note the stale dependency. That is a real result from perhaps forty lines of code. But it is a starting point, and its weaknesses define the rest of the series. Its output is free-form prose, not something a pipeline can act on. It re-derives security knowledge on every run instead of using purpose-built tools. It has no second opinion, so a confident hallucination ships as a finding. It cannot rank what matters most, cannot run in CI, and has had no thought given to running untrusted code safely.
Each of those gaps is a part of this series. Part 2 gives the agent eyes: custom MCP tools (a secret scanner, a dependency auditor, a TLS checker) built with the SDK's tool decorator and create_sdk_mcp_server, so the model calls a precise instrument rather than improvising with Grep. Part 3 turns prose into structured, schema-validated findings the rest of the pipeline can consume. Part 4 adds a verification pass that re-examines each candidate and culls the false positives, including, we hope, that TEST_ONLY fixture. Part 5 builds a prioritisation model so the report leads with what is genuinely dangerous. And Part 6 wires the auditor into CI, scores it against the ground-truth set you just met, and hardens the whole thing for running over code you do not trust.
For now, the foundation is laid: we know why an agent suits this problem, we have the loop and the SDK doing the orchestration, we have a target with a known answer key, and we have a minimal auditor that already earns its keep. In the next part we stop letting the agent improvise and start handing it proper tools.
NK