Low-False-Positive Findings: Structured Output and Adversarial Verification
In Part 2 we gave the auditor tools and let it roam a repository, reporting back what it found in prose. That works for a demo and falls apart in production. A raw agent emits findings as paragraphs ("There appears to be a hardcoded credential around line 40 of config.py, which could allow…"), and prose is the worst possible interface for a security finding. You cannot reliably parse it, you cannot deduplicate it, you cannot route it to a ticketing system, and you certainly cannot diff this run against the last one. Worse, a fluent model is a confident model: it will flag the TEST_ONLY fixture key we seeded in Part 1 as a live secret, and it will report verify=False that happens to sit inside a commented-out line, all in the same authoritative tone it uses for genuine issues.
This part fixes both problems, and they turn out to be the same problem viewed from two angles. First we make the output structured: every finding is forced through a strict JSON schema so downstream tooling has something machine-readable to stand on. Then we make each finding earn its place by running an independent, adversarial verification pass whose entire job is to refute the claim. Only findings that survive a skeptic make it into the report. Together these two moves are what turn an interesting toy into a scanner you would let near a pull request.
A quick note before the code: the Claude API surface evolves, so treat the exact parameters here as correct-as-of-writing and check the current docs when you build. We will call the Anthropic Python SDK directly for the structured pieces. The query/ClaudeAgentOptions machinery and custom tools from earlier parts still apply for the exploration phase; structuring and verification are better served by plain, schema-constrained message calls.
Why prose findings are a liability
Consider what a consumer of the auditor actually needs. A CI gate needs a boolean and a severity. A dashboard needs a stable identifier so it can tell "the same finding as yesterday" from "a new finding". A triage engineer needs a file, a line, and a one-line reason, without three sentences of hedging. None of that is reliably extractable from free text, and brittle regex over model output is exactly the kind of fragility we are trying to design away.
There is a subtler cost too. When the model writes prose, it is optimising for sounding helpful, and helpfulness rewards volume. It will surface anything that pattern-matches to a vulnerability, because in prose there is no penalty for a soft "this could potentially be an issue". Structure imposes discipline: when a finding must declare a category from a fixed enum, a numeric confidence, and a concrete evidence snippet, the vague near-misses have nowhere to hide.
A strict schema for a finding
Start by deciding what a finding is. The schema below is deliberately tight. Note the JSON-schema rules that the structured-output engine enforces: every object needs additionalProperties: false and an explicit required array listing every property, and the only types in play are objects, arrays, strings, integers, numbers, booleans, null, and enums. We use enums aggressively (for category and severity) because a constrained vocabulary is what makes findings comparable across runs.
FINDING_SCHEMA = {
"type": "object",
"additionalProperties": False,
"properties": {
"id": {
"type": "string",
"description": "Stable slug, e.g. 'hardcoded-secret-config-py-41'.",
},
"file": {"type": "string"},
"line": {"type": "integer"},
"category": {
"type": "string",
"enum": [
"hardcoded_secret",
"sql_injection",
"command_injection",
"insecure_tls",
"weak_crypto",
"path_traversal",
"deserialization",
"other",
],
},
"severity": {
"type": "string",
"enum": ["critical", "high", "medium", "low", "info"],
},
"description": {
"type": "string",
"description": "One sentence: what is wrong and why it matters.",
},
"evidence": {
"type": "string",
"description": "The exact offending code snippet, copied verbatim.",
},
"confidence": {
"type": "number",
"description": "0.0-1.0 self-assessed likelihood this is a real issue.",
},
},
"required": [
"id", "file", "line", "category",
"severity", "description", "evidence", "confidence",
],
}
# The auditor emits a list of findings, so wrap the object in an array.
FINDINGS_SCHEMA = {
"type": "object",
"additionalProperties": False,
"properties": {
"findings": {"type": "array", "items": FINDING_SCHEMA},
},
"required": ["findings"],
}The evidence field is doing more work than it looks. By forcing the model to copy the offending code verbatim, we give the verification pass something concrete to argue against, and we make it much harder for the model to hallucinate a finding that has no corresponding source. The confidence score is the auditor's own first guess; we will treat it as a hint, never as a gate. That strictness is deliberate. A downstream tool that receives data shaped like this can act on it without a single line of defensive parsing.
Forcing the auditor to fill it in
With the schema defined, the call that produces structured findings is almost anticlimactic. We pass the schema through output_config.format, and the API guarantees the response validates against it. The older top-level output_format parameter is deprecated, so use output_config.format. Below, repo_summary is whatever the exploration phase from Part 2 gathered: the relevant files and their contents, already narrowed down. We are running a dedicated structuring pass over that material rather than asking the roaming agent to format as it goes.
from anthropic import Anthropic
client = Anthropic()
STRUCTURING_PROMPT = """You are a security code auditor. From the code below,
report every plausible security finding. Report anything you are uncertain
about — a later verification step will filter false positives, so favour
recall here. Copy the exact offending line into 'evidence'."""
def collect_findings(repo_summary: str) -> list[dict]:
response = client.messages.create(
model="claude-opus-4-8",
max_tokens=16000,
output_config={
"format": {"type": "json_schema", "schema": FINDINGS_SCHEMA}
},
messages=[
{"role": "user",
"content": f"{STRUCTURING_PROMPT}\n\n<code>\n{repo_summary}\n</code>"},
],
)
import json
data = json.loads(response.content[0].text)
return data["findings"]If you prefer the SDK to validate and parse for you, client.messages.parse(...) takes the same output_config.format and returns the parsed object directly, sparing you the manual json.loads. Either way the model can no longer hand back a paragraph; it hands back a list of objects that conform, every time. Notice that the prompt deliberately tells the auditor to favour recall, in other words to over-report. That is intentional. Splitting the job into a generous finder and a strict verifier is precisely what lets each stage do one thing well.
A confident wrong answer and a confident right answer look identical in prose. The only way to tell them apart is to make a fresh, sceptical reader try to knock each one down, then keep only the ones still standing.
The adversarial verification pass
This is the core of the part. For each candidate finding, we spin up a separate call whose sole purpose is to refute it. We hand the verifier the claim and the surrounding code and ask a pointed question: is this a real, exploitable issue, or a false positive? We also instruct it to default to false-positive when uncertain. That single instruction is what tunes the whole system for precision: when the evidence is ambiguous, we drop the finding rather than risk crying wolf.
The verdict comes back through its own strict schema, so the filtering logic stays trivial. We also enable adaptive thinking on the verifier, because refutation is genuinely a reasoning task. It has to read the code in context, consider whether the flagged construct is reachable, and decide whether the surrounding lines neutralise it. Use thinking={"type": "adaptive"}; do not pass budget_tokens, which returns a 400 on this model.
VERDICT_SCHEMA = {
"type": "object",
"additionalProperties": False,
"properties": {
"is_real": {
"type": "boolean",
"description": "True only if this is a genuine, exploitable issue.",
},
"reasoning": {
"type": "string",
"description": "Why it is real, or why it is a false positive.",
},
},
"required": ["is_real", "reasoning"],
}
VERIFIER_PROMPT = """You are a hostile reviewer whose job is to REFUTE a
security finding. You are shown a claim and the code it refers to. Decide
whether this is a real, exploitable vulnerability or a false positive.
Treat these as false positives unless proven otherwise:
- test fixtures, sample/placeholder values, anything marked TEST_ONLY
- code inside comments or unreachable branches
- values that never leave the process or touch untrusted input
Default to is_real = false when the evidence is ambiguous. We would rather
miss a borderline issue than report one that wastes an engineer's time."""
def verify_finding(finding: dict, code_context: str) -> dict:
claim = (
f"Category: {finding['category']}\n"
f"File: {finding['file']}, line {finding['line']}\n"
f"Claim: {finding['description']}\n"
f"Evidence: {finding['evidence']}"
)
response = client.messages.create(
model="claude-opus-4-8",
max_tokens=4000,
thinking={"type": "adaptive"},
output_config={
"format": {"type": "json_schema", "schema": VERDICT_SCHEMA}
},
messages=[
{"role": "user",
"content": (
f"{VERIFIER_PROMPT}\n\n<finding>\n{claim}\n</finding>\n\n"
f"<code>\n{code_context}\n</code>"
)},
],
)
import json
# With adaptive thinking, the JSON answer is the last text block.
text = next(b.text for b in reversed(response.content) if b.type == "text")
return json.loads(text)
def filter_findings(findings: list[dict], context_for) -> list[dict]:
survivors = []
for f in findings:
verdict = verify_finding(f, context_for(f["file"]))
if verdict["is_real"]:
f["verification_reasoning"] = verdict["reasoning"]
survivors.append(f)
return survivorsHere context_for is a small helper that returns the relevant source for a finding's file, ideally a window around the flagged line plus enough surrounding code to judge reachability. The filter itself is four lines, because all the judgement lives in the verifier and all the structure lives in the schema. Run the seeded repository from Part 1 through this and the benign look-alikes fall away: the TEST_ONLY fixture key is refuted because it is plainly a placeholder, the verify=False in a comment is refuted because it is inert, and the genuinely seeded issues (the real hardcoded secret, the string-formatted SQL query) survive with reasoning attached.
Why a fresh verifier beats self-critique
It is tempting to skip the second call and instead ask the original auditor to "double-check your own findings". Resist it. A model reviewing its own output in the same context is anchored: it has already committed to the finding, it has the prose it wrote sitting in its context window, and it is far more likely to rationalise the claim than to demolish it. Self-critique within one conversation tends to produce agreement, not scrutiny.
A separate, narrowly-scoped verifier has none of that baggage. It arrives cold, sees only the claim and the code, and has been told its job is to refute. There is no sunk cost, no prior commitment, no surrounding narrative pulling it toward "yes". Independent judgement is the whole reason to do this, for the same reason code review works better when the reviewer is not the author. The narrow scope helps too: a verifier that evaluates exactly one finding against exactly one slice of code makes a sharper decision than one asked to re-grade a whole list at once.
And the "default to refuted when unsure" rule is what sets the operating point. Every detector trades recall against precision; by instructing the verifier to drop ambiguous cases, we deliberately choose precision. For a tool that interrupts engineers, that is the right trade. A scanner that fires only on findings it is confident about gets trusted and acted upon, whereas one that buries real issues under false alarms gets muted within a week.
Going further: independent verifiers and majority vote
If you want to push precision harder, run more than one verifier per finding and require consensus. Because each call is independent and stateless, this is embarrassingly parallel: fan out, collect verdicts, and keep the finding only if a majority (or, for the strictest gate, all) of them vote is_real.
from concurrent.futures import ThreadPoolExecutor
def verify_with_jury(finding, code_context, jurors=3, threshold=None):
threshold = threshold or (jurors // 2 + 1) # simple majority
with ThreadPoolExecutor(max_workers=jurors) as pool:
verdicts = list(pool.map(
lambda _: verify_finding(finding, code_context),
range(jurors),
))
real_votes = sum(1 for v in verdicts if v["is_real"])
keep = real_votes >= threshold
return keep, verdictsA three-juror majority smooths out the occasional verifier that misreads a snippet, and raising the threshold to unanimity gives you a very conservative gate for high-noise codebases. The cost is linear in juror count, so reserve the jury for findings where a false positive is expensive, or for the lower-confidence findings the auditor was least sure about, which are the ones the confidence field flagged.
Where this leaves us
We started with an agent that produced confident prose and turned it into a pipeline that produces machine-readable findings each of which has survived a hostile reading. The two changes reinforce each other: structure gives the verifier a precise, comparable claim to attack, and verification gives the structure its credibility. The benign look-alikes from Part 1 are rejected by design, and the real seeded issues come through with an audit trail explaining why they stand.
What we have not yet addressed is everything that happens after a finding survives. A real codebase produces dozens of confirmed findings at once, and not all of them deserve equal attention. A hardcoded secret in a deployment script is not the same emergency as a weak hash in a logging path. And many of the most serious issues are not visible in a single file at all: a tainted value enters in one module, flows through three more, and reaches a dangerous sink somewhere else entirely. The next part takes on both: ranking confirmed findings so the worst rise to the top, and reasoning across multiple files so the auditor can follow a vulnerability from source to sink.
NK