By the end of Part 3 our auditor produced something genuinely useful: a list of verified findings. Each one had survived a verification pass that re-read the surrounding code, discarded the hallucinated and the already-mitigated, and kept only the issues the agent could actually point at. That is a real improvement over a raw linter dump. But hand that list to a senior engineer on a Friday afternoon and watch what happens. They scroll, they sigh, and they ask the only question that matters: "What do I fix first?"
A flat list does not answer that question. Forty verified findings are not forty units of work; some are exploitable today on a network-facing path, some are theoretical, and a good number are the same root cause wearing three different hats. Triage is the difference between a report that gets actioned and one that gets bookmarked. In this part we build the triage layer: a transparent severity model, deduplication by root cause, and (the part a linter can never do) multi-file reasoning that connects a fact in one file to its consequence in another. We keep using the Claude Agent SDK (query, ClaudeAgentOptions, custom @tool definitions exposed through create_sdk_mcp_server) for the agentic exploration, and the Anthropic Python SDK with structured outputs for the deterministic ranking step.
The guiding principle: do not trust a single model-assigned severity label. Asking the model "is this critical or low?" in one breath produces inconsistent, uncalibrated answers that drift between runs. Instead we decompose severity into parts the model can reason about independently, then combine those parts with code we control.
Severity as a function, not a vibe
The instinct is to add "severity": "high" to the schema and let the model fill it in. Resist it. Severity is a judgement that compounds two largely independent questions, and conflating them is exactly why model-assigned labels are noisy:
- Impact: if this is exploited, what does the attacker gain? Remote code execution and a leaked signing key are not the same magnitude as a verbose error message.
- Exposure: can it actually be reached? A SQL injection on an internal admin tool behind mutual TLS is a different proposition from the same flaw on a public, unauthenticated endpoint.
So we ask the model to score those two axes, with a tight rubric for each, and we compute the final severity ourselves. The model does the part it is good at (reading code and judging consequence); our code does the part it is good at (applying the same rule every time). Here is the rubric we hand the model via structured output, and the deterministic function that maps the two scores onto Critical/High/Medium/Low.
from anthropic import Anthropic
client = Anthropic()
# Each finding from Part 3 is a dict: rule, file, line, snippet, description.
SCORING_SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["impact", "exposure", "impact_reason", "exposure_reason"],
"properties": {
"impact": {
"type": "integer",
"enum": [1, 2, 3, 4],
"description": (
"What an attacker gains. "
"1=informational; 2=limited (single user/record); "
"3=serious (data theft, privilege escalation); "
"4=critical (RCE, full credential/key compromise)."
),
},
"exposure": {
"type": "integer",
"enum": [1, 2, 3, 4],
"description": (
"Reachability. 1=test/dev-only or dead code; "
"2=internal, authenticated path; "
"3=production path behind some control; "
"4=network-facing and unauthenticated."
),
},
"impact_reason": {"type": "string"},
"exposure_reason": {"type": "string"},
},
}
def score_finding(finding: dict) -> dict:
"""Ask the model to score impact and exposure against a fixed rubric."""
prompt = (
"Score this verified security finding on two axes using the schema. "
"Judge exposure from the code path, not from the rule name.\n\n"
f"Rule: {finding['rule']}\n"
f"File: {finding['file']}:{finding['line']}\n"
f"Description: {finding['description']}\n\n"
f"Code:\n{finding['snippet']}"
)
resp = client.messages.create(
model="claude-opus-4-8",
max_tokens=1024,
output_config={"format": {"type": "json_schema", "schema": SCORING_SCHEMA}},
messages=[{"role": "user", "content": prompt}],
)
# With output_config.format the result arrives as JSON in the text block.
import json
scores = json.loads(resp.content[0].text)
return {**finding, **scores, "severity": derive_severity(scores)}
def derive_severity(scores: dict) -> str:
"""Combine impact and exposure deterministically. Same input, same label."""
impact, exposure = scores["impact"], scores["exposure"]
product = impact * exposure # 1..16
if impact == 4 and exposure >= 3: # critical impact on a reachable path
return "Critical"
if product >= 9:
return "High"
if product >= 4:
return "Medium"
return "Low"The payoff is auditability. When a finding lands as "High", the report can show why: impact 3, exposure 3, with the model's one-line reasons attached. Two engineers can disagree with the rubric, but they will disagree with it consistently, and they can tune the thresholds in derive_severity without re-running the model. That is what "transparent" buys you. The ranking behaves like a spreadsheet formula.
A severity label you cannot explain is a severity label nobody will trust. Decompose the judgement, let the model score the parts, and keep the arithmetic in code you can read.
Deduplication: same root cause, many faces
Run a scan over a real repository and the same underlying mistake will surface repeatedly. A single hard-coded API key defined in config.py gets flagged once at its definition, again in every module that imports it, and a third time by a secrets-pattern rule that matches the literal. Three findings, one fix. Present them as three and you have inflated the report, scattered the engineer's attention, and buried the signal.
The cheap, deterministic dedup catches the obvious clusters: normalise the secret value (or the rule-plus-file pair) and group on it. This is fast and never wrong in the way a model can be wrong, so it runs first.
import hashlib
from collections import defaultdict
def normalise_secret(snippet: str) -> str | None:
"""Extract and hash a literal secret so the same value collapses to one key."""
import re
m = re.search(r"""['"]([A-Za-z0-9_\-]{16,})['"]""", snippet)
if not m:
return None
return hashlib.sha256(m.group(1).encode()).hexdigest()[:16]
def cheap_dedup(findings: list[dict]) -> list[list[dict]]:
"""Group findings that are obviously the same issue. Returns clusters."""
groups: dict[str, list[dict]] = defaultdict(list)
for f in findings:
secret = normalise_secret(f["snippet"])
# Prefer grouping by leaked value; fall back to rule + file.
key = secret if secret else f"{f['rule']}::{f['file']}"
groups[key].append(f)
return list(groups.values())The cheap pass cannot see semantic duplicates: two findings describing the same logical weakness in different words, or the same misconfiguration reached through two call sites. For those we ask the model to cluster, again through structured output so the result is a clean list of group indices rather than prose we have to parse. One caution matters more than the code: be conservative. Over-eager deduplication is worse than none, because it silently drops a genuinely distinct issue into another's shadow. We instruct the model to merge only when findings share a single root cause and a single fix, and we keep the cheap grouping as a floor the model is not allowed to override.
CLUSTER_SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["clusters"],
"properties": {
"clusters": {
"type": "array",
"description": "Each inner array lists finding ids that share ONE root cause and ONE fix.",
"items": {"type": "array", "items": {"type": "integer"}},
}
},
}
def semantic_dedup(findings: list[dict]) -> list[list[int]]:
listing = "\n".join(
f"[{i}] {f['rule']} @ {f['file']}:{f['line']} — {f['description']}"
for i, f in enumerate(findings)
)
resp = client.messages.create(
model="claude-opus-4-8",
max_tokens=2048,
output_config={"format": {"type": "json_schema", "schema": CLUSTER_SCHEMA}},
messages=[{
"role": "user",
"content": (
"Cluster these findings ONLY where they share a single root cause "
"and a single remediation. When in doubt, keep them separate — "
"do not merge distinct issues.\n\n" + listing
),
}],
)
import json
return json.loads(resp.content[0].text)["clusters"]Multi-file reasoning: the reason you bought an agent
Everything so far a sufficiently clever script could approximate. This is the part that justifies an agent over a linter. A linter sees one file at a time; it can tell you a file contains a hard-coded secret, and separately that another file disables TLS verification, but it cannot tell you that those two facts are the same incident. Connecting facts across files, following a value from where it is defined to where it is used, is reasoning, and reasoning is what the Agent SDK gives us.
Take the running example from our test target, vault-svc. Three findings come back from Part 3, and on their own each looks middling:
- A hard-coded secret
SIGNING_KEY = "sk_live_8f3c..."inconfig.py. A leaked credential, but maybe it is unused scaffolding? - An import-and-use of that key to sign outbound requests in
client.py. Interesting, but is the value real? - A
verify=Falseon the veryrequestscall that carries the signed payload inclient.py: disabled TLS verification, common enough to be ignored.
Reasoned together, the three medium findings become one high-severity chain: a real signing key, used to authenticate a request, sent over a connection an attacker can transparently intercept. The compromise of the key and the interceptability of the channel compound. To let the agent see this, we give it read-only navigation tools and a prompt that explicitly asks it to follow usage across files and report the chain, with a structured related_findings link so the connection survives into the report.
from claude_agent_sdk import query, ClaudeAgentOptions, tool, create_sdk_mcp_server
@tool("grep_repo", "Search the repository for a regex pattern.", {"pattern": str})
async def grep_repo(args):
import subprocess
out = subprocess.run(
["rg", "-n", "--no-heading", args["pattern"], REPO_ROOT],
capture_output=True, text=True, timeout=20,
).stdout[:6000]
return {"content": [{"type": "text", "text": out or "no matches"}]}
@tool("read_file", "Read a file (optionally a line range) from the repo.", {"path": str})
async def read_file(args):
with open(f"{REPO_ROOT}/{args['path']}", encoding="utf-8") as fh:
return {"content": [{"type": "text", "text": fh.read()[:8000]}]}
nav_server = create_sdk_mcp_server(
name="repo-nav", version="1.0.0", tools=[grep_repo, read_file]
)
CHAIN_PROMPT = """You are a security auditor connecting related findings into chains.
You are given a set of verified findings. For each finding that involves a secret,
a credential, or a transport setting, use grep_repo and read_file to trace how the
value flows ACROSS files: where it is defined, imported, and used.
Report any chain where two or more findings COMPOUND into a higher severity than
any one alone — e.g. a hard-coded key that is then used to sign a request sent over
an unverified TLS connection. Explain the chain, name every file:line involved, and
give the combined severity and a single remediation.
Findings:
{findings}
"""
async def find_chains(findings: list[dict]) -> str:
listing = "\n".join(
f"- {f['rule']} @ {f['file']}:{f['line']}: {f['description']}"
for f in findings
)
options = ClaudeAgentOptions(
model="claude-opus-4-8",
mcp_servers={"repo-nav": nav_server},
allowed_tools=["mcp__repo-nav__grep_repo", "mcp__repo-nav__read_file"],
system_prompt="Trace values across files. Be specific about file:line.",
max_turns=12,
)
chain_report = []
async for message in query(
prompt=CHAIN_PROMPT.format(findings=listing), options=options
):
for block in getattr(message, "content", []):
if getattr(block, "type", None) == "text":
chain_report.append(block.text)
return "\n".join(chain_report)Given the three vault-svc findings, the agent greps for SIGNING_KEY, finds it defined in config.py and imported in client.py, reads the client.py request construction, sees the signed payload travel through a call with verify=False, and reports the chain. We capture that as a structured link so the report can render the three findings as one promoted item rather than three orphans.
def link_chain(findings: list[dict], primary_idx: int, member_idxs: list[int],
combined_severity: str, explanation: str) -> dict:
"""Attach a related_findings link and promote severity on the primary finding."""
primary = dict(findings[primary_idx])
primary["severity"] = combined_severity # promoted above any member's own
primary["chain_explanation"] = explanation
primary["related_findings"] = [
{"file": findings[i]["file"], "line": findings[i]["line"],
"rule": findings[i]["rule"]}
for i in member_idxs if i != primary_idx
]
return primaryAssembling the triaged report
With scoring, deduplication, and chaining in place, the final assembly is plumbing, which is how it should be. Each stage did one job and handed a clean structure to the next, so the report builder just orders by severity, groups by root cause, and prints the evidence and remediation hint per item. We sort with a fixed severity rank so the output is stable across runs, and we render related findings as nested context rather than separate entries.
SEVERITY_RANK = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3}
def assemble_report(findings: list[dict]) -> str:
# findings here are post-scoring, post-dedup, post-chaining: one entry per root cause.
ordered = sorted(
findings,
key=lambda f: (SEVERITY_RANK[f["severity"]], f["file"], f["line"]),
)
lines = ["# Security Audit — Triaged Findings", ""]
for f in ordered:
lines.append(f"## [{f['severity']}] {f['rule']}")
lines.append(f"- **Location:** `{f['file']}:{f['line']}`")
lines.append(f"- **Evidence:** {f['description']}")
if f.get("chain_explanation"):
lines.append(f"- **Chain:** {f['chain_explanation']}")
for rel in f.get("related_findings", []):
lines.append(
f" - linked: `{rel['file']}:{rel['line']}` ({rel['rule']})"
)
lines.append(f"- **Fix:** {f.get('remediation', 'See description.')}")
lines.append("")
return "\n".join(lines)For the vault-svc example the top of the report now reads as a single Critical entry, "Hard-coded signing key used over an unverified TLS connection", with config.py as the primary location, client.py linked beneath it as the usage and transport sites, one sentence of chain explanation, and one remediation that fixes all three: move the key into a secrets manager and restore certificate verification. An engineer reading that knows immediately what to do first, and the three lower-severity findings that fed into it are visible as supporting context instead of competing for attention.
What we have built, and where it goes next
This pattern recurs throughout agentic engineering: let the model judge, let your code decide. The model scored impact and exposure, proposed clusters, and traced values across files. All of those need reading and reasoning. Our code derived severity from a rubric, enforced a conservative dedup floor, and assembled the report deterministically, all of which needs to be consistent and explainable. Mixing those responsibilities is where most "the AI graded it itself" pipelines go wrong; separating them is what makes the output something a senior engineer will actually trust on a Friday afternoon.
A triaged report is the artefact, but an artefact sitting in someone's downloads folder changes nothing. In Part 5 we take the auditor out of the interactive session and run it headless in CI, wiring query into a non-interactive pipeline, gating a pull request on the highest severity it returns, and making sure a Critical chain like the one above blocks the merge instead of merely describing the breach after it ships.
NK