Running the Auditor in CI: Headless, Gated, Cost-Controlled

For the first four parts of this series, the security auditor has been an interactive companion: you ran it at a terminal, watched it reason, approved a tool call when it asked, and read its findings on the screen. That is exactly the right shape for development and exploration. It is the wrong shape for the place where a security gate actually earns its keep: the continuous-integration pipeline that runs on every pull request, where there is no human sitting at a keyboard to nudge it along.

This part is about that transition. We will take the agent we have been building and turn it into a CI step that runs to completion unattended, scopes itself to the code that actually changed, emits a machine-readable report, and exits with a status code that fails the build when it finds a confirmed high-severity issue. Three concerns dominate this conversion, and they are all about giving up control safely: headless execution (no human in the loop), pre-declared permissions (no human to approve tools), and cost and loop control (no human to notice the agent has wandered off and burned through your budget).

The throughline of this whole series has been trust: what you let the agent see, and what you let it do. CI is where that question becomes sharpest, because the code under audit on a pull request is, by definition, code you have not yet reviewed. I treat that as the load-bearing constraint throughout.

From interactive tool to CI step

An interactive session is a conversation. A CI step is a function: it takes an input (the diff, the repository), produces an output (findings), and returns a status code that the pipeline interprets as pass or fail. The mental shift is to stop thinking of the auditor as something you talk to and start thinking of it as something the pipeline calls.

That reframing dictates the contract. The build should fail when the auditor confirms a High or Critical finding, and pass otherwise, including the case where it finds only low-severity issues worth noting but not blocking. The gate is the exit code: 0 means clean or low-severity-only, non-zero means a confirmed blocker. Everything the auditor produces in between (its reasoning, its tool calls, its prose) is diagnostic detail. It is not the contract. We gate on the structured result, never on the free text. More on why that matters when we get to determinism.

Headless execution with the Agent SDK

The Claude Agent SDK gives us a one-shot, non-interactive entry point: query(...), an async iterator that drives the agent loop to completion and yields messages as it goes. For CI we do not want to render those messages to a human. We want to run the loop to the end, capture the final structured result, and translate it into an exit code. The final message in the stream is a ResultMessage, and its .result field carries the agent's concluding output.

Here is the headless entrypoint. It runs the audit, parses the structured findings out of the result, decides on a severity threshold, and exits with the corresponding status code:

#!/usr/bin/env python3
"""CI entrypoint for the security auditor. Exit 0 = clean / low only; non-zero = blocker."""
import asyncio
import json
import os
import sys
from claude_agent_sdk import query, ClaudeAgentOptions

BLOCKING = {"HIGH", "CRITICAL"}

# Exit codes the pipeline interprets:
EXIT_CLEAN = 0       # no findings, or low/medium only
EXIT_BLOCKED = 1     # at least one confirmed High/Critical
EXIT_ERROR = 2       # the audit itself failed to run


async def run_audit(prompt: str, options: ClaudeAgentOptions) -> str | None:
    """Drive the agent loop to completion and return the final result text."""
    final = None
    async for message in query(prompt=prompt, options=options):
        # Collect the concluding ResultMessage; ignore intermediate turns.
        if type(message).__name__ == "ResultMessage":
            final = message.result
    return final


def parse_findings(result: str) -> list[dict]:
    """The auditor is instructed to emit a JSON array as its final output."""
    start, end = result.find("["), result.rfind("]")
    if start == -1 or end == -1:
        raise ValueError("no JSON findings array in auditor output")
    return json.loads(result[start : end + 1])


def main() -> int:
    diff = sys.stdin.read()
    if not diff.strip():
        print("No diff on stdin; nothing to audit.", file=sys.stderr)
        return EXIT_CLEAN

    prompt = (
        "Audit the following pull-request diff for security vulnerabilities. "
        "Verify each candidate issue against the surrounding code before reporting it. "
        "Return ONLY a JSON array of findings; each object must have "
        "'severity' (LOW|MEDIUM|HIGH|CRITICAL), 'file', 'line', 'title', "
        "'rationale', and 'verified' (boolean).\n\n"
        f"--- DIFF ---\n{diff}"
    )

    options = build_options()  # defined in the next section

    try:
        result = asyncio.run(run_audit(prompt, options))
    except Exception as exc:  # network, auth, SDK error
        print(f"Auditor failed to run: {exc}", file=sys.stderr)
        return EXIT_ERROR

    if not result:
        print("Auditor produced no result.", file=sys.stderr)
        return EXIT_ERROR

    findings = parse_findings(result)
    with open("audit-report.json", "w", encoding="utf-8") as fh:
        json.dump(findings, fh, indent=2)

    blockers = [f for f in findings if f.get("verified") and f["severity"].upper() in BLOCKING]
    print(f"{len(findings)} finding(s); {len(blockers)} confirmed blocker(s).")
    for f in blockers:
        print(f"  [{f['severity']}] {f['file']}:{f['line']} — {f['title']}")

    return EXIT_BLOCKED if blockers else EXIT_CLEAN


if __name__ == "__main__":
    sys.exit(main())

Three things stand out. The loop collects the ResultMessage and ignores everything else, because in CI we have no use for the intermediate narration. The script distinguishes three outcomes, not two: a failed run (exit 2) is not the same as a clean audit (exit 0), and conflating them would let an authentication error masquerade as a passing security gate. And the gate keys on verified blockers only: a candidate the agent itself could not confirm does not fail the build, which keeps the false-positive rate from grinding the pipeline to a halt.

Permissions without a human in the loop

This is the part that most teams get wrong, and it is the part this series has been building towards. In an interactive session, the trust model has a human backstop: when the agent wants to run a tool, it can prompt, and you decide. In CI there is no one to prompt. If you carry over an interactive permission posture into an unattended run, you get one of two failure modes. Either the agent blocks forever waiting for an approval that never comes, or (far worse) you have configured it to bypass approvals entirely and it now runs arbitrary tools against untrusted pull-request code.

The pull request is the threat model. The auditor's job is to read code you have not yet reviewed; granting that same auditor unrestricted tool access turns a review step into a remote-code-execution surface. Pre-declare what it may do, and assume everything it reads is hostile.

The discipline is to pre-declare permissions. Instead of approving tools one at a time, you decide the entire allowlist up front and run in a non-interactive permission mode so the agent never pauses to ask. Concretely, that means three settings on ClaudeAgentOptions:

• An explicit allowed_tools allowlist. Tools not on this list are simply not available to the agent, so they are never run. For an auditor this is a deliberately narrow set: read files, search, perhaps run a read-only linter. It does not include the ability to write, push, or make network calls.
• A non-interactive permission_mode. The "default" mode prompts a human for approval, which never resolves in CI. Automation needs a mode that does not prompt. The SDK has evolved its mode names over releases, so check the version you are pinning for the exact string. The principle that does not change is allowlist plus no prompts. The mode you pick must rely on the allowlist for safety, not on a human.
• A disallowed_tools blocklist for dangerous patterns. Even within an allowed tool, you want to deny destructive shapes: anything that writes, deletes, pushes, or exfiltrates. Belt and braces: the allowlist says what is permitted, the blocklist explicitly forbids the patterns you most fear.

Here is the options builder, with the permission posture made explicit and the audit scoped read-only:

from claude_agent_sdk import ClaudeAgentOptions

AUDITOR_SYSTEM_PROMPT = (
    "You are a security code auditor running unattended in CI. "
    "You may read and search the repository to confirm findings, but you "
    "must not modify, create, or delete files, and must not make network "
    "calls. Report only issues you can verify against the code you can see."
)


def build_options() -> ClaudeAgentOptions:
    return ClaudeAgentOptions(
        system_prompt=AUDITOR_SYSTEM_PROMPT,
        model="claude-opus-4-8",
        # Narrow allowlist: read-only investigation only. Anything not
        # listed here is unavailable to the agent — it will not be run.
        allowed_tools=["Read", "Grep", "Glob"],
        # Defence in depth: explicitly forbid destructive / exfiltration
        # patterns even if a broader tool were ever added to the allowlist.
        disallowed_tools=[
            "Bash(rm*)",
            "Bash(git push*)",
            "Bash(curl*)",
            "Bash(wget*)",
            "Write",
            "Edit",
        ],
        # Non-interactive mode: never pause for human approval. The exact
        # mode name has shifted across SDK releases — check the version you
        # pin. The invariant is: allowlist + no prompts. Do NOT use a
        # bypass-everything mode on untrusted PR code.
        permission_mode="dontAsk",  # non-interactive: deny anything not on the allowlist
        # Bound the loop so a confused agent cannot iterate forever.
        max_turns=12,
        # Scope tool execution to the checked-out repository.
        cwd=os.environ.get("GITHUB_WORKSPACE", "."),
    )

If you take one line from that block, make it the comment: do not use a bypass-everything mode on untrusted PR code. The SDK offers permissive modes precisely so that you can choose them deliberately, and an auditor on a public-facing pull request is the textbook case for not choosing them. The allowlist is your safety boundary; the permission mode merely says you trust that boundary enough to stop asking a human about it.

Controlling cost and runaway loops

An interactive operator is also a cost governor. When a session starts thrashing (re-reading the same files, chasing a phantom across the whole repository), the human notices and stops it. Unattended, nothing stops it but the limits you set. Four levers keep an automated audit cheap and bounded.

Bound the loop with max_turns. This is the hard ceiling on iterations, and it is non-negotiable for unattended runs. A confused agent that keeps deciding it needs one more tool call will otherwise burn turns, and tokens, until something else stops it. We set it to a dozen above; tune it to the smallest number that lets a genuine audit complete, then leave a little headroom.

Scope the audit to the diff, not the whole repository. Re-auditing every file on every pull request is the single biggest waste of budget in a naive setup. The code that changed is the code that introduced risk; that is what we pipe to the agent on standard input and what the prompt directs it at. The full checkout is still on disk so the agent can read surrounding context with Read and Grep when it needs to confirm a finding, but the audit target is the changeset, which on a typical PR is a tiny fraction of the codebase.

Choose model and effort sensibly. Pin the model explicitly (model="claude-opus-4-8" above) so a CI runner's defaults cannot silently swap it. For most diffs a capable model at moderate effort confirms real issues without over-deliberating; reserve the heaviest settings for the audits where a missed Critical is genuinely costly. Pinning also makes runs reproducible: you know exactly what produced a given verdict.

Emit a machine-readable artifact plus a human summary. The script writes audit-report.json for downstream tooling (dashboards, trend analysis, a bot that comments on the PR) and prints a short human summary to the log so an engineer reading the failed build understands why it failed without opening the JSON. The structured artifact is what other systems consume; the summary is courtesy to the human who has to act on it.

The GitHub Actions workflow

With the entrypoint and its options in place, the pipeline glue is small. The workflow checks out enough history to compute the diff, installs dependencies, makes the API key available from secrets, computes the changed-files diff against the base branch, and pipes it into the auditor. The job's pass/fail is the script's exit code. Actions fails the step on any non-zero exit, which is exactly the gate we want.

name: Security Audit

on:
  pull_request:
    branches: [main]

jobs:
  audit:
    runs-on: ubuntu-latest
    permissions:
      contents: read          # the workflow itself needs no write access
      pull-requests: write     # only so a later step can post a comment
    steps:
      - name: Check out PR head with base history
        uses: actions/checkout@v4
        with:
          fetch-depth: 0       # need history to diff against the base branch

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install claude-agent-sdk

      - name: Run the security auditor over the PR diff
        env:
          # Auth in CI comes from a repository secret, never a checked-in key.
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          git diff --unified=3 \
            "origin/${{ github.base_ref }}...HEAD" \
            -- '*.py' '*.js' '*.ts' '*.go' \
            | python ci/audit.py

      - name: Upload the JSON report
        if: always()           # publish the artifact even when the gate fails
        uses: actions/upload-artifact@v4
        with:
          name: audit-report
          path: audit-report.json

Note the deliberate choices. The workflow's own contents permission is read, since the audit job has no business writing to the repository, and GitHub's token scoping enforces that independently of anything the agent does. The if: always() on the upload step ensures the report artifact is published even when the gate fails, because a failing build is precisely when you most want the findings. And the merge gate is implicit and robust: because the audit step exits non-zero on a confirmed blocker, the job fails, and a branch-protection rule requiring this job to pass is all it takes to block the merge. There is no separate "fail the build" step to forget; the exit code is the gate.

Flakiness, determinism, and the override path

Agents are non-deterministic. The same diff audited twice may produce findings phrased differently, ordered differently, or occasionally a borderline finding that surfaces in one run and not the next. If you treat that variability naively, your security gate becomes a flaky test that engineers learn to re-run until it passes, which is worse than no gate at all because it trains people to ignore it.

The defence is structural, and we have already built most of it. Gate on the structured, verified output and on severity thresholds, never on free text. The exit code is a function of the verified flag and the severity field of each finding, both of which are constrained, low-cardinality values. Whether the agent describes a SQL injection as "unsanitised input reaching a query" or "string-formatted SQL" does not change the verdict; what changes it is whether the finding is CRITICAL and verified. By collapsing the rich, variable prose down to a small set of categorical decisions, you make the gate as deterministic as the underlying judgement allows. Asking the agent to verify before reporting, and gating only on confirmed findings, strips out most of the borderline noise that would otherwise cause flakiness.

That still leaves false positives, because no auditor's judgement is perfect. The answer is not to weaken the gate but to give it a documented override path. A maintainer who has reviewed a flagged finding and determined it is a false positive should be able to suppress it through an allowlist file of acknowledged findings keyed by a stable fingerprint (file, rule, and a hash of the relevant lines), checked in alongside the code and consulted by the entrypoint before it decides on blockers. The override is auditable: it lives in version control, it names who suppressed what and why in the commit message, and it is reviewed like any other change. An override is a recorded human decision; a backdoor is a quiet bypass.

Where this leaves us

We have turned an interactive companion into an unattended gate without surrendering the trust model that makes it safe to point at code we have not yet reviewed. The headless entrypoint runs the agent loop to completion and translates a structured result into an exit code. The permission posture is pre-declared: a narrow allowlist, a blocklist for the patterns we fear, and a non-interactive mode that never bypasses the boundary on untrusted input. Cost stays bounded through a turn ceiling, diff-scoped auditing, and a pinned model. And the gate is deterministic where it counts, with a documented, version-controlled override for the inevitable false positive.

What we have not yet done is harden the auditor itself against the code it reads. A pull request can contain adversarial content as well as vulnerabilities: comments and strings crafted to manipulate the agent into ignoring a finding or misreporting severity. Running headless in CI raises the stakes on exactly that risk, because there is no human to catch the manipulation in the moment. The next and final part takes that on directly: hardening the auditor against prompt injection, scoping its context, and making its verdicts trustworthy even when the input is hostile.