Routing Uncertainty in Code Review Agents

The purpose of this post is to break down my foray into building a simple agent - specifically a code reviewer agent. Using LLM-assisted coding assistants has become integral to my workflows, and I want to better understand what is happening under the hood.

What Is An Agent? And, What Is a Harness?

I would like to get a couple terms defined as they tend to be ambiguous and dependent on context.

First, an agent in the context for this blog post is a while loop with an LLM that:

• Receives a task.
• Determines what to do with it.
• Selects and executes tools.
• Observes the results.
• Self-repeats the loop until it finds its potential result and subsequent output satisfactory.

The harness is everything the agent has access to to complete tasks:

• Tool execution
• Runtime environment
• Context management
• State or memory management
• Self-correction mechanisms (feedback loops)

This Project

The focus of this code reviewer agent was how ambiguity is handled. The agent receives a diff but it’s not clear from the code snippets in the diff to determine whether the code is buggy or something worse (example: a leaked secret). I was curious about when and how the agent might prompt for more information from the author to make a judgement call on the code it receives.

The project is local and still pretty rough. A GitHub push event hits an Express.js server, the server pulls out the repository and commit SHA. The diff is shown in the terminal and then sent to a selected LLM via OpenRouter. The interactions with the code author occur in the CLI via Ink (which renders a React.js-based UI in the terminal).

The flow is:

1. GitHub sends a push event.
2. The local server gets the repo name and commit SHA.
3. The Ink app fetches and displays the diff.
4. The diff goes to OpenRouter.
5. The model returns either normal text or a submitReview call.
6. The local app parses the result and shows findings or follow-up questions.

The Express server and Ink UI run in the same Node process. That was mostly a convenience choice. Ink gave me a quick way to see the diff, the messages being sent to the model, and the review output without building a web UI.

Code review is a good place to test ambiguity because findings are not all the same. Some are clear bugs. Some are style issues. Some depend on what the project is trying to do.

For example, a React component using jQuery to mutate the DOM might be a real problem. It might also be required for a third-party widget. The diff alone may not say which one is true.

My original goal was simple:

• high-confidence findings can go straight into the review
• medium- and low-confidence findings should ask the author for context
• the answer should change the final result

That last point is where the design started to get interesting. If the model asks the user a question, the answer needs to do something: it should include a finding, remove it, lower confidence, raise confidence, or continue the review with new context.

The Review Loop

The review loop starts with a system prompt and a diff. The model can respond with normal assistant text, or it can call submitReview.

Each turn’s result replaces the running findings. If the result has questions, the app asks me in the terminal, adds my answer back into the message history, and calls the model again — so the next turn’s findings reflect my answer.

Two clarifications before the diagram. submitReview is just the single tool the model calls to return its result — the findings and any questions — not an action that posts a review to GitHub or opens a pull request. And the steps below are simplified: when the model raises a question, the loop pauses, collects the author’s answer, and the model re-emits its findings on the next turn. The diagram shows the shape of that loop — it can cycle back to the model several times — but it leaves out per-turn detail, like a single turn carrying more than one question, and the caps on how many times it will retry or loop.

flowchart TD
    A[Start review] --> B[Create messages from prompt and diff]
    B --> C[Call OpenRouter]
    C --> D{Model response}

    D --> E[Store assistant response]
    E --> F[Return review result]

    D -->|submitReview| G[Parse result]
    G -->|Invalid| H[Send error back to model]
    H --> C

    G -->|Valid| I[Record this turn's findings]
    I --> J{Any questions?}

    J -->|No| F
    J -->|Yes| K[Ask in terminal]
    K --> L[Add answer to messages]
    L --> C

This is an early example of the model encountering an issue and surfacing it in the review summary:

{
  "findings": [
    {
      "description": "This React component mutates the DOM with jQuery, which can conflict with React's rendering model.",
      "suggestion": "Prefer React-managed state unless this DOM manipulation is required by an external widget."
    }
  ],
  "humanReviewRequests": []
}

The model finished the review without asking for clarification (the humanReviewRequests array is empty). The phrase “unless this DOM manipulation is required by an external widget” is the giveaway - the program never asks whether it is true because it implicitly decided this is the case.

A better result would be:

{
  "findings": [],
  "humanReviewRequests": [
    {
      "finding": "This React component mutates the DOM with jQuery.",
      "reason": "The diff does not show whether the jQuery integration is required by a third-party widget."
    }
  ]
}

It surfaces the uncertainty instead of burying it in a suggestion. It still leaves the important part implied, though — it says why the model is unsure without saying what my answer is supposed to change. I come back to that gap once the tool shape can support it.

The Initial Tools Setup

The single-tool loop above is where the design landed. Getting there took a couple of iterations. My first version had two tools:

• reportFindings
• requestHumanReview

At the time this seemed reasonable. One tool reports issues. The other asks questions when the model needs more context.

In practice, this setup relied too heavily on the LLM’s judgement. It had to find the issue, decide how confident it was, report the finding, and then remember to make a separate call if the confidence was low enough.

Some findings were clear bugs. Others were style or maintainability problems. A few were cases where the right answer depended on project intent.

One example, in a React.js based Todo webapp, was inverting the logic for whether completed tasks should render with a strikethrough on the todo text:

className={`${completed ? "" : "text-decoration-line-through"}`}

Other examples depended more on project intent.

The model could usually find the issues and assign expected confidence levels. The problem was that it still wanted to finish the review instead of stopping to ask.

I made a few updates to the system prompt to coax the model into the expected behavior: examples were added, confidence levels clarified and stated directly that medium- and low-confidence findings should ask for human review. These efforts helped, but smoke testing revealed it was not solidly asking for a review when expected.

I also swapped out models from Minimax M2.7 to DeepSeek V3.2 to see whether the issue was model-specific. The same pattern showed up. The model was still inconsistent with asking for an author’s review of a low/med confidence finding.

At that point I stopped treating this as only a prompt and LLM selection problem. The tool design was probably part of the issue.

Changing the Tool Shape

Instead of making the model choose between two tools, it now returns a single object that holds both the findings and any questions it cannot resolve from the diff alone. At this point a question only carries a reason and the suspected finding — enough to surface doubt, but not yet to say what an answer should change:

export const HumanReviewRequestSchema = z.object({
  reason: z
    .string()
    .describe(
      "Why you cannot confirm this finding without human input. Be specific about what context is missing.",
    ),
  finding: z
    .string()
    .describe(
      "The potential issue you suspect but cannot verify from the diff alone.",
    ),
});

export const ReviewResultSchema = z.object({
  findings: z
    .array(FindingSchema)
    .describe("All code issues found in the diff."),
  humanReviewRequests: z
    .array(HumanReviewRequestSchema)
    .describe(
      "Requests for human input on suspected issues that cannot be confirmed from the diff alone.",
    ),
});

export const SUBMIT_REVIEW_TOOL = {
  type: "function",
  function: {
    name: "submitReview",
    description:
      "Submit your complete code review with all findings and any human review requests. Call this ONCE with ALL findings and humanReviewRequests.",
    parameters: zodToJsonSchema(ReviewResultSchema),
  },
} as const;

This change improved the LLM’s behavior. It no longer had to remember to call a second tool. It also means findings and questions are part of the same data structure, a ReviewResult, and the findings are updated based on the answers in humanReviewRequests.

Before:

flowchart TD
    A[Model finds issue] --> B[reportFindings]
    A --> C[Maybe requestHumanReview]
    C -. often skipped .-> D[No question asked]

After:

flowchart TD
    E[Model reviews diff] --> F[submitReview]
    F --> G[findings]
    F --> H[humanReviewRequests]
    H --> J{Any requests?}
    J -->|No| R[Return findings]
    J -->|Yes| K[Ask author in terminal]
    K --> L[Add answer back to messages]
    L --> E

This time the diagram makes the loop explicit: a single submitReview call returns both findings and any requests, and an outstanding request sends the author’s answer back for another turn. The new shape fixed one problem and made the next one more obvious. The schema can show that the model is unsure, but it cannot say what a human answer should do.

Right now the answer gets added back into the conversation, and the model decides what to do on the next turn. That works, though I would rather make the next step explicit.

Binding Answers to Actions

I am now using the following types to bind answers to actions:

type ReviewAction =
  | "include_finding"
  | "suppress_finding"
  | "lower_confidence"
  | "escalate"
  | "continue";

type HumanReviewRequest = {
  finding: string;
  reason: string;
  snippet?: string;
  question: string;
  answerEffects: {
    yes: ReviewAction;
    no: ReviewAction;
  };
};

The point is: do not ask a question unless the answer can change the review output.

A question should include:

• the possible finding
• why the model needs more context
• an optional code snippet for the reviewer
• the question to ask
• what happens for each answer

That keeps the program from having to guess whether “yes” means “yes, this is a bug” or “yes, the exception applies.”

For the jQuery finding, that shape looks like this:

{
  "finding": "This React component mutates the DOM with jQuery.",
  "reason": "The diff does not show whether the jQuery integration is required by a third-party widget.",
  "snippet": "useEffect(() => { $('#widget').html(content); }, [content]);",
  "question": "Is this jQuery DOM mutation required by a third-party widget?",
  "answerEffects": {
    "yes": "suppress_finding",
    "no": "include_finding"
  }
}

If the answer is yes, the finding is suppressed; if it is no, it goes into the review summary. If the author is not sure, the answer can map to lower_confidence instead — keep the finding, but flag it as uncertain rather than forcing the call.

One caveat: right now answerEffects is a contract the model is asked to honor, not something the harness enforces. The author’s answer goes back into the conversation and the model applies the effect on its final submitReview call. The schema makes the intended outcome explicit; deterministically applying it in code is the step I still want to make.

Demo

This is a demo of the code review agent inspecting an addition of a jQuery snippet in a React Component. It demonstrates the agent not having enough context to make a judgement of the diff, requesting said context from the author, and rendering a summary of its findings based on the feedback from the author.

Next, I want to test it with single-turn and multi-turn evals — the subject of a follow-up post — and try a path where the agent opens a pull request once a finding is concrete enough to act on.