Running Evals on the Code Review Agent

This post follows up on my previous post: I’ve prototyped a code review agent, and I’m experimenting with ways to test that it works as expected. This is where evaluations, or evals, come into play.

The previous post explained the reasoning behind the following types:

type ReviewAction =
  | "include_finding"
  | "suppress_finding"
  | "lower_confidence"
  | "escalate"
  | "continue";

type HumanReviewRequest = {
  reason: string;
  finding: string;
  snippet?: string;
  question: string;
  answerEffects: { yes: ReviewAction; no: ReviewAction };
};

When the agent’s confidence in a finding is low or medium, it should request human review. Testing how those confidence levels work, and how the downstream tool calls behave, is the heart of the eval work I have been investigating.

Eval Organization

Because LLMs are probabilistic, traditional deterministic software tests are not a good fit for evaluating agent behavior. Evals are functionally the equivalent of unit tests that, given a set of inputs, will measure the nondeterministic response from an AI agent. They compare metrics between runs to track improvements or regressions. They provide a way to convert qualitative outputs (such as asking for more context on a piece of code) to quantitative scores.

There are multiple ways to influence the agent and observe the effects through an eval. Examples include but are not limited to:

• System prompt: Does it properly detail the responsibility of the code review agent and guide it accordingly? Confidence levels and a bug taxonomy are outlined here.
• Tool descriptions: submitReview is the only tool available to the agent. Is it properly described so the agent knows when and how to use it?
• Chat history: This includes the request to review the code diff, the agent’s tool usage, the author’s responses to ambiguous-code follow-ups, and the code summary.
• LLM: The model used and its variables, such as reasoning level, temperature, top-p, etc.

For the purposes of the code review agent, I split the evals into three discrete groups. Each group is scored by different metrics.

The bug-detection group is the heart of the suite. Each case plants a known bug in a diff and scores the agent against the three columns in that row: catchBug asks whether it surfaced the expected type of finding at all, lineAccuracy asks whether that finding landed on the right line (within ±2), and confidence asks whether it was reported at the expected confidence level. Together they answer a single question: did the agent find the planted bug, in the right place, and was it appropriately sure of itself?

One wrinkle in these three checks: logic_error and edge_case are scored as interchangeable, since a hardcoded crash site could defensibly fall into either category. security and style stay distinct, so a real misclassification there is still caught.

Rather than walk through all seven evaluators, I’ll start with the simplest one: humanReviewRequested, from the human-review row of the table (its code symbol is humanReviewRequestedEvaluator). I’m leading with it because it’s the binary base case — did the agent ask for a human review when it should have, yes or no? The others, like answerEffectsHonored and the line- and confidence-accuracy checks, are more nuanced, so this is the cleanest one to read first:

export const humanReviewRequestedEvaluator = (
  output: EvalOutput,
  target?: EvalTarget,
) => {
  // Only applies to cases that expect a human-review request.
  if (!target || target.isClean || !target.expectsHumanReview) {
    return null as unknown as number;
  }
  return output.humanReview.length > 0 ? 1 : 0;
};

Returning 1 means the output included at least one human-review request. Returning null tells Laminar to skip this evaluator, because it was called in a test run for another group and should not penalize the agent.

Faking the Human in the Loop

A free-form question and a literal yes/no keystroke have no inherent binding, so the eval needs some contract that fixes the meaning of the response. Suppose the agent is unsure about a hardcoded value that might be a leaked secret. Example:

1
// Two phrasings of the same review request for a hardcoded `pk_live_…`
2
// value. The `yes`/`no` mapping flips because the meaning of "yes"
3
// is reversed between the two questions.
4
const asSecretQuestion: HumanReviewRequest = {
5
  reason: "Hardcoded credential pattern (pk_live_…) replacing env var",
6
  finding: "Likely leaked production credential",
7
  snippet: "analytics.init('pk_live_8f3a2b1c9d');",
8
  question: "Is this a real secret?",
9
  answerEffects: { yes: "include_finding", no: "suppress_finding" },
10
};
11

12
const asPublishableQuestion: HumanReviewRequest = {
13
  reason: "Hardcoded credential pattern (pk_live_…) replacing env var",
14
  finding: "Likely leaked production credential",
15
  snippet: "analytics.init('pk_live_8f3a2b1c9d');",
16
  question: "Is this a safe, publishable key?",
17
  answerEffects: { yes: "suppress_finding", no: "include_finding" },
18
};

The agent could ask ‘Is this a real secret?’ (line 8), where a yes should include the finding (line 9) — or ‘Is this a safe, publishable key?’ (line 16), where that same yes should suppress it (line 17). On its own, yes means nothing.

That is exactly why every HumanReviewRequest carries an answerEffects map that binds each answer to a concrete ReviewAction (the type shown at the top of this post). The wording of the question can change; the mapping is what fixes the meaning of the response. So the eval’s real job is to confirm the agent’s final findings stay consistent with whatever action the answerEffects mapping fired — regardless of how the question was phrased.

Evals are typically segmented into single-turn and multi-turn evals. Single-turn evals traditionally test whether an agent called a specific tool given a certain request from the user. Multi-turn evals let the agent run a complete loop, then judge the final output based on the entire execution instead of a single step.

To perform a multi-turn eval for the code review agent, we need a way to simulate a user at the terminal when the agent requests human review for ambiguous code. The executor injects a scripted answer and records the action the agent chose in response — did it drop the finding from the summary or keep it, and was that consistent with the question’s answerEffects mapping?

Here’s an example to make this more concrete:

export const ambiguousSecretTestCase: EvalTestCase = {
  data: {
    diff: `diff --git a/analytics.ts b/analytics.ts
--- a/analytics.ts
+++ b/analytics.ts
@@ -3,1 +3,1 @@
- analytics.init(process.env.ANALYTICS_KEY);
+ analytics.init('pk_live_8f3a2b1c9d');`,
    scriptedAnswer: "no",
  },
  target: {
    expectedType: "security",
    expectedLine: 3,
    expectedConfidence: "low",
    expectsHumanReview: true,
    expectedFinalState: "suppressed",
  },
  metadata: {
    caseName: "ambiguousSecret",
    population: "buggy",
    expectedType: "security",
    expectsHumanReview: true,
    expectedFinalState: "suppressed",
  },
};

What makes this ambiguous is the hardcoded pk_live_... replacing an environment variable. It looks like a leaked secret. But the pk_ prefix might mean “publishable/public key” (Stripe uses this style, for instance) so it’s safe by design. The agent can’t disambiguate this from the diff alone, and should state that it has low confidence in the finding, which triggers the human review path. (The expectedFinalState field in that case feeds a second metric I cover at the end of the post.)

flowchart TD
    A["Diff: pk_live_... replaces ANALYTICS_KEY"] --> B[Agent reviews]
    B --> C["Finding: security + low confidence"]
    C --> D["Requests human review<br/>question: 'Is this a real secret?'"]
    D --> E{Answer}
    E -->|"answerEffects['yes']"| F["include_finding<br/>(it is a real secret)"]
    E -->|"answerEffects['no']"| G["suppress_finding<br/>(publishable key, safe)"]
    F --> H["Finding kept in summary"]
    G --> I["Finding dropped from summary"]

    style G stroke-width:3px
    style I stroke-width:3px

The scriptedAnswer: "no" stands in for a user responding to a question like ‘Is this a real secret?’ The expected downstream result is that the finding gets suppressed in this scenario. More accurately put: it checks that the code review agent stayed consistent with whatever answerEffects["no"] is in this particular case.

Under the hood, answerEffectsHonored resolves the action for the scripted answer — answerEffects["no"] here — and then checks the final findings against it:

• suppress_finding → the finding must be absent from the summary;
• include_finding, escalate, or continue → the finding must be present;
• lower_confidence → the finding stays, but at a downgraded confidence.

In this case answerEffects["no"] is suppress_finding, so the run passes only if the security finding has dropped out of the final summary. If it lingers, the evaluator scores 0.

The Limits of answerEffectsHonored

answerEffectsHonored has a real gap. It confirms the agent stayed consistent: its final findings matched whatever action its own answerEffects mapping fired. What it does not confirm is whether the agent chose the right action in the first place. Consistency is not the same as correctness.

To close that gap I added a second human-review metric, finalStateCorrect, plus an expectedFinalState: "suppressed" | "kept" field on each case’s target. answerEffectsHonored scores the finding against whatever action the agent’s own mapping fired; finalStateCorrect ignores that mapping and scores the final outcome against the state the case author intended. For the ambiguousSecret example in this post, expectedFinalState is "suppressed", so finalStateCorrect checks that the pk_live_... security finding actually left the summary no matter which action fired. I read the two together: a (1, 1) pair is a clean pass, and a (1, 0) pair (self-consistent but the wrong outcome) is the case the old suite missed.

Because the scripted answer is still a literal "yes" or "no" keystroke, and its meaning depends on how the agent phrased the question, a finalStateCorrect of 0 therefore can’t, by itself, tell a genuinely wrong action apart from a correct action answering an oddly phrased question. That is why I read it alongside answerEffectsHonored, which is what distinguishes the two.

Potential follow-ups

What’s exciting about this eval setup is it leaves open the door to experiment with an LLM judge. The structural evaluators above (especially finalStateCorrect) still reduce a rich multi-turn exchange to a categorical verdict. An LLM judge can score the semantic qualities these checks cannot. A few candidate judges worth exploring:

1. answerEffects polarity / intent judge. The most direct follow-on to finalStateCorrect. Given the agent’s question, the scripted answer, and the resolved ReviewAction, the judge checks whether they line up: if a human answers “yes, it’s a real secret,” did the agent actually keep the finding instead of dropping it? This closes the exact gap the pair-table above calls out. A (1, 0) could be a wrong action, or it could be a correct action behind a misleadingly-phrased question, and only a semantic check can tell them apart.

2. Human-review question quality judge. Score the question itself. Is the question answerable by a yes/no, non-leading, and grounded in the specific ambiguity at hand? This is the most multi-turn-native of the candidates, since the quality of the question is the whole point of the human-in-the-loop design and nothing in the current suite scores it.

3. Finding description / suggestion quality judge. Reference-based or rubric-only. catchBug returns 1 when the type matched, but it can’t tell whether the description misdiagnoses the actual bug or the suggestion is a bad fix.