Build an LLM-as-Judge Evaluation Harness in TypeScript: Scorers, Rubrics, and a CI Gate

Every team that ships an AI feature hits the same wall at roughly the same time. The demo works. The prompt is tuned. The thing goes to production and helps real users. Then someone changes the prompt to fix one annoying case, or swaps the model to cut cost, or adds a retrieval step — and nobody can say whether the product just got better or quietly got worse. The unit tests are green because the unit tests check that JSON parses, not that the answer was any good. The only real signal is a support ticket three days later.

This is the gap evaluation closes, and in 2026 it has moved from a nice-to-have to the center of the production AI stack. The tooling ecosystem spent the last two years building frameworks to call models; the conversation this year is about how to grade them. The reason is simple arithmetic: a non-deterministic system with no regression test is a system you cannot safely change, and a system you cannot safely change is one that rots. Evals are the regression test for probabilistic software.

What almost nobody tells you is that the machinery underneath an eval harness is not complicated. It is a list of test cases, a set of scoring functions, a loop that runs them with bounded concurrency, and a gate that fails your build when the numbers drop. You can buy this bundled into a proprietary platform with a per-seat price and a dashboard, or you can own it in about two hundred and fifty lines of TypeScript that runs anywhere, talks to any model, and is trivially testable without spending a cent on tokens. This tutorial builds the second thing.

By the end you will have a small, dependency-free harness with four moving parts: a Scorer interface that unifies cheap deterministic checks and expensive LLM-as-judge grading, a runEval runner that handles concurrency and error isolation, a rubricScorer that turns any model into a grader, and a gate that wires the result to your CI exit code. Every piece is covered by tests, and the whole thing is provider-agnostic — you plug in Anthropic, OpenAI, a local model, or a deterministic mock through a single one-line interface.

The companion code lives at CrashBytes/ByteSizedExamples/llm-eval-harness-typescript. Clone it, run npm install && npm test, and follow along.

Why evals became the center of gravity

Before we write code, it is worth being precise about what an eval actually buys you, because the answer shapes every design decision that follows. A test for deterministic software asserts an exact output: given this input, the function returns exactly that. A model does not work that way. Ask the same question twice and you get two phrasings of the same idea, or two different ideas, or one right answer and one confidently wrong one. You cannot assert equality. What you can assert is that the output clears a quality bar, and that the fraction of outputs clearing that bar does not fall when you change something.

That reframing — from "is this exactly right" to "what fraction of a representative sample is good enough" — is the entire game. It means an eval is really three things stacked together: a dataset that represents what your users actually do, a scoring method that approximates human judgment cheaply enough to run on every commit, and a threshold that turns a distribution of scores into a single pass-or-fail decision. Get those three right and you can change your prompt, your model, or your architecture with the same confidence a backend engineer changes a function with a green test suite. Get them wrong and you are flying blind with a dashboard that makes you feel safe.

The chart above is illustrative, not measured, but the shape matches what teams report once they instrument this properly: manual spot-checks catch the egregious failures and miss the subtle ones, deterministic asserts catch format and keyword regressions but are blind to whether an answer is actually good, and the combination of cheap deterministic checks with an LLM judge for the fuzzy criteria gets you most of the way to what a careful human reviewer would flag — at a cost low enough to run in CI. That combination is what we are building.

What we are actually building

The harness has four public primitives, each a small file you can read in one sitting:

• TestCase and Scorer — the data model. A test case is an input, an optional gold answer, and metadata. A scorer maps one model output to a normalized score between zero and one. Everything else composes these two.
• Deterministic scorers — exactMatch, includes, regexMatch, similarity, and jsonField. Free, fast, and the first line of defense.
• rubricScorer — the LLM-as-judge. It builds a strict grading prompt from a rubric, calls a model, and parses the verdict into a number.
• runEval and gate — the runner that executes a whole suite with bounded concurrency and per-case error isolation, and the CI gate that turns the resulting report into a build decision.

The design rule that holds the whole thing together: every scorer, cheap or expensive, returns a number in the same range and implements the same interface. The runner does not know or care whether a score came from a regex or from Claude. That uniformity is what lets you weight a free substring check and an expensive judgment together into one decision.

Part 1: the data model

Start with the types, because they are the contract every other file depends on. Create src/types.ts:

export interface TestCase {
  id: string
  input: string
  expected?: string
  tags?: string[]
  metadata?: Record<string, unknown>
}

export type Target = (
  input: string,
  testCase: TestCase
) => Promise<string> | string

export interface ScoreArgs {
  output: string
  expected?: string
  testCase: TestCase
}

export interface Scorer {
  name: string
  score(args: ScoreArgs): Promise<number> | number
}

Four ideas, and they are worth slowing down on. A TestCase is string-in, string-out plus an optional expected answer — this mirrors how nearly every real LLM eval works, and resisting the urge to make it generic over arbitrary input types keeps the whole harness readable. A Target is the system under test: hand it an input, get back text. Crucially the target is a plain function, so it can wrap your production agent, a single model call, or a deterministic stub. And a Scorer returns a number that may be a promise, because a deterministic check resolves instantly but a judge call hits the network.

The result types describe what a run produces:

export interface CaseResult {
  id: string
  input: string
  output: string
  score: number
  passed: boolean
  scores: Record<string, number>
  details?: Record<string, unknown>
  error?: string
}

export interface RunReport {
  total: number
  passed: number
  failed: number
  passRate: number
  meanScore: number
  byScorer: Record<string, number>
  results: CaseResult[]
}

A CaseResult keeps both the weighted aggregate score and the raw per-scorer scores, because when a case fails you want to know which scorer flagged it — was the format wrong, or was the content wrong? The error field exists so a target that throws fails that one case without aborting the run, a property we will lean on hard in the runner. The RunReport is the suite-level summary: how many passed, the mean score, and a per-scorer breakdown so you can see at a glance that, say, your factual-accuracy criterion is dragging while tone is fine.

Part 2: deterministic scorers

Reach for the cheap checks first. They cost nothing, run in microseconds, and catch a surprising share of regressions — a prompt change that breaks your JSON format or drops a required keyword does not need a language model to detect. Create src/scorers.ts:

import type { Scorer, ScoreArgs } from './types.js'

export const exactMatch = (opts: { caseSensitive?: boolean } = {}): Scorer => ({
  name: 'exactMatch',
  score({ output, expected }: ScoreArgs) {
    if (expected === undefined) return 0
    const norm = (s: string) =>
      (opts.caseSensitive ? s : s.toLowerCase()).trim()
    return norm(output) === norm(expected) ? 1 : 0
  },
})

export const includes = (opts: { caseSensitive?: boolean } = {}): Scorer => ({
  name: 'includes',
  score({ output, expected }: ScoreArgs) {
    if (!expected) return 0
    const haystack = opts.caseSensitive ? output : output.toLowerCase()
    const needle = opts.caseSensitive ? expected : expected.toLowerCase()
    return haystack.includes(needle) ? 1 : 0
  },
})

export const regexMatch = (pattern: RegExp): Scorer => ({
  name: 'regexMatch',
  score({ output }: ScoreArgs) {
    return pattern.test(output) ? 1 : 0
  },
})

Each scorer is a factory that returns the Scorer shape, which lets callers configure behavior (case sensitivity, a pattern) while the runner sees a uniform object. exactMatch and includes are your bread and butter for classification-style tasks with known answers; regexMatch is how you assert format without asserting content — that a phone number looks like a phone number, that an answer ends with a citation, that no forbidden phrase appears.

For fuzzier reference comparison, a normalized edit distance gives partial credit instead of a hard zero:

export function levenshtein(a: string, b: string): number {
  const m = a.length
  const n = b.length
  if (m === 0) return n
  if (n === 0) return m

  let prev = new Array<number>(n + 1)
  let curr = new Array<number>(n + 1)
  for (let j = 0; j <= n; j++) prev[j] = j

  for (let i = 1; i <= m; i++) {
    curr[0] = i
    for (let j = 1; j <= n; j++) {
      const cost = a[i - 1] === b[j - 1] ? 0 : 1
      curr[j] = Math.min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + cost)
    }
    ;[prev, curr] = [curr, prev]
  }
  return prev[n]
}

export const similarity = (): Scorer => ({
  name: 'similarity',
  score({ output, expected }: ScoreArgs) {
    if (expected === undefined) return 0
    const a = output.trim()
    const b = expected.trim()
    const maxLen = Math.max(a.length, b.length)
    if (maxLen === 0) return 1
    return 1 - levenshtein(a, b) / maxLen
  },
})

The similarity scorer maps an edit distance onto the zero-to-one range by dividing by the longer string's length and subtracting from one. Identical strings score one; completely different strings approach zero. It is a blunt instrument — it does not understand meaning, only characters — but it is genuinely useful for catching outputs that have drifted far from a reference while tolerating trivial differences in punctuation or whitespace.

The last deterministic scorer parses structured output and checks a single field, which is how you grade the increasingly common case of a model emitting JSON:

import { extractJson } from './json.js'

export const jsonField = (field: string): Scorer => ({
  name: `jsonField:${field}`,
  score({ output, expected }: ScoreArgs) {
    if (expected === undefined) return 0
    try {
      const parsed = JSON.parse(extractJson(output)) as Record<string, unknown>
      return String(parsed?.[field]) === expected ? 1 : 0
    } catch {
      return 0
    }
  },
})

Note the try/catch returning zero rather than throwing: an unparseable output is a failure of that case, not a crash of the harness. That extractJson helper is doing real work — models love to wrap JSON in prose and code fences — so it is worth its own small file. It walks the string respecting string literals and escapes, and returns just the first balanced object or array:

export function extractJson(raw: string): string {
  const fenced = raw.match(/```(?:json)?\s*([\s\S]*?)```/i)
  const text = fenced ? fenced[1] : raw

  const start = text.search(/[{[]/)
  if (start === -1) throw new Error('no JSON object or array found in response')

  const open = text[start]
  const close = open === '{' ? '}' : ']'
  let depth = 0
  let inString = false
  let escaped = false

  for (let i = start; i < text.length; i++) {
    const ch = text[i]
    if (inString) {
      if (escaped) escaped = false
      else if (ch === '\\') escaped = true
      else if (ch === '"') inString = false
      continue
    }
    if (ch === '"') inString = true
    else if (ch === open) depth++
    else if (ch === close) {
      depth--
      if (depth === 0) return text.slice(start, i + 1)
    }
  }

  throw new Error('unbalanced JSON in response')
}

The relative cost of these checks versus a judge call is not a rounding error — it is the whole reason to run them first. A deterministic scorer is free and finishes before the event loop yields; a judge call is a network round trip and a token bill.

The judge bar is on a different planet — thousands of times the cost and latency of a substring check. That asymmetry is the argument for a layered strategy: filter with the cheap checks, spend the judge only on the criteria that genuinely need understanding. We will weight them accordingly in the runner.

Part 3: the runner

The runner is where the pieces become a system. It takes a suite of test cases and an options object — the target, the scorers, a pass threshold, optional per-scorer weights, a concurrency cap, and a retry count — and returns a RunReport. Two properties matter more than anything else: it must cap how many targets run at once, and a single throwing target must never sink the run.

First a dependency-free concurrency limiter in src/concurrency.ts:

export function pLimit(
  concurrency: number
): <T>(fn: () => Promise<T>) => Promise<T> {
  if (!Number.isInteger(concurrency) || concurrency < 1) {
    throw new Error('concurrency must be an integer >= 1')
  }

  let active = 0
  const queue: Array<() => void> = []

  const next = () => {
    active--
    const run = queue.shift()
    if (run) run()
  }

  return <T>(fn: () => Promise<T>): Promise<T> =>
    new Promise<T>((resolve, reject) => {
      const run = () => {
        active++
        fn().then(resolve, reject).finally(next)
      }
      if (active < concurrency) run()
      else queue.push(run)
    })
}

This is the same pattern any production agent runtime uses under the hood: a counter of in-flight work and a queue of waiting thunks. When a slot frees up, next pulls the next thunk and runs it. Without this you have two bad options — run cases one at a time and wait forever, or fire them all at once and get rate-limited into oblivion. A bounded pool is the only sane default, and four is a reasonable starting concurrency for most provider tiers.

Now the runner itself, in src/runner.ts:

import type {
  CaseResult,
  RunReport,
  Scorer,
  Target,
  TestCase,
} from './types.js'
import { pLimit } from './concurrency.js'

export interface RunOptions {
  target: Target
  scorers: Scorer[]
  threshold?: number
  weights?: Record<string, number>
  concurrency?: number
  retries?: number
}

async function withRetries<T>(
  fn: () => Promise<T>,
  retries: number
): Promise<T> {
  let lastError: unknown
  for (let attempt = 0; attempt <= retries; attempt++) {
    try {
      return await fn()
    } catch (err) {
      lastError = err
    }
  }
  throw lastError
}

function clamp01(n: number): number {
  if (Number.isNaN(n)) return 0
  return Math.min(1, Math.max(0, n))
}

The two helpers encode two policies. withRetries gives a flaky target a few more chances before its case is recorded as an error — transient provider hiccups should not show up as quality regressions. clamp01 is defensive: a buggy scorer that returns 1.4 or a NaN cannot corrupt the aggregate, because every score is forced back into the valid range before it counts. The main loop ties them together:

export async function runEval(
  cases: TestCase[],
  options: RunOptions
): Promise<RunReport> {
  const { target, scorers } = options
  const threshold = options.threshold ?? 0.7
  const concurrency = options.concurrency ?? 4
  const retries = options.retries ?? 0
  const weights = options.weights ?? {}
  const totalWeight = scorers.reduce(
    (sum, s) => sum + (weights[s.name] ?? 1),
    0
  )
  const limit = pLimit(concurrency)

  const results = await Promise.all(
    cases.map(testCase =>
      limit(async (): Promise<CaseResult> => {
        try {
          const output = await withRetries(
            async () => target(testCase.input, testCase),
            retries
          )

          const scores: Record<string, number> = {}
          let weighted = 0
          for (const scorer of scorers) {
            const value = clamp01(
              await scorer.score({
                output,
                expected: testCase.expected,
                testCase,
              })
            )
            scores[scorer.name] = value
            weighted += value * (weights[scorer.name] ?? 1)
          }
          const score = totalWeight === 0 ? 0 : weighted / totalWeight

          return {
            id: testCase.id,
            input: testCase.input,
            output,
            score,
            passed: score >= threshold,
            scores,
          }
        } catch (err) {
          return {
            id: testCase.id,
            input: testCase.input,
            output: '',
            score: 0,
            passed: false,
            scores: {},
            error: err instanceof Error ? err.message : String(err),
          }
        }
      })
    )
  )

  const passed = results.filter(r => r.passed).length
  const meanScore = results.length
    ? results.reduce((sum, r) => sum + r.score, 0) / results.length
    : 0

  const byScorer: Record<string, number> = {}
  for (const scorer of scorers) {
    const values = results
      .map(r => r.scores[scorer.name])
      .filter((v): v is number => typeof v === 'number')
    byScorer[scorer.name] = values.length
      ? values.reduce((a, b) => a + b, 0) / values.length
      : 0
  }

  return {
    total: results.length,
    passed,
    failed: results.length - passed,
    passRate: results.length ? passed / results.length : 0,
    meanScore,
    byScorer,
    results,
  }
}

Read the try/catch carefully, because it is the most important design decision in the file. The target call and every scorer run inside the try; if anything throws, the catch records a failed CaseResult with the error message and the loop moves on. One model timeout in a five-hundred-case suite costs you one case, not the whole run. The weighting is the second key idea: each scorer contributes its score times its weight, normalized by the total weight, so you can say "the judge counts double, the substring check counts once" and get a single blended number per case. A case passes when that blended number clears the threshold.

Concurrency is not free real estate, though — there is a point past which more parallelism buys nothing and just risks rate limits.

The curve flattens hard after eight: the first few slots eliminate almost all the waiting, and beyond that you are bounded by the provider, not by your loop. Pick a concurrency that respects your rate limit with headroom to spare; chasing the last few seconds with thirty-two parallel requests is how you turn a green eval run into a 429 storm. This is the same fan-out-with-a-bounded-pool pattern covered in the parallel subagent orchestrator tutorial — once you have the limiter, it shows up everywhere.

Part 4: the LLM-as-judge

Deterministic scorers cannot tell you whether an answer is helpful, whether a summary is faithful to its source, or whether a tone is appropriate. Those are judgment calls, and the practical way to make them at scale is to ask a strong model to grade against an explicit rubric. This is the technique that took over evaluation in 2026, and the reason is that a well-prompted judge agrees with careful human reviewers often enough to be useful, at a fraction of the cost and latency of a human.

The whole judge rests on one tiny seam — a function from a prompt to text:

export type JudgeFn = (prompt: string) => Promise<string> | string

That is the entire provider abstraction. In production it wraps the Anthropic SDK; in tests it is a deterministic fake. Nothing else in the harness knows what model is behind it. The grading logic lives in a rubric — a list of named criteria scored on an integer scale — and two pure functions that build the prompt and parse the verdict. Keeping prompt-building and parsing free of any I/O is what makes them unit-testable without a network. Here is the prompt builder from src/rubric.ts:

export function buildJudgePrompt(rubric: Rubric, sample: JudgeSample): string {
  const scale = rubric.scale ?? 5
  const criteriaList = rubric.criteria
    .map((c, i) => `${i + 1}. ${c.name}: ${c.description}`)
    .join('\n')
  const referenceBlock = sample.expected
    ? `\nReference answer:\n"""\n${sample.expected}\n"""\n`
    : ''
  const keys = rubric.criteria
    .map(c => `"${c.name}": <integer 1-${scale}>`)
    .join(', ')

  return [
    `You are a strict evaluator. Score the AI response against each criterion on a 1-${scale} scale ` +
      `(1 = fails completely, ${scale} = perfect). Be conservative; do not award the top score unless the response is excellent.`,
    ``,
    `Criteria:`,
    criteriaList,
    ``,
    `User input:`,
    `"""\n${sample.input}\n"""`,
    referenceBlock,
    `AI response:`,
    `"""\n${sample.output}\n"""`,
    ``,
    `Respond with ONLY a JSON object of the form:`,
    `{ "scores": { ${keys} }, "rationale": "<one short sentence>" }`,
  ].join('\n')
}

Three details in this prompt matter disproportionately. The instruction to "be conservative" counteracts a well-documented failure mode where judges drift toward leniency and score everything a four or five. Asking for a one-sentence rationale alongside the scores improves the scores themselves — the same chain-of-thought effect that helps any reasoning task — and gives you something to read when a verdict looks wrong. And demanding a fixed JSON shape is what makes the response machine-parseable instead of a paragraph you have to scrape.

Parsing has to be defensive, because models do not always honor "ONLY a JSON object":

export function parseJudgeResponse(
  raw: string,
  rubric: Rubric
): ParsedJudgement {
  const obj = JSON.parse(extractJson(raw)) as {
    scores?: Record<string, unknown>
    rationale?: unknown
  }
  if (
    !obj ||
    typeof obj !== 'object' ||
    !obj.scores ||
    typeof obj.scores !== 'object'
  ) {
    throw new Error('judge response missing a "scores" object')
  }

  const scale = rubric.scale ?? 5
  const scores: Record<string, number> = {}
  for (const c of rubric.criteria) {
    const value = Number((obj.scores as Record<string, unknown>)[c.name])
    if (!Number.isFinite(value)) {
      throw new Error(`judge omitted a numeric score for criterion "${c.name}"`)
    }
    scores[c.name] = Math.min(scale, Math.max(1, value))
  }

  return {
    scores,
    rationale: typeof obj.rationale === 'string' ? obj.rationale : undefined,
  }
}

The parser validates that every criterion got a finite number and clamps each to the legal range, so a judge that returns a nine on a five-point scale gets pulled back to five rather than poisoning the math. A missing criterion throws, which — thanks to the runner's per-case isolation — fails that case loudly instead of silently scoring it zero. Finally the scorer in src/judge.ts collapses the per-criterion grades into one weighted number:

export function rubricScorer(
  judge: JudgeFn,
  rubric: Rubric,
  options: RubricScorerOptions = {}
): Scorer {
  const scale = rubric.scale ?? 5
  if (scale < 2) throw new Error('rubric scale must be >= 2')
  const totalWeight = rubric.criteria.reduce(
    (sum, c) => sum + (c.weight ?? 1),
    0
  )
  if (totalWeight <= 0)
    throw new Error('rubric criteria weights must sum to > 0')

  return {
    name: options.name ?? 'rubric',
    async score({ output, expected, testCase }) {
      const prompt = buildJudgePrompt(rubric, {
        input: testCase.input,
        output,
        expected,
      })
      const raw = await judge(prompt)
      const { scores } = parseJudgeResponse(raw, rubric)

      let weighted = 0
      for (const c of rubric.criteria) {
        const normalized = (scores[c.name] - 1) / (scale - 1)
        weighted += normalized * (c.weight ?? 1)
      }
      return weighted / totalWeight
    },
  }
}

The mapping (score - 1) / (scale - 1) turns a one-through-five grade into the zero-through-one range every other scorer uses: a one becomes zero, a five becomes one, a three lands at exactly one half. Now a judge verdict and a regex check speak the same language and the runner can blend them without special cases.

Not all judging techniques are equally reliable, and the difference between a naive judge and a well-constructed one is large.

The progression is the practical roadmap for hardening a judge: start with a rubric and a rationale, give it a reference answer when you have one, and for the criteria that matter most, run a small panel of independent judgments and take the majority. Each step costs more and buys more agreement; the harness supports all of them because a panel is just three rubricScorer calls combined, and a reference answer is already wired through expected.

Part 5: the CI gate

A report nobody acts on is a dashboard, not a test. The point of the harness is to fail a build when quality drops, and that is the job of src/gate.ts:

import type { RunReport } from './types.js'

export interface GateOptions {
  minPassRate?: number
  minMeanScore?: number
  baseline?: RunReport
  maxRegression?: number
}

export interface GateResult {
  ok: boolean
  reasons: string[]
}

export function gate(report: RunReport, options: GateOptions): GateResult {
  const reasons: string[] = []

  if (
    options.minPassRate !== undefined &&
    report.passRate < options.minPassRate
  ) {
    reasons.push(
      `pass rate ${pct(report.passRate)} is below required ${pct(options.minPassRate)}`
    )
  }

  if (
    options.minMeanScore !== undefined &&
    report.meanScore < options.minMeanScore
  ) {
    reasons.push(
      `mean score ${report.meanScore.toFixed(3)} is below required ${options.minMeanScore.toFixed(3)}`
    )
  }

  if (options.baseline && options.maxRegression !== undefined) {
    const drop = options.baseline.meanScore - report.meanScore
    if (drop > options.maxRegression) {
      reasons.push(
        `mean score regressed by ${drop.toFixed(3)} vs baseline ` +
          `(max allowed ${options.maxRegression.toFixed(3)})`
      )
    }
  }

  return { ok: reasons.length === 0, reasons }
}

The gate supports two complementary policies. Absolute floors — a minimum pass rate and a minimum mean score — catch the case where quality is simply unacceptable. Regression detection compares this run against a stored baseline report and fails if the mean score dropped by more than a tolerance you set, which catches the subtler and more common case: nothing is broken, the numbers just crept down after a change that looked harmless. The reasons array is the difference between a useful CI failure and an infuriating one — instead of "eval failed," you get "mean score regressed by 0.071 vs baseline (max allowed 0.050)," which tells the engineer exactly what happened.

In a real pipeline you commit a baseline report to the repo, run the suite on every pull request, and treat a regression beyond tolerance as a blocking failure. The picture over a series of commits is what makes the gate worth the trouble.

The model-swap commit dips below the floor and the gate stops it before it merges; the revert restores the score; the retrieval change clears the bar and ships. That is the entire value proposition rendered as a chart — a probabilistic system with the same change-safety a deterministic one gets from unit tests. The discipline mirrors the one in the durable agent memory layer tutorial: small, tested, observable primitives that you can reason about and trust.

Part 6: wiring a real provider

Everything so far ran without a single token spent, because the judge was a fake. Swapping in a real model is one function. Here is the judge bound to Claude:

import Anthropic from '@anthropic-ai/sdk'
import { rubricScorer, type Rubric } from './src/index.js'

const anthropic = new Anthropic()

const judge = async (prompt: string) => {
  const msg = await anthropic.messages.create({
    model: 'claude-opus-4-8',
    max_tokens: 256,
    messages: [{ role: 'user', content: prompt }],
  })
  return msg.content.map(b => (b.type === 'text' ? b.text : '')).join('')
}

const rubric: Rubric = {
  criteria: [
    {
      name: 'helpfulness',
      description: 'directly and completely answers the user',
      weight: 2,
    },
    {
      name: 'faithfulness',
      description: 'makes no claims unsupported by the input',
    },
    { name: 'safety', description: 'no harmful or policy-violating content' },
  ],
  scale: 5,
}

const judgeScorer = rubricScorer(judge, rubric)

That judge function is the only provider-specific code in the entire project. Point it at a different lab, a local model behind an OpenAI-compatible endpoint, or a router, and nothing downstream changes. Use a strong model as the judge even if your production system runs a cheaper one — the judge runs far less often than the system under test, so paying for quality grading is usually the right trade. If you are choosing models for either role, the frontier-model supercycle analysis maps the current field.

Putting it together, a complete eval run reads like plain English:

import { runEval, gate, includes } from './src/index.js'

const report = await runEval(testCases, {
  target: input => myAgent(input),
  scorers: [includes(), judgeScorer],
  weights: { includes: 1, rubric: 3 },
  threshold: 0.7,
  concurrency: 4,
})

const result = gate(report, {
  minPassRate: 0.85,
  minMeanScore: 0.75,
  maxRegression: 0.05,
  baseline: loadBaseline(),
})

if (!result.ok) {
  for (const reason of result.reasons) console.error(`FAIL: ${reason}`)
  process.exit(1)
}

Run the suite, gate the result, exit non-zero on failure. Drop that into a CI job and you have an eval gate.

Part 7: pitfalls that bite in production

The harness is honest about what it is, but using it well means knowing where LLM-as-judge goes wrong. These are the failure modes worth designing against from the start.

The first is judge leniency and self-preference. Judges drift generous, and they tend to prefer outputs from the same model family. The "be conservative" instruction helps with the first; using a judge from a different family than the system under test helps with the second. Never let a model grade its own homework if you can avoid it.

The second is positional and verbosity bias. When a judge compares two answers, it favors whichever came first and whichever is longer, regardless of quality. Our harness sidesteps the worst of this by scoring each output absolutely against a rubric rather than ranking pairs, but if you build a pairwise variant, randomize order and control for length.

The third is distribution drift in your dataset. An eval is only as good as how well its cases represent real traffic. A suite frozen six months ago grades you on yesterday's problems. Treat the dataset as living: sample real (consented, anonymized) production inputs periodically, especially the ones your users phrase in ways you did not anticipate.

The fourth is gaming the metric. Optimize hard enough against any single score and the model learns to satisfy the letter of the rubric while missing its spirit. The defense is a rubric with several independent criteria and a healthy fraction of deterministic checks the judge cannot talk its way around. Security intersects here too — adversarial inputs can try to manipulate the judge itself, a close cousin of the attacks in the prompt-injection threat model.

None of these are reasons to skip evals — they are reasons to run them with eyes open. A judge that agrees with humans seventy-five percent of the time is imperfect, but it is vastly better than the zero percent coverage you get from shipping and watching tickets, and the deterministic scorers backstop the criteria the judge handles poorly. The combination is the point. Where the field goes from here is the subject of my prediction that an eval gate becomes a standard CI primitive for AI apps by the end of 2027 — the same trajectory continuous integration itself followed from optional to table stakes.

Where this fits

You now have a complete, tested evaluation harness in about two hundred and fifty lines: a uniform Scorer interface, five deterministic scorers, an LLM-as-judge rubric scorer, a concurrent runner that isolates failures and retries flakiness, and a CI gate with absolute floors and regression detection. It is provider-agnostic, runs its full test suite without spending a token, and drops into any CI pipeline as a blocking check.

The deeper lesson is that evaluation is not a separate discipline bolted onto AI engineering — it is the regression-testing half of it, the part that makes everything else changeable. Once you can measure whether a change made your system better or worse, you can iterate on prompts, models, and architecture with the same confidence you bring to ordinary refactoring. Without it you are guessing, and guessing about a non-deterministic system in production is how good features quietly become bad ones. Clone the repo, point the judge at a real model, write twenty test cases that look like your actual traffic, and wire the gate into CI. That twenty-case suite will catch more regressions than any amount of manual spot-checking, and it grows with you.

Further reading

• Build a Parallel Subagent Orchestrator in TypeScript — the fan-out-with-bounded-concurrency pattern this harness reuses in its runner.
• Build a Durable Agent Memory Layer in TypeScript — the companion piece on small, tested agent-infrastructure primitives.
• Evals move to the center of the AI stack as agent tooling matures — the industry context behind why evaluation became a 2026 headline.
• Prediction: an eval gate becomes a standard CI primitive for AI apps by the end of 2027 — my dated, falsifiable claim on where this is heading.
• Companion code on GitHub — the full, tested project from this tutorial.