Building A Multi-Model Evaluation Harness In TypeScript - A Practical Tutorial For Engineering Teams Who Have Decided They Need Their Own

Why This Tutorial Exists

Over the last eighteen months the question "should our team build a private AI evaluation harness?" has moved from a sophisticated question that only a few engineering organizations were asking to a basic question that most serious engineering organizations are answering in the affirmative. As I argued in my analysis of the rise of private eval harnesses, the investment has become close to unavoidable for any enterprise with meaningful AI spend.

This tutorial is the companion hands-on guide. It walks through the implementation of a minimal but production-viable evaluation harness in TypeScript, with the specific focus of making the architecture clear and the tradeoffs explicit. The code examples are runnable — not pseudocode — and the companion GitHub repository has the complete implementation with tests and configuration.

Target audience: a senior or staff engineer at an organization that has decided to build a private eval harness and wants a credible starting point. The tutorial does not assume you have built an eval harness before, but it does assume TypeScript fluency and familiarity with enterprise software patterns.

Companion repository: github.com/CrashBytes/multi-model-eval-harness-tutorial

Architecture Overview

The harness we will build has five components, each mapping to a directory in the repository.

packages/
  endpoints/      - Endpoint abstraction and adapters
  corpus/         - Task corpus and ground-truth management
  runner/         - Evaluation execution
  scoring/        - Scoring pipeline with variance analysis
  reporting/      - Result aggregation and comparison views

The reason the architecture is split this way is that each of these components evolves on a different cadence. Endpoints change when vendors add or remove APIs; the corpus changes when the workload evolves; the runner and scoring are the stable infrastructure; reporting is the stakeholder-facing layer that iterates based on what the decision-makers actually find useful.

The separation also makes it possible to replace any single component without touching the others. If you eventually decide to migrate scoring to a different framework, the endpoints and corpus do not have to change.

Step 1: The Endpoint Abstraction

The first thing to build is the endpoint abstraction. This is the layer that makes multi-model comparison possible at all. Do this first, because every other component in the harness depends on being able to call "the model" without caring which model it is.

Create packages/endpoints/src/types.ts:

export interface GenerationRequest {
  prompt: string
  maxTokens?: number
  temperature?: number
  stopSequences?: string[]
}

export interface GenerationResponse {
  text: string
  tokensUsed: { input: number; output: number }
  latencyMs: number
  finishReason: 'stop' | 'length' | 'error'
  rawResponse?: unknown
}

export interface ModelEndpoint {
  id: string
  provider: string
  model: string
  generate(request: GenerationRequest): Promise<GenerationResponse>
}

Three observations about this interface.

First, keep it minimal. Every field you add here becomes a field that every adapter has to implement consistently, and the cost of consistency rises faster than linearly with interface complexity. Start with the fields you know you need; add more only when a specific adapter or scorer demands them.

Second, GenerationResponse includes latencyMs as a first-class field. Latency is not an afterthought. If the harness cannot measure latency cleanly, the harness cannot be used for production deployment decisions. Include it from day one.

Third, rawResponse is included as an escape hatch for provider- specific fields. You will not use it in normal scoring, but you will need it when debugging an adapter or when a specific scorer needs something beyond the standard fields.

Now the adapter for a generic OpenAI-compatible endpoint. Create packages/endpoints/src/openai-compatible.ts:

import { ModelEndpoint, GenerationRequest, GenerationResponse } from './types'

export interface OpenAICompatibleConfig {
  id: string
  baseURL: string
  apiKey: string
  model: string
  provider: string
}

export class OpenAICompatibleEndpoint implements ModelEndpoint {
  readonly id: string
  readonly provider: string
  readonly model: string
  private baseURL: string
  private apiKey: string

  constructor(config: OpenAICompatibleConfig) {
    this.id = config.id
    this.baseURL = config.baseURL
    this.apiKey = config.apiKey
    this.model = config.model
    this.provider = config.provider
  }

  async generate(request: GenerationRequest): Promise<GenerationResponse> {
    const start = Date.now()
    const response = await fetch(`${this.baseURL}/chat/completions`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${this.apiKey}`,
      },
      body: JSON.stringify({
        model: this.model,
        messages: [{ role: 'user', content: request.prompt }],
        max_tokens: request.maxTokens,
        temperature: request.temperature ?? 0,
        stop: request.stopSequences,
      }),
    })
    const latencyMs = Date.now() - start

    if (!response.ok) {
      throw new Error(`Endpoint ${this.id} failed: ${response.status}`)
    }

    const body = await response.json()
    return {
      text: body.choices[0].message.content,
      tokensUsed: {
        input: body.usage.prompt_tokens,
        output: body.usage.completion_tokens,
      },
      latencyMs,
      finishReason: body.choices[0].finish_reason,
      rawResponse: body,
    }
  }
}

This adapter covers OpenAI, Anthropic (via Anthropic's OpenAI-compatible endpoint), most self-hosted inference servers (vLLM, TGI), and all the major third-party providers. For providers that are not OpenAI-compatible — notably some of the older Google endpoints — you would write a dedicated adapter following the same interface pattern.

Step 2: The Corpus And Ground Truth

The corpus is where the harness actually earns its value. This is the component you should spend the most engineering attention on, and it is also the component that is least amenable to prescription — the right corpus for your organization depends on your workload.

The pattern worth showing in code is the generic structure. Create packages/corpus/src/types.ts:

export interface EvalTask<TInput = unknown, TReference = unknown> {
  id: string
  category: string
  input: TInput
  reference: TReference
  metadata: Record<string, unknown>
}

export interface TaskCorpus<TInput = unknown, TReference = unknown> {
  id: string
  version: string
  tasks: EvalTask<TInput, TReference>[]
}

The parametric typing matters. A coding-assistance corpus has different input and reference types than a document-classification corpus. Parametric typing means the scorer can be specific to the task type without forcing all corpora into the same schema.

For a concrete example, a coding-assistance task type:

export interface CodingTaskInput {
  prompt: string
  filesBefore: Record<string, string>
  issueDescription: string
}

export interface CodingTaskReference {
  filesAfter: Record<string, string>
  testCommand: string
  successCriteria: string
}

export type CodingTask = EvalTask<CodingTaskInput, CodingTaskReference>

The reference for a coding task is the set of files after a correct resolution, plus the test command that validates correctness, plus a human-readable success criterion. The test command is what makes the scoring objective; the success criterion is what makes it interpretable when the scoring gets complicated.

Corpus versioning is not optional. If you change the corpus, the scores are no longer comparable across runs. The simplest implementation is a version string in the corpus, plus a transformation log documenting what changed between versions. In practice, you will want to keep multiple corpus versions available and to run evaluations against fixed versions rather than always against the latest.

Step 3: The Runner

The runner is the boring-but-critical part of the harness. It takes a corpus and a set of endpoints and runs every task against every endpoint with appropriate parallelism, retry logic, and error handling.

Create packages/runner/src/runner.ts:

import { ModelEndpoint } from '../../endpoints/src/types'
import { EvalTask, TaskCorpus } from '../../corpus/src/types'

export interface RunResult {
  taskId: string
  endpointId: string
  runIndex: number
  generated: string
  tokensUsed: { input: number; output: number }
  latencyMs: number
  error?: string
}

export interface RunnerConfig {
  runsPerTask: number
  concurrency: number
  promptTemplate: (task: EvalTask) => string
}

export async function runEvaluation(
  corpus: TaskCorpus,
  endpoints: ModelEndpoint[],
  config: RunnerConfig
): Promise<RunResult[]> {
  const results: RunResult[] = []
  const queue: Array<() => Promise<void>> = []

  for (const task of corpus.tasks) {
    for (const endpoint of endpoints) {
      for (let runIndex = 0; runIndex < config.runsPerTask; runIndex++) {
        queue.push(async () => {
          try {
            const response = await endpoint.generate({
              prompt: config.promptTemplate(task),
              temperature: 0,
            })
            results.push({
              taskId: task.id,
              endpointId: endpoint.id,
              runIndex,
              generated: response.text,
              tokensUsed: response.tokensUsed,
              latencyMs: response.latencyMs,
            })
          } catch (err) {
            results.push({
              taskId: task.id,
              endpointId: endpoint.id,
              runIndex,
              generated: '',
              tokensUsed: { input: 0, output: 0 },
              latencyMs: 0,
              error: err instanceof Error ? err.message : String(err),
            })
          }
        })
      }
    }
  }

  await processQueue(queue, config.concurrency)
  return results
}

async function processQueue(
  queue: Array<() => Promise<void>>,
  concurrency: number
): Promise<void> {
  const workers = Array.from({ length: concurrency }, async () => {
    while (queue.length > 0) {
      const task = queue.shift()
      if (task) await task()
    }
  })
  await Promise.all(workers)
}

Three things to notice about this runner.

First, runsPerTask is first-class. Running each task multiple times is not optional for a serious harness. Single-run scores are misleading; the variance across runs is typically three to six percentage points on most benchmarks, and decisions need to be made against the variance, not against a single sample.

Second, error handling is part of the normal flow. A task that fails produces a RunResult with an error field and an empty generated. This is important because you want to see the failure rate in the results, not have failures silently disappear from the corpus. Scoring will treat errors as failures, which is almost always the correct behavior.

Third, the concurrency is explicit. Higher concurrency reduces evaluation wall-clock time but increases pressure on vendor rate limits and can produce different latency measurements than production conditions. Tune the concurrency to match your production traffic pattern, not just to minimize evaluation time.

Step 4: The Scoring Pipeline

Scoring is where the harness produces its outputs. The key insight is that scoring is per-task-type, not per-harness. A coding-task scorer is fundamentally different from a classification-task scorer. The harness architecture should make this explicit.

Create packages/scoring/src/types.ts:

export interface TaskScore {
  taskId: string
  endpointId: string
  runIndex: number
  passed: boolean
  partialCredit?: number
  failureReason?: string
  notes?: string
}

export interface Scorer<TInput, TReference> {
  name: string
  score(
    task: EvalTask<TInput, TReference>,
    generated: string
  ): Promise<TaskScore>
}

And a concrete scorer for coding tasks:

import { execSync } from 'node:child_process'
import { mkdtempSync, writeFileSync, rmSync } from 'node:fs'
import { join } from 'node:path'
import { tmpdir } from 'node:os'

export const codingTaskScorer: Scorer<CodingTaskInput, CodingTaskReference> = {
  name: 'coding-test-execution',

  async score(task, generated) {
    const workdir = mkdtempSync(join(tmpdir(), 'eval-'))
    try {
      const generatedFiles = parseGeneratedFiles(generated)
      for (const [path, content] of Object.entries(task.input.filesBefore)) {
        writeFileSync(join(workdir, path), content)
      }
      for (const [path, content] of Object.entries(generatedFiles)) {
        writeFileSync(join(workdir, path), content)
      }
      execSync(task.reference.testCommand, {
        cwd: workdir,
        stdio: 'pipe',
      })
      return {
        taskId: task.id,
        endpointId: '',
        runIndex: 0,
        passed: true,
      }
    } catch (err) {
      return {
        taskId: task.id,
        endpointId: '',
        runIndex: 0,
        passed: false,
        failureReason: err instanceof Error ? err.message : String(err),
      }
    } finally {
      rmSync(workdir, { recursive: true, force: true })
    }
  },
}

This scorer runs the reference test command against the generated files and reports pass or fail. The specifics of parseGeneratedFiles depend on how you prompt the model to produce its output — for a structured-output prompt the parsing is trivial; for a free-text prompt the parsing can be surprisingly tricky. Most mature harnesses iterate on the prompt-and-parsing together.

Step 5: Variance Analysis And Reporting

This is the step where the harness stops being an evaluation runner and becomes a decision instrument. The output you actually want is not a table of pass/fail results but an analysis of how the models compare, with appropriate statistical treatment of variance.

The core computation:

export interface ModelComparisonStats {
  endpointId: string
  totalTasks: number
  passRate: number
  passRateStdDev: number
  avgLatencyMs: number
  avgCostPerTask: number
  ninetyFifthPercentileLatencyMs: number
}

export function analyzeResults(
  results: RunResult[],
  scores: TaskScore[],
  costPerThousandInputTokens: Record<string, number>,
  costPerThousandOutputTokens: Record<string, number>
): ModelComparisonStats[] {
  const byEndpoint = new Map<
    string,
    { results: RunResult[]; scores: TaskScore[] }
  >()
  for (const result of results) {
    if (!byEndpoint.has(result.endpointId)) {
      byEndpoint.set(result.endpointId, { results: [], scores: [] })
    }
    byEndpoint.get(result.endpointId)!.results.push(result)
  }
  for (const score of scores) {
    if (byEndpoint.has(score.endpointId)) {
      byEndpoint.get(score.endpointId)!.scores.push(score)
    }
  }

  const stats: ModelComparisonStats[] = []
  for (const [endpointId, data] of byEndpoint) {
    const byTaskId = new Map<string, boolean[]>()
    for (const score of data.scores) {
      if (!byTaskId.has(score.taskId)) byTaskId.set(score.taskId, [])
      byTaskId.get(score.taskId)!.push(score.passed)
    }
    const taskPassRates = Array.from(byTaskId.values()).map(
      runs => runs.filter(p => p).length / runs.length
    )
    const passRate =
      taskPassRates.reduce((s, r) => s + r, 0) / taskPassRates.length
    const passRateStdDev = Math.sqrt(
      taskPassRates.reduce((s, r) => s + Math.pow(r - passRate, 2), 0) /
        taskPassRates.length
    )
    const latencies = data.results.map(r => r.latencyMs).sort((a, b) => a - b)
    const avgLatencyMs = latencies.reduce((s, l) => s + l, 0) / latencies.length
    const p95LatencyMs = latencies[Math.floor(latencies.length * 0.95)]
    const costPerTask =
      data.results.reduce(
        (s, r) =>
          s +
          (r.tokensUsed.input * (costPerThousandInputTokens[endpointId] ?? 0)) /
            1000 +
          (r.tokensUsed.output *
            (costPerThousandOutputTokens[endpointId] ?? 0)) /
            1000,
        0
      ) / data.results.length

    stats.push({
      endpointId,
      totalTasks: byTaskId.size,
      passRate,
      passRateStdDev,
      avgLatencyMs,
      avgCostPerTask: costPerTask,
      ninetyFifthPercentileLatencyMs: p95LatencyMs,
    })
  }
  return stats
}

The key insight here is the standard deviation calculation. This is what separates "the harness says model A wins" from "the harness produces statistically defensible comparison between model A and model B." Decisions need to be made against the confidence intervals, not against the point estimates. A harness that reports pass rates without reporting the variance is producing a single number where the useful output is a distribution.

Step 6: Running The Harness

The end-to-end example ties everything together:

import { OpenAICompatibleEndpoint } from './endpoints/src/openai-compatible'
import { loadCodingCorpus } from './corpus/src/coding-corpus'
import { runEvaluation } from './runner/src/runner'
import { codingTaskScorer } from './scoring/src/coding-scorer'
import { analyzeResults } from './reporting/src/analyze'

async function main() {
  const endpoints = [
    new OpenAICompatibleEndpoint({
      id: 'incumbent',
      provider: 'VendorA',
      model: 'premier-model-v4',
      baseURL: process.env.VENDOR_A_URL!,
      apiKey: process.env.VENDOR_A_KEY!,
    }),
    new OpenAICompatibleEndpoint({
      id: 'challenger',
      provider: 'VendorB',
      model: 'open-model-744b',
      baseURL: process.env.VENDOR_B_URL!,
      apiKey: process.env.VENDOR_B_KEY!,
    }),
  ]

  const corpus = await loadCodingCorpus('corpus-v3.json')

  const results = await runEvaluation(corpus, endpoints, {
    runsPerTask: 5,
    concurrency: 4,
    promptTemplate: task => buildCodingPrompt(task),
  })

  const scores = await Promise.all(
    results.map(async r => {
      if (r.error) {
        return {
          taskId: r.taskId,
          endpointId: r.endpointId,
          runIndex: r.runIndex,
          passed: false,
          failureReason: r.error,
        }
      }
      const task = corpus.tasks.find(t => t.id === r.taskId)!
      const score = await codingTaskScorer.score(task, r.generated)
      return { ...score, endpointId: r.endpointId, runIndex: r.runIndex }
    })
  )

  const stats = analyzeResults(
    results,
    scores,
    {
      incumbent: 0.008,
      challenger: 0.001,
    },
    {
      incumbent: 0.024,
      challenger: 0.003,
    }
  )

  console.log(JSON.stringify(stats, null, 2))
}

main().catch(console.error)

This is runnable code, given a corpus and working API credentials. The output is a JSON comparison of the endpoints on pass rate, variance, latency, and cost-per-task. It is the simplest version of the decision instrument described in the companion analysis article.

Step 7: From Prototype To Production

The code above is a working prototype. Turning it into a production harness requires several more layers.

Persistence. The prototype runs everything in memory. A production harness stores results in a database so that runs can be compared over time. A simple Postgres schema with tables for runs, scores, and stats is sufficient for most organizations.

Scheduling. A production harness runs on a schedule — weekly, monthly, and on-demand — not just as a one-off. A small scheduler layer that triggers the main function periodically and records results is the minimum needed.

Reporting UI. The JSON output is not a stakeholder-readable artifact. A production harness has a reporting layer that produces the kind of comparison views that executives and procurement teams can consume. A minimal version is a dashboard with time-series of pass rates and latency across endpoints; a mature version is a set of specific vendor-comparison reports triggered by specific procurement events.

Methodology versioning. The prototype does not version the methodology. A production harness does. Each run is tagged with a methodology version, corpus version, and scorer version, so that apples-to-apples comparisons across time are possible.

Access control. The harness produces information that is competitively sensitive. Access control is needed on both the results and on the underlying corpus. Role-based access control with audit logging is the minimum.

Cost and rate limit management. Running evaluations against production APIs costs money. A production harness budgets the evaluations — "this run is authorized for up to $500 in API spend" — and aborts if costs exceed budget. Similarly, rate limit handling needs to be more robust than the prototype's simple concurrency control.

Step 8: Testing The Harness

A harness that is not itself tested produces untrustworthy outputs. This is a recurring failure pattern: teams invest heavily in the harness infrastructure and then discover, months into its use, that the scoring has a subtle bug that has been biasing every evaluation in favor of one vendor or another.

The minimum testing discipline for a harness has three layers.

Scorer unit tests. For every scorer in the harness, there should be a test suite that verifies the scorer produces expected results against a small set of hand-crafted known-good and known-bad task- generation pairs. The test cases should cover edge cases — malformed generated output, partial correctness, generation that looks correct but fails the reference test — and the test cases should be curated by an engineer with domain knowledge, not auto-generated.

import { describe, it, expect } from 'vitest'
import { codingTaskScorer } from './coding-scorer'

describe('codingTaskScorer', () => {
  it('passes when the generated patch makes the test suite pass', async () => {
    const task = loadFixture('fixtures/simple-fix.json')
    const correctPatch = readFile('fixtures/simple-fix-correct.ts')
    const score = await codingTaskScorer.score(task, correctPatch)
    expect(score.passed).toBe(true)
  })

  it('fails when the generated patch does not apply cleanly', async () => {
    const task = loadFixture('fixtures/simple-fix.json')
    const malformed = 'this is not valid typescript'
    const score = await codingTaskScorer.score(task, malformed)
    expect(score.passed).toBe(false)
    expect(score.failureReason).toMatch(/parse|syntax/i)
  })

  it('fails when the generated patch compiles but fails the test', async () => {
    const task = loadFixture('fixtures/simple-fix.json')
    const wrongApproach = readFile('fixtures/simple-fix-wrong.ts')
    const score = await codingTaskScorer.score(task, wrongApproach)
    expect(score.passed).toBe(false)
  })
})

Runner integration tests. The runner should be tested against a small set of fixture endpoints that produce deterministic outputs, so that the runner's concurrency, error handling, and result aggregation can be verified without hitting real vendor APIs.

class FixtureEndpoint implements ModelEndpoint {
  readonly id: string
  readonly provider = 'fixture'
  readonly model = 'fixture'
  private responses: string[]
  private index = 0

  constructor(id: string, responses: string[]) {
    this.id = id
    this.responses = responses
  }

  async generate(_request: GenerationRequest): Promise<GenerationResponse> {
    const text = this.responses[this.index % this.responses.length]
    this.index++
    return {
      text,
      tokensUsed: { input: 10, output: 20 },
      latencyMs: 50,
      finishReason: 'stop',
    }
  }
}

describe('runEvaluation', () => {
  it('runs every task against every endpoint the configured number of times', async () => {
    const endpoint = new FixtureEndpoint('test', ['response'])
    const corpus = buildFixtureCorpus(3)
    const results = await runEvaluation(corpus, [endpoint], {
      runsPerTask: 2,
      concurrency: 2,
      promptTemplate: t => String(t.id),
    })
    expect(results.length).toBe(6)
  })
})

End-to-end smoke tests. A scheduled end-to-end test that runs the harness against fixture endpoints and verifies the complete output shape. This test catches integration regressions that the unit tests miss.

Step 9: CI/CD For The Harness

A harness that is used for real decisions should be under the same engineering discipline as production code. A minimal CI setup:

name: Harness CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test
      - run: npm run build

The scheduled evaluation run is separate from CI. It runs against real vendor APIs, on a schedule, and produces artifacts that are persisted to the harness database. A minimal scheduled-run workflow:

name: Weekly Evaluation

on:
  schedule:
    - cron: '0 6 * * 1'
  workflow_dispatch:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npm run evaluate -- --corpus corpus-v3.json --endpoints all
        env:
          VENDOR_A_KEY: ${{ secrets.VENDOR_A_KEY }}
          VENDOR_B_KEY: ${{ secrets.VENDOR_B_KEY }}
      - run: npm run report -- --format html --output reports/
      - uses: actions/upload-artifact@v4
        with:
          name: weekly-eval-report
          path: reports/

Protect the scheduled-run credentials in environment-specific secrets. Do not run the scheduled evaluation in pull request CI — the cost adds up and the security implications of exposing credentials to PR workflows are substantial.

Step 10: Extending The Harness With New Task Types

The architecture supports new task types cleanly. The pattern is: define the task input and reference types, write the scorer, add the scorer to the scorer registry, update the reporting layer to handle the new task type if needed.

A concrete example — adding a classification task type:

export interface ClassificationTaskInput {
  text: string
  categories: string[]
  instruction: string
}

export interface ClassificationTaskReference {
  correctCategory: string
}

export type ClassificationTask = EvalTask<
  ClassificationTaskInput,
  ClassificationTaskReference
>

export const classificationScorer: Scorer<
  ClassificationTaskInput,
  ClassificationTaskReference
> = {
  name: 'classification-exact-match',

  async score(task, generated) {
    const predicted = extractCategory(generated, task.input.categories)
    const passed = predicted === task.reference.correctCategory
    return {
      taskId: task.id,
      endpointId: '',
      runIndex: 0,
      passed,
      failureReason: passed
        ? undefined
        : `expected ${task.reference.correctCategory}, got ${predicted}`,
    }
  },
}

The extractCategory helper is a small but important piece — it parses the model's generated output and extracts the predicted category, handling variations in how the model might format its answer. The robustness of this helper directly affects the reliability of the classification scores.

Each new task type is essentially a new corpus-scorer pair plus any reporting-layer adjustments. The harness infrastructure — runner, endpoint abstraction, variance analysis, CI — is shared across all task types.

Common Mistakes To Avoid

Three specific mistakes recur in early harness implementations, and they are worth naming explicitly.

Scoring in-band with generation. The prototype runs scoring after generation is complete, which is correct. A surprisingly common mistake is to score each task immediately after generation, inside the same worker loop. This conflates scoring errors with generation errors and makes the results harder to debug. Keep them separated.

Using LLM-as-judge without validation. LLM-as-judge scoring is popular and has its place, but it is not free. An LLM scorer introduces its own biases and variance, and the scorer's reliability should be validated against human judgment on a sample of the corpus before the scorer is used for production decisions. Organizations that skip this validation step often end up with scores that reflect the scorer's idiosyncrasies more than the model under test.

Treating the corpus as static. The corpus is not a fixed artifact. It drifts as your workload drifts, and the harness needs a corpus-evolution process. The minimum is quarterly review of the corpus against sampled production traffic, with documented updates and versioning.

Operational Harness Practices

A harness that runs once a quarter produces approximately zero value. A harness that runs consistently, with documented change management, with stakeholders who know how to consume its outputs, and with the engineering discipline that keeps its methodology stable across runs — that harness produces substantial ongoing value. The difference between the two is operational practice, not code.

Several specific practices are worth making explicit.

Scheduled runs should be boring. The scheduled weekly evaluation should be the uneventful background cadence. Its purpose is to produce stable baseline scores that the organization can trust. The weekly run should not be the venue for trying out new scorers, new corpus versions, or new endpoints. Those experiments belong in ad-hoc runs that are clearly tagged as non-baseline.

Change management on the corpus is load-bearing. Every modification to the corpus — adding tasks, removing tasks, adjusting reference answers — should be tagged with a corpus version bump, reviewed by at least one other engineer, and documented in a corpus changelog. Corpora that mutate silently produce scores that cannot be compared across time, and the harness stops being a decision instrument.

Methodology changes require re-baselining. When the methodology changes — new scorers, changed prompt templates, new variance analysis — the historical scores may no longer be comparable with the new-methodology scores. The honest response is to re-run the evaluation under both methodologies for a transition period, and to clearly label which scores are under which methodology. The cheap shortcut — assume the methodologies are compatible — is how harnesses stop being trustworthy.

Stakeholders need explicit onboarding. The first time the harness produces a surprising result that contradicts a stakeholder's prior, the stakeholder's instinct will be to question the harness rather than the prior. This is a normal psychological response. Mature harness programs have stakeholder onboarding that walks through the methodology, shows the validation tests, and establishes the harness's credibility before it is asked to contradict anyone's beliefs. Doing this before the first controversial result is substantially easier than doing it after.

The harness needs a named technical owner. Without a single named owner, the harness drifts — corpus updates are inconsistent, methodology changes accumulate without review, and the harness's results lose their crispness. Name an owner explicitly, document what they are responsible for, and make that ownership visible at the leadership level.

The harness needs documented escalation paths. When the harness produces a surprising result — "the challenger vendor is within variance of the incumbent" — there should be a documented path for how that result moves into vendor-selection or vendor-renewal conversations. Without that path, the result sits in the harness dashboard and does not drive the decision it should.

Cost And Performance Tuning

A production harness has a cost profile worth understanding. For a corpus of three hundred tasks, five runs per task, across three endpoints, the raw evaluation runs produce roughly four and a half thousand API calls. At enterprise-rate pricing on a top-tier closed-frontier model, that is on the order of two hundred to six hundred dollars per full evaluation run, depending on the task size.

A weekly evaluation cadence therefore costs roughly ten to thirty thousand dollars annually in API spend. This is modest relative to the value the harness produces, but it is not free, and the cost structure affects how the harness is used.

Several patterns reduce cost without substantially degrading value.

Sample the corpus for weekly runs, evaluate the full corpus quarterly. Running a fifty-task subset weekly gives you directional signal; running the full three-hundred-task corpus quarterly gives you the statistical confidence for vendor decisions. Using both cadences together substantially reduces cost while preserving the decision-grade data.

Cache generated outputs when the methodology allows. If the same endpoint, prompt, and temperature produces deterministic outputs, cache the output rather than re-generating. For temperature-zero deterministic runs this is straightforward; for higher-temperature sampling it is not applicable.

Tier the endpoints by cost. Evaluate every endpoint on a small "smoke test" subset on every run. Evaluate the full corpus only against endpoints that are candidates for production deployment. This reduces the cost of tracking many endpoints in parallel.

Use smaller corpora for prompt-engineering iteration. When iterating on prompts, a fifteen-task diagnostic corpus produces faster and cheaper feedback than the full evaluation corpus. Keep the full corpus for decision-grade evaluations.

Conclusion And Next Steps

The harness in this tutorial is deliberately minimal. It covers the core concepts without distracting detail. The companion repository has an extended version with persistence, scheduling, and more robust error handling.

For the broader strategic framing of why the harness matters, see my analysis of private eval harnesses. For the related tutorial on integrating an agent platform against your MCP-enabled tool stack, see the AWS Bedrock getting-started tutorial. For the security-discipline counterpart to harness-driven evaluation discipline, see the MCP authentication crisis piece.

If you work through this tutorial end-to-end and commit to running the harness against a real workload, you will have the foundational capability that separates the organizations making good AI vendor decisions in 2026 from those that are still relying on vendor- curated demonstrations. The investment is real but measured, and the compounding returns on the harness — better vendor decisions, better negotiation leverage, better open-source deployment optionality — are substantial.

The companion repository is at github.com/CrashBytes/multi-model-eval-harness-tutorial. Issues and pull requests welcome.

Appendix: A Minimal Corpus Format Specification

For readers who want to start building a corpus before reading the full companion repository, here is the minimal JSON schema that the code in this tutorial expects.

interface CorpusFile {
  id: string
  version: string
  description: string
  created: string
  tasks: TaskFile[]
}

interface TaskFile {
  id: string
  category: string
  taskType: 'coding' | 'classification' | 'extraction' | 'generation'
  input: unknown
  reference: unknown
  metadata: {
    createdBy: string
    createdAt: string
    reviewedBy?: string
    difficulty?: 'easy' | 'medium' | 'hard'
    expectedPassRate?: number
  }
}

For a coding-task corpus, the input and reference fields follow the CodingTaskInput and CodingTaskReference structure introduced earlier. The expectedPassRate metadata field is optional but useful — it represents the harness author's expectation of how difficult the task is, and substantial divergence between expected and actual pass rates flags tasks that may need review.

The corpus file is committed to the repository, versioned alongside the code, and reviewed through the normal pull-request process. This treats the corpus as source code, which is the correct mental model — it is the single most important asset in the harness.

Appendix: When Not To Build This

A specific caveat for organizations considering this project. Not every organization should build a harness. The baseline prerequisites are worth stating explicitly.

• Your organization has annual AI API spend above roughly five hundred thousand dollars. Below that threshold, the harness investment is hard to justify against the decision impact.
• Your organization has at least one workload where the decision between vendors is genuinely contested — that is, where a harness result could change a real decision. If every decision is already made for non-capability reasons (regulatory, contractual, political), the harness will produce numbers that do not drive decisions.
• Your organization has staff-plus engineers who can own the harness over its operational life. Harnesses without named technical owners decay.
• Your organization has or can build the ground-truth labeling capability. The reference-answer layer is the gating investment.

Organizations missing any of these prerequisites may be better served by starting with vendor-provided evaluation tools, by collaborating with peer organizations that have built harnesses, or by deferring the harness investment until the prerequisites are in place.

For organizations that meet the prerequisites, the harness is close to unavoidable as a 2026 engineering investment. This tutorial should provide a credible starting point.