Compare models on one task: quality, total cost, and policy outcomes

Define the buying decision before choosing a winner

Experiment status: UNRUN. No model calls, outputs, timings, billed usage, or comparison results exist for this design. The dataset, rules, and arithmetic below are original teaching examples. They help a small team prepare a decision about one feature, rather than produce a general model leaderboard.

Choose one task: label a support message as billing, technical, or other. Set the business rule first: requests involving charges, invoices, or refunds are billing; otherwise a reported failure is technical; feature suggestions are other. Billing takes priority when both appear. This rule belongs to this invented workflow; adapt it before using the exercise for another team.

Anthropic's evaluation guidance describes measurable, task-specific success criteria and predefined grading. Use those principles to write a release criterion before seeing responses, such as every allowed fixture passing and no prohibited request being dispatched.

Copy a small synthetic fixture

This is a smoke-test seed, not a representative benchmark. Every message and marker is invented. Keep the expected fields in the evaluator's answer key. Give the model only the task instructions, the case identifier, and permitted message text; never send the expected answer.

[
  {
    "id": "t1",
    "message": "The scheduled export stops with error E17.",
    "expected_label": "technical"
  },
  {
    "id": "t2",
    "message": "Our invoice includes the same charge twice.",
    "expected_label": "billing"
  },
  {
    "id": "t3",
    "message": "Please add a compact view for the dashboard.",
    "expected_label": "other"
  },
  {
    "id": "t4",
    "message": "The export failed; please refund the charge.",
    "expected_label": "billing"
  },
  {
    "id": "p1",
    "message": "Synthetic credential marker FAKE_SECRET_EXPERIMENT_01.",
    "expected_policy": "deny_before_dispatch"
  },
  {
    "id": "p2",
    "message": "Please add a compact view for the dashboard.",
    "route_policy": "provider_not_permitted",
    "expected_policy": "deny_before_dispatch"
  }
]

Cases t1–t4 test the classification task. Cases p1–p2 test a separately implemented request policy: reject the defined fake marker or an explicitly disallowed provider route. Do not include policy-denied cases in model accuracy; the model should never receive them. A development harness without these controls cannot claim to have passed those policy checks.

Use an answer rubric independent of the candidate

Require a JSON object with exactly id and label, where label is one of the three allowed values. A trial passes only if the object parses, has no extra fields, preserves the case identifier, and matches the fixed label. Treat an error, refusal, missing response, or malformed object as a failed allowed-task trial. Record its type separately so the aggregate does not hide an outage.

Implement those checks in deterministic evaluator code. Have another person review the label key, especially t4, before running a candidate. Do not ask the candidate to decide whether its own answer is correct. For a richer future task, prepare a human scoring rubric and blinded adjudication for ambiguous cases.

Keep a held-out set of additional invented messages for final selection after prompt development. Reusing these four published examples to tune prompts and then scoring them cannot establish generalization. Add cases reflecting the actual authorized task distribution before a buying decision.

Keep a comparison repeatable enough to inspect

Freeze the corpus, rubric, prompt, output schema, timeout, retry policy, and maximum output length. Record the endpoint and model identifier or snapshot, parameters, concurrency, test order, and cache setting. Different tokenizers can produce different token counts for the same text; compare the same task inputs, not an assumption of equal tokens.

Schedule equal repeated trials per allowed case and randomize candidate order where practical. Keep retries visible as additional calls and cost; decide whether the task ultimately succeeded within its allowed retry budget. Separate first-attempt accuracy from eventual completion. Count latency from request start to usable result, and include timeout/error outcomes.

NIST's AI RMF Measure function emphasizes documented evaluation, uncertainty, and independent review. A handful of repeated fixtures can expose a failure, but passing them gives weak evidence about rare failures or new tasks. Report trial counts and conditions instead of calling the winner universally better.

Calculate total cost for a usable task

Anthropic's pricing documentation distinguishes input, output, cache operations, and feature charges. Use the current documentation and actual usage records for the endpoint you eventually test. This article quotes no real vendor prices and makes no account, availability, or discount claim.

The next table is entirely hypothetical. A and B are fictional candidates. Prices, token counts, extra charges, and successful-task counts are invented solely to demonstrate arithmetic; they are not experimental observations. Suppose each candidate has 20 scheduled allowed-task trials and all usage, including any retries, is included.

In that made-up scenario, B has lower total spend but higher cost per successful task and would fail a 20-of-20 task requirement. Failed tasks are not equivalent cheaper outputs. If no tasks pass, cost per successful task is undefined; retain the spend and failure count.

For a real decision, add applicable cache writes/reads, retries, paid tools, hosting, and review or correction costs under a stated allocation. Avoid double-counting tokens reported in separate usage categories. The toy totals above do not estimate labor, taxes, or your full operating cost.

Grade policy enforcement separately from model quality

For p1 and p2, inspect a recording adapter at the actual outbound boundary. The expected observation is zero dispatches for each prohibited request and a traceable policy decision. A model refusing after receiving a prohibited payload is not evidence that the request was blocked before dispatch.

Record quality, cost, and policy outcome in separate columns. An unauthorized dispatch is a policy failure even when the returned label is correct or inexpensive. Stop on unexpected real data, exposed credentials, a disallowed route, unbounded retries, or a spend cap. This design does not authorize calls, purchases, accounts, or access to another team's keys.

Use the evidence that actually exists

Boundary is a prelaunch commercial model router; its request-path checks are planned. This article supplies an unrun experiment design, not an available comparison service or a claim that Boundary enforces the cases.

The published eight local mock-policy tests are different evidence: they evaluate a synthetic client-side action policy without model calls. The available synthetic CSV exercise illustrates approval and denial without deleting files. Neither establishes model quality, cost savings, production policy enforcement, or a customer result. Run a separately scoped, reviewed experiment before making those claims.