Set-Bench: an agent benchmark for Anthropic and Mistral models

An agent loop re-sends its whole context every turn — system prompt, tool schemas, the growing message history. So what it costs to run one is dominated by two levers almost no model leaderboard measures: which model you picked, and whether its prefix caches. Across eight production-shaped agent axes (multi-turn loops, sub-agent orchestration, memory-grounded reasoning, workflow composition, long-context tool use, back-tracking tool-chains, cron cold-starts, real-world strategy), n=10 each, the spread is brutal: a small Mistral drives the loop to a correct terminal state for $0.00005–$0.0002 per task, where Haiku costs ~$0.006: a 30–100× gap.

Disclosure first: lynox sells managed hosting for both Anthropic and Mistral, so a benchmark from us that makes the cheaper one we also sell (Mistral) look good is suspect by default. Fair. So the numbers are public (source-available, ELv2; every result JSON in the repo, link at the end), the methodology’s weak spots are below, and I’m not claiming Mistral is better. Only, specifically, cheaper to run these loops. (Scope is Anthropic and Mistral, the two providers we’ve parity-verified across all eight axes; OpenAI and Gemini adapters exist, but I won’t publish numbers I haven’t parity-checked.) Try to break it.

Two levers that decide agent-loop cost

Lever 1 — model base price, and it dominates. Before any caching, ministral-3b-2512 runs these loops at $0.00043 cold vs Haiku’s $0.0073: 17× cheaper just on list price. Caching is the multiplier on top; the base model is most of the bill. “Anthropic-quality, Mistral-cost” starts to look structural for loop-shaped work, provided you only need the task done, not the prettiest answer (see caveats).

Lever 2 — caching, and not every model caches. In a loop the re-sent prefix is most of the bill, so prompt caching collapses it: the models that cache run 2–5× cheaper warm than cold (ministral 3b/8b/14b ~5×, Sonnet/Opus ~2.2–2.6×). The non-obvious part: Claude Haiku 4.5 didn’t cache the few-thousand-token prefixes these turns use. A direct API probe settles it. An identical 2.9K-token cache_control block cached fully on Sonnet (2863 tokens read on the second call) and 0 on Haiku; only our 14K-token long-context axis ever crossed Haiku’s (much higher) cache minimum. So Haiku pays full freight every turn: a model that doesn’t cache is structurally expensive in a loop, on top of its base price. (Its warm≈cold number in the table is correct, not a harness miss.)

Where this is weak: read before trusting it

The tools are mocks. The loops run against deterministic stubs: web_search returns canned hits, spawn_agent fixed output. A deliberate trade for reproducibility (real tools = network flake, drift, a bill every run), but it means this measures loop mechanics and cost, not real-world tool quality. “Small Mistral keeps up” holds against predictable tool outputs; messy real-world outputs are a harder test, and this harness doesn’t prove parity there.

Pass/fail is a floor, not a quality score. A cell passes if its output matches a regex-pinned ground truth we wrote; it completed the task, not that the answer is as good as Opus’s. Most cells pass 100%, which hides the dimension where frontier models likely pull ahead. So the cost claim above is “cheaper to clear the same bar,” not “equal quality.”

We won’t publish a cross-vendor quality score. The fix for the above is an LLM judge, but a single judge has self-preference (an Anthropic judge rates Anthropic prose higher). We ran a two-vendor panel on harder open-ended axes: the Anthropic judge (Sonnet) rated Anthropic answers well above Mistral; the Mistral judge rated everyone top marks and couldn’t discriminate. No agreement, so no quality column, only cost/pass/latency/cache. Note which way that cuts: the only judge that discriminated favored Anthropic, so withholding quality withholds a number that would flatter the frontier models, not Mistral.

And: the harness holds twelve axes; we tabulate eight. The other four are deliberately withheld. Two are closed reasoning puzzles that chain-of-thought equalises (a 3B model solves them once it shows work, so they ceiling and don’t discriminate), and two are open-ended design axes scored by an LLM judge we couldn’t make bias-free (the quality column we won’t publish, above). The eight here were fixed before any runs, not picked for flattering results. temperature is pinned to 0.7 for every cell except Opus (which rejects the param). n=10 → point estimates with wide intervals; “100%” is 10/10, not a law.

Two within-Mistral findings (where the judge is fair)

mistral-medium-2604 (the nominal Magistral successor) matched or trailed mistral-large-2512 on every axis but one: no tier-justifying win at ~6× the cost. ministral-14b-2512 passed all eight axes at ~5× lower cost (warm). And pin your snapshots: mistral-large-latest averaged 83% pass vs the pinned -2512’s 98% (n=10; it did win long-context, 100% vs 80%), consistent with Mistral shipping fixes through -latest first.

Why we built it

Not to crown a winner, but to make our own model-routing defensible, and to have a regression gate open enough that you’d catch a snapshot regressing before we do. The interesting layer in agent runtimes is which model for which task. The harness, scenarios, and every result JSON are in the repo (source-available, ELv2). If a scenario looks unrepresentative or a pass-check too lax, that’s a fair PR; tell me where it’s wrong.