Kosten, Pass und Latenz — offen gemessen.
Anthropic und Mistral über acht Agent-Achsen, n=10 pro Zelle. Harness und jedes Ergebnis-JSON sind öffentlich. Selbst laufen lassen oder auseinandernehmen.
Zuerst lesen
Die Vorbehalte kommen vor den Zahlen.
Wir hosten beide, Anthropic und Mistral, und dieser Benchmark bevorzugt den günstigeren, den wir ebenfalls verkaufen — behandle das mit der angemessenen Skepsis. Die Achsen laufen gegen deterministische Mock-Tools. Das misst Loop-Mechanik und Kosten, nicht echte Tool-Qualität. Pass ist Regex-gepinnt: ein Boden, keine Qualitätsnote. Es gibt keine Quality-Spalte, weil sich ein Cross-Vendor-Quality-Score mit verfügbaren Judges nicht bias-frei machen lässt (warum). Und n=10 heisst Punktschätzungen mit weiten Intervallen — „100 %“ ist 10/10, kein Gesetz.
Die acht Achsen: Multi-Turn-Loops · Sub-Agent-Orchestrierung · Memory-gestütztes Reasoning · Workflow-Komposition · Long-Context-Tool-Nutzung · backtrackende Tool-Chains · Cron-Cold-Starts · Realwelt-Strategie.
Multi-Turn-Loop
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00009 | $0.00044 | 94% | 1.6s | 2.3s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00013 | $0.00066 | 94% | 3.0s | 3.7s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00022 | $0.00087 | 88% | 5.3s | 5.7s |
mistral-large-2512 | pinned | 100% | $0.00096 | $0.00239 | 77% | 12.0s | 14.2s |
anthropic-haiku-4-5 | pinned | 100% | $0.00793 | $0.00793 | 0% | 3.4s | 4.5s |
mistral-medium-2604 | pinned | 100% | $0.01281 | $0.01686 | 30% | 3.0s | 3.3s |
anthropic-sonnet-4-6 | pinned | 100% | $0.01370 | $0.02468 | 82% | 8.3s | 12.1s |
anthropic-opus-4-7 | pinned | 100% | $0.02486 | $0.05377 | 90% | 7.8s | 12.0s |
mistral-large-latest | latest | 80% | $0.00107 | $0.00191 | 57% | 7.7s | 18.6s |
Sub-Agent-Orchestrierung
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00007 | $0.00029 | 91% | 1.5s | 1.7s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00011 | $0.00043 | 89% | 2.7s | 3.8s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00016 | $0.00057 | 87% | 3.4s | 4.7s |
mistral-large-2512 | pinned | 100% | $0.00101 | $0.00172 | 59% | 7.7s | 10.9s |
mistral-medium-2604 | pinned | 100% | $0.00471 | $0.00582 | 31% | 1.6s | 2.3s |
anthropic-haiku-4-5 | pinned | 100% | $0.00588 | $0.00588 | 0% | 3.5s | 6.2s |
anthropic-sonnet-4-6 | pinned | 100% | $0.01014 | $0.01824 | 83% | 8.0s | 10.8s |
anthropic-opus-4-7 | pinned | 100% | $0.02519 | $0.03994 | 77% | 7.3s | 8.2s |
mistral-large-latest | latest | 90% | $0.00090 | $0.00163 | 64% | 4.5s | 19.1s |
Memory-gestütztes Reasoning
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00006 | $0.00034 | 94% | 0.6s | 0.8s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00006 | $0.00034 | 93% | 0.5s | 0.8s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00008 | $0.00045 | 93% | 0.5s | 0.6s |
mistral-large-2512 | pinned | 100% | $0.00048 | $0.00118 | 68% | 8.5s | 9.8s |
mistral-medium-2604 | pinned | 100% | $0.00299 | $0.00367 | 22% | 0.8s | 1.0s |
anthropic-haiku-4-5 | pinned | 100% | $0.00409 | $0.00409 | 0% | 1.9s | 3.9s |
anthropic-sonnet-4-6 | pinned | 100% | $0.00443 | $0.01189 | 95% | 3.6s | 4.9s |
anthropic-opus-4-7 | pinned | 100% | $0.01077 | $0.02645 | 88% | 3.4s | 8.2s |
mistral-large-latest | latest | 70% | $0.00049 | $0.00088 | 52% | 1.5s | 16.3s |
Workflow-Komposition
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00013 | $0.00074 | 95% | 1.5s | 3.6s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00014 | $0.00071 | 93% | 1.6s | 2.0s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00014 | $0.00084 | 95% | 1.8s | 2.2s |
mistral-large-2512 | pinned | 100% | $0.00065 | $0.00145 | 70% | 8.3s | 10.5s |
mistral-medium-2604 | pinned | 100% | $0.00359 | $0.00475 | 34% | 1.1s | 1.2s |
anthropic-haiku-4-5 | pinned | 100% | $0.00609 | $0.00609 | 0% | 3.1s | 5.1s |
anthropic-sonnet-4-6 | pinned | 100% | $0.00787 | $0.02088 | 94% | 6.7s | 8.6s |
anthropic-opus-4-7 | pinned | 100% | $0.02887 | $0.04472 | 87% | 7.0s | 10.9s |
mistral-large-latest | latest | 80% | $0.00043 | $0.00124 | 84% | 2.9s | 17.1s |
Long-Context mit Tools
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00013 | $0.00066 | 89% | 0.5s | 0.7s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00020 | $0.00100 | 89% | 0.6s | 0.7s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00027 | $0.00133 | 89% | 1.0s | 1.7s |
mistral-large-latest | latest | 100% | $0.00219 | $0.00337 | 40% | 1.3s | 1.7s |
mistral-medium-2604 | pinned | 100% | $0.00324 | $0.01033 | 80% | 0.7s | 2.8s |
anthropic-haiku-4-5 | pinned | 100% | $0.00444 | $0.01713 | 96% | 2.6s | 4.9s |
anthropic-sonnet-4-6 | pinned | 100% | $0.01290 | $0.05132 | 98% | 5.0s | 9.5s |
anthropic-opus-4-7 | pinned | 100% | $0.01725 | $0.05638 | 99% | 2.5s | 19.6s |
mistral-large-2512 | pinned | 80% | $0.00092 | $0.00269 | 75% | 1.6s | 16.8s |
Tool-Chain mit Backtrack
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00008 | $0.00047 | 95% | 0.9s | 1.3s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00011 | $0.00071 | 95% | 1.1s | 2.0s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00015 | $0.00095 | 95% | 1.2s | 1.8s |
mistral-large-2512 | pinned | 100% | $0.00060 | $0.00127 | 63% | 7.7s | 9.0s |
mistral-medium-2604 | pinned | 100% | $0.00640 | $0.00760 | 19% | 1.4s | 1.5s |
anthropic-sonnet-4-6 | pinned | 100% | $0.00725 | $0.02536 | 97% | 7.9s | 9.3s |
anthropic-haiku-4-5 | pinned | 100% | $0.00825 | $0.00825 | 0% | 3.7s | 5.1s |
anthropic-opus-4-7 | pinned | 100% | $0.01611 | $0.05407 | 94% | 6.8s | 9.0s |
mistral-large-latest | latest | 80% | $0.00036 | $0.00108 | 80% | 2.3s | 17.2s |
Cron-Task / Cold-Start
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00004 | $0.00023 | 91% | 0.4s | 0.7s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00007 | $0.00035 | 91% | 0.5s | 0.6s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00013 | $0.00046 | 81% | 0.7s | 1.3s |
mistral-large-2512 | pinned | 100% | $0.00053 | $0.00118 | 63% | 8.4s | 8.6s |
mistral-medium-2604 | pinned | 100% | $0.00271 | $0.00368 | 31% | 0.7s | 0.8s |
anthropic-sonnet-4-6 | pinned | 100% | $0.00381 | $0.01163 | 96% | 3.6s | 4.4s |
anthropic-haiku-4-5 | pinned | 100% | $0.00417 | $0.00417 | 0% | 2.2s | 3.9s |
anthropic-opus-4-7 | pinned | 100% | $0.00956 | $0.02526 | 89% | 3.4s | 5.7s |
mistral-large-latest | latest | 70% | $0.00024 | $0.00089 | 85% | 1.4s | 16.4s |
Realwelt-Strategie
| Modell | Tag | Pass | Kosten (warm) | Kosten (kalt) | Cache-Hit | p50 | p95 |
|---|---|---|---|---|---|---|---|
mistral-ministral-3b-2512 | pinned | 100% | $0.00006 | $0.00026 | 92% | 1.0s | 1.8s |
mistral-ministral-8b-2512 | pinned | 100% | $0.00010 | $0.00040 | 92% | 2.0s | 2.8s |
mistral-ministral-14b-2512 | pinned | 100% | $0.00013 | $0.00053 | 92% | 2.4s | 2.8s |
mistral-large-2512 | pinned | 100% | $0.00081 | $0.00148 | 62% | 7.1s | 11.4s |
mistral-medium-2604 | pinned | 100% | $0.00353 | $0.00477 | 37% | 1.3s | 1.5s |
anthropic-haiku-4-5 | pinned | 100% | $0.00490 | $0.00490 | 0% | 3.7s | 6.5s |
anthropic-sonnet-4-6 | pinned | 100% | $0.00832 | $0.01615 | 93% | 9.5s | 11.0s |
anthropic-opus-4-7 | pinned | 100% | $0.01974 | $0.03545 | 88% | 8.6s | 9.8s |
mistral-large-latest | latest | 90% | $0.00077 | $0.00140 | 60% | 4.4s | 19.7s |
Der Interessenkonflikt
Ein Benchmark ist nur so viel wert wie das, was sein Autor durch die Veröffentlichung zu verlieren bereit ist.
Das günstigere Modell gewinnt die meisten dieser Zellen — und wir verkaufen es. Genau deshalb sind Harness, Roh-JSON und auch die verlorenen Läufe öffentlich: damit Sie uns nichts davon glauben müssen. Die vollständige Aufarbeitung, Interessenkonflikt und alles, steht in den Befunden.
Anmerkungen zu den Zahlen
Kosten pro Task. Warm rechnet Cache-Read-Tokens zum publizierten Cache-Read-Tarif; kalt ohne Cache-Rabatt. Gepinnte Snapshots (z.B. mistral-large-2512) sind den -latest-Tags vorzuziehen, die Mistral still rollt. Bewertete Antwort-Qualität wird im Harness gemessen, aber hier nicht publiziert: ein Cross-Vendor-Quality-Score lässt sich mit verfügbaren Judges nicht bias-frei machen; siehe die Befunde.
Set-Bench ist Teil von lynox, einem source-available professionellen Agent (Elastic License v2). Das Model-Routing, das es absichert, ist der Grund, warum wir es gebaut haben. Quellcode auf GitHub.