Benchmarks

Transcription is solved.
Knowing who spoke is not.

Four systems, 29 public recordings, one scorer applied to all of them.

We are building a search engine for audio, so the number we optimize is not how many words a model gets right. It is whether the words end up in the right speaker's mouth. Our numbers come from the production pipeline rather than a lab build, we publish every column including the ones we lose, and every claim of ours that died getting here is at the bottom of this page. A benchmark you can't attack isn't one you should trust.

The full grid

Three domains, three systems, every scoring setting.

These are the industry's yardsticks rather than ours, and we publish them so you can compare us on your terms. DER is diarization error, who spoke when; its confusion component is the wrong-mouth rate, words put in the wrong person's mouth. WER is word error, text only. Speech regions are rebuilt from each system's own word timings under one shared rule, scored at every merge-gap setting with a ±0.25s collar and overlap excluded. Every figure is an error rate, so lower is better, and the best value in each column is bold, whoever earned it.

Wrong-mouth rate: words attributed to the wrong speaker Confusion component of DER at merge gap 0.3s. It moves less than a third of a point across the full sweep for every system. 0 5 10 15 Meetings Audioscrape Audioscrape, Meetings: 1.11% 1.11 Deepgram Deepgram, Meetings: 2.70% 2.70 AssemblyAI AssemblyAI, Meetings: 1.04% 1.04 Business calls Audioscrape Audioscrape, Business calls: 1.57% 1.57 Deepgram Deepgram, Business calls: 3.85% 3.85 AssemblyAI AssemblyAI, Business calls: 11.91% 11.91 In the wild Audioscrape Audioscrape, In the wild: 1.81% 1.81 Deepgram Deepgram, In the wild: 1.11% 1.11 AssemblyAI AssemblyAI, In the wild: 2.39% 2.39 Lower is better. Share of scored speech time, %.
Wrong-mouth rate: words attributed to the wrong speaker Confusion component of DER at merge gap 0.3s. 0 5 10 15 Meetings Audioscrape Audioscrape, Meetings: 1.11% 1.11 Deepgram Deepgram, Meetings: 2.70% 2.70 AssemblyAI AssemblyAI, Meetings: 1.04% 1.04 Business calls Audioscrape Audioscrape, Business calls: 1.57% 1.57 Deepgram Deepgram, Business calls: 3.85% 3.85 AssemblyAI AssemblyAI, Business calls: 11.91% 11.91 In the wild Audioscrape Audioscrape, In the wild: 1.81% 1.81 Deepgram Deepgram, In the wild: 1.11% 1.11 AssemblyAI AssemblyAI, In the wild: 2.39% 2.39 Lower is better. Share of scored speech time, %.
Overall diarization error, across the whole merge-gap sweep Every setting of the one free scoring parameter, not just the one that flatters us. AudioscrapeDeepgramAssemblyAI 0 10 20 30 40 50 Audioscrape, Meetings at gap 0.0s: 24.94% Audioscrape, Meetings at gap 0.3s: 15.98% Audioscrape, Meetings at gap 0.5s: 12.66% 12.7 Deepgram, Meetings at gap 0.0s: 20.33% Deepgram, Meetings at gap 0.3s: 18.74% Deepgram, Meetings at gap 0.5s: 16.61% 16.6 AssemblyAI, Meetings at gap 0.0s: 40.41% AssemblyAI, Meetings at gap 0.3s: 32.19% AssemblyAI, Meetings at gap 0.5s: 27.97% 28.0 0.0 0.3 0.5 Meetings Audioscrape, Business calls at gap 0.0s: 12.09% Audioscrape, Business calls at gap 0.3s: 4.55% Audioscrape, Business calls at gap 0.5s: 3.87% 3.9 Deepgram, Business calls at gap 0.0s: 10.30% Deepgram, Business calls at gap 0.3s: 9.71% Deepgram, Business calls at gap 0.5s: 9.54% 9.5 AssemblyAI, Business calls at gap 0.0s: 28.97% AssemblyAI, Business calls at gap 0.3s: 20.58% AssemblyAI, Business calls at gap 0.5s: 18.61% 18.6 0.0 0.3 0.5 Business calls Audioscrape, In the wild at gap 0.0s: 16.51% Audioscrape, In the wild at gap 0.3s: 8.82% Audioscrape, In the wild at gap 0.5s: 7.41% 7.4 Deepgram, In the wild at gap 0.0s: 7.10% Deepgram, In the wild at gap 0.3s: 5.21% Deepgram, In the wild at gap 0.5s: 3.58% 3.6 AssemblyAI, In the wild at gap 0.0s: 25.87% AssemblyAI, In the wild at gap 0.3s: 17.17% AssemblyAI, In the wild at gap 0.5s: 11.77% 11.8 0.0 0.3 0.5 In the wild Lower is better. Merge gap in seconds across each panel; share of scored speech time, %.
Overall diarization error, across the merge-gap sweep Every setting of the one free scoring parameter. Audioscrape Deepgram AssemblyAI Meetings 0 10 20 30 40 50 Audioscrape, Meetings: 24.94% Audioscrape, Meetings: 15.98% Audioscrape, Meetings: 12.66% 12.7 Deepgram, Meetings: 20.33% Deepgram, Meetings: 18.74% Deepgram, Meetings: 16.61% 16.6 AssemblyAI, Meetings: 40.41% AssemblyAI, Meetings: 32.19% AssemblyAI, Meetings: 27.97% 28.0 0.0 0.3 0.5 Business calls 0 10 20 30 40 50 Audioscrape, Business calls: 12.09% Audioscrape, Business calls: 4.55% Audioscrape, Business calls: 3.87% 3.9 Deepgram, Business calls: 10.30% Deepgram, Business calls: 9.71% Deepgram, Business calls: 9.54% 9.5 AssemblyAI, Business calls: 28.97% AssemblyAI, Business calls: 20.58% AssemblyAI, Business calls: 18.61% 18.6 0.0 0.3 0.5 In the wild 0 10 20 30 40 50 Audioscrape, In the wild: 16.51% Audioscrape, In the wild: 8.82% Audioscrape, In the wild: 7.41% 7.4 Deepgram, In the wild: 7.10% Deepgram, In the wild: 5.21% Deepgram, In the wild: 3.58% 3.6 AssemblyAI, In the wild: 25.87% AssemblyAI, In the wild: 17.17% AssemblyAI, In the wild: 11.77% 11.8 0.0 0.3 0.5 Lower is better. Merge gap in seconds; share of speech time, %.
Word error rate: raw transcription accuracy In-the-wild clips ship diarization references only, so they carry no WER. 0 10 20 30 Meetings Audioscrape Audioscrape, Meetings: 21.27% 21.27 Deepgram Deepgram, Meetings: 22.22% 22.22 AssemblyAI AssemblyAI, Meetings: 25.54% 25.54 Business calls Audioscrape Audioscrape, Business calls: 13.53% 13.53 Deepgram Deepgram, Business calls: 17.04% 17.04 AssemblyAI AssemblyAI, Business calls: 12.39% 12.39 Lower is better. %.
Word error rate: raw transcription accuracy In-the-wild clips ship diarization references only, so no WER. 0 10 20 30 Meetings Audioscrape Audioscrape, Meetings: 21.27% 21.27 Deepgram Deepgram, Meetings: 22.22% 22.22 AssemblyAI AssemblyAI, Meetings: 25.54% 25.54 Business calls Audioscrape Audioscrape, Business calls: 13.53% 13.53 Deepgram Deepgram, Business calls: 17.04% 17.04 AssemblyAI AssemblyAI, Business calls: 12.39% 12.39 Lower is better. %.

Charts read the same data as the tables below; the tables carry the exact values and the per-setting detail.

Meetings: AMI test set, 8 meetings, 4-way overlapped (hardest condition)

system WER DER at merge gap confusion at merge gap
0s 0.3s 0.5s 0s 0.3s 0.5s
Audioscrape21.27%24.94%15.98%12.66%0.96%1.11%1.15%
Deepgram nova-322.22%20.33%18.74%16.61%2.64%2.70%2.76%
AssemblyAI universal-3-5-pro25.54%40.41%32.19%27.97%0.99%1.04%1.09%

Our wrong-mouth rate is ~2.4× lower than Deepgram's at every setting; AssemblyAI edges us on confusion at the two looser settings while trailing far behind on overall DER. We now also lead overall DER at 0.3s and 0.5s, and word accuracy by about a point. We do not claim the word-accuracy lead: our own three runs of this corpus scored 23.3%, 22.3% and 21.3%, a spread wider than the margin. Treat it as a tie and read the attribution columns instead.

Business calls: Earnings-21 eval10, 11 calls, 5–15 speakers

system WER DER at merge gap confusion at merge gap
0s 0.3s 0.5s 0s 0.3s 0.5s
Audioscrape13.53%12.09%4.55%3.87%1.39%1.57%1.60%
Deepgram nova-317.04%10.30%9.71%9.54%3.80%3.85%3.89%
AssemblyAI universal-3-5-pro12.39%28.97%20.58%18.61%10.54%11.91%12.34%

AssemblyAI has the best raw transcription here and we are about a point behind it. Deepgram holds overall DER at the tightest setting, where sparse word timings penalize us most, but we take the other two settings by roughly half and the wrong-mouth rate at all three. These calls run to 15 speakers; attribution is where the systems separate.

In the wild: VoxConverse test sample, 8 English clips

system DER at merge gap confusion at merge gap
0s 0.3s 0.5s 0s 0.3s 0.5s
Audioscrape16.51%8.82%7.41%1.68%1.81%1.84%
Deepgram nova-37.10%5.21%3.58%1.09%1.11%1.12%
AssemblyAI universal-3-5-pro25.87%17.17%11.77%2.16%2.39%2.57%

Deepgram wins this domain outright, on every column and at every setting. Two of the ten clips are German and are excluded, because the Deepgram model we called is English-only and scoring them would measure a language gap rather than diarization; that exclusion is what turned our apparent win here into a loss, and the graveyard below has the story. We lead AssemblyAI at the three settings shown, and lose to it past a 1.0s merge. These are short recordings where re-runs move DER by a point or two, so read sub-point gaps as ties.

And the obvious question

What about just using OpenAI?

We ran the same fixtures through OpenAI's newest transcription model, gpt-transcribe, in August 2026. On business calls it is the most accurate transcriber in this comparison, at 11.9% word error against AssemblyAI's 12.4% and our 13.5%. On meeting audio the same model scores 45.5%, roughly twice everyone else, and 92% of that error is words it never returned rather than words it got wrong.

The more useful finding is not a number. OpenAI's newest transcription model does not label speakers at all, and the only model of theirs that does is fourteen months older and refuses any recording longer than 23 minutes. On the 13 fixtures short enough for it to accept, its wrong-mouth rate is 19.4% against our 1.8%; on one four-person meeting it reported sixteen speakers.

Worth knowing if you are building on that API: hand it a long recording in one request and it returns a fraction of the audio, with no error and full billing. We chunk at ten minutes for that reason.

How to read it

What we claim, and what we don't.

  • We claim, on meetings and business calls: better overall diarization than AssemblyAI at every setting we swept (0 to 2.0s, ten cells out of ten), and a wrong-mouth rate 2 to 2.4x lower than Deepgram at every setting. These are the two domains our customers record.
  • We do not claim word accuracy, including the lead this page now shows. We are about a point ahead of Deepgram on meetings, but three runs of our own pipeline on this corpus scored 23.3%, 22.3% and 21.3%. A lead smaller than your own run-to-run spread is not a lead.
  • We do not measure retrieval itself. If the goal is a searchable archive, the honest gap in this page is that every number on it is transcription or attribution, not search quality. A retrieval benchmark is owed.
  • We do not claim in-the-wild audio, in either direction. Deepgram beats us there on every column at every setting. And although we lead AssemblyAI across the three settings printed above, AssemblyAI passes us once the merge rule opens past 1.0s. Both facts are in the table.
The graveyard

Claims we measured, then killed ourselves.

August 8 — "2 to 3x better diarization error than Deepgram." Died when we re-scored from word timings instead of vendor utterances. Utterance-level DER rewards transcript formatting, not diarization: one vendor's utterances claimed 15.9 minutes of speech inside a 13.1-minute meeting. Scored fairly, the lead did not survive the parameter range.

August 10 — "We beat both vendors on word accuracy." True on the first meeting we measured, dead as soon as we expanded to 29 recordings. A benchmark of one recording is an anecdote. The interesting part is what happened next: after two diarization fixes shipped, the August 14 run put us back in front on meeting word accuracy. We still do not claim it, for the reason given above. A number has to beat your own noise before it beats a competitor.

August 14 — "Better diarization than AssemblyAI at every setting." True at every setting we published, which turned out to be the problem. Our tables sweep the merge rule from 0 to 0.5s; sweeping further, AssemblyAI passes us on in-the-wild audio at 1.0s. The claim was resting on where we chose to stop. It is now scoped to the two domains that survive the full sweep.

August 14 — "The best in-the-wild diarization of the three." Killed by our own data, days before publishing. Two of the ten VoxConverse clips are German, and the Deepgram model we called is English-only, so it returned one and five words for them. Our scorer treated an empty transcript as an ordinary hypothesis and charged Deepgram a ~100% miss for a language gap rather than for diarization. Nothing errored; the number simply looked like a win. Removing those two clips moved Deepgram from 24.1% to 5.2% and turned our win into a clear loss. The harness now fails whenever any system returns under 30% of what the others returned on the same recording, because an empty answer scores badly rather than loudly.

We keep this section here because it is the test you should run on every vendor benchmark, including this page: what did they measure and decide not to show you?

Method

How we measured it.

  • Fixtures: public corpora only: 8 AMI meetings (CC-BY, test split), the 11-call Earnings-21 eval10 subset (CC-BY-SA), 10 VoxConverse test clips sampled one per duration decile (CC-BY), of which the 8 English ones are scored. The 2 German clips are excluded because one vendor's model is English-only, and scoring them would measure a language gap rather than diarization. No customer audio is ever sent to third-party APIs.
  • Identical audio: every system fetched the same file. Our numbers come from the production pipeline, not a tuned lab configuration.
  • Identical scorers: pyannote.metrics for diarization (±0.25s collar, overlap-excluded), jiwer for WER, one text normalization for every system.
  • Fair segmentation: speech regions rebuilt from each system's word timings under one documented merge rule, reported across the parameter range, because a lead that exists at one setting only is not a lead. The tables stop at 0.5s deliberately: past roughly that point the rule stops bridging pauses inside a phrase and starts asserting speech across real inter-turn silence, which inflates false alarm rather than measuring diarization (our own earnings DER gets worse beyond 1.0s for exactly that reason). We swept to 2.0s anyway to check that nothing flips. One thing does, and it is called out in the in-the-wild note.
  • Versions and dates: Deepgram nova-3 and AssemblyAI universal-3-5-pro (their strongest tiers), measured 2026-08-10. Our column was re-measured on 2026-08-14 against the build then in production, after two diarization fixes shipped; the audio, references and scorers are byte-identical, so the vendor columns were not re-run. That asymmetry is real and worth stating plainly: our side moved and theirs did not. Vendors update models silently, so numbers older than a quarter should be re-run, ours included. We re-ran all 27 fixtures through production again on 2026-08-25, after a diarization change that shipped on 2026-08-23: every cell in our column reproduced to the decimal and the transcripts came back byte-identical, so the 2026-08-14 numbers stand as measured and confirmed.
  • The harness: the adapters, the scorers, the merge-rule sweep and the reference builders are a working repository, and we send it on request along with an evaluation workspace so you can run your own audio through the same scoring. We do not publish it openly for one honest reason: our column is scored from the production pipeline's word timings, which our public API does not return today, so a public copy could benchmark the other three systems and not us. A reproduction that cannot check the system publishing it is not worth much.
  • Known limits: English-language fixtures; AMI is near worst-case overlap; reference conventions differ per corpus, which is why results are per-domain, never one blended headline.
The benchmark that matters

The only benchmark that decides anything is yours.

We provision an evaluation workspace with your recordings, your language and your speakers, and you score it with your own reviewers. If the answer is that someone else fits your audio better, that is a useful afternoon for both of us.

Set up an evaluation Try the API