Research
Benchmarking real-time streaming speech-to-text
Part two measured accuracy on files you upload and wait for. It ended on a thread it did not pull: a product that shows words while the person is still talking does not upload a file, it holds a streaming connection open and feeds audio as it arrives. That is a different API, a different failure mode, and, it turns out, a different accuracy number. This is that measurement.
Two things change when you move from batch to streaming. The engine now has to decide what a word was before it has heard the end of the sentence, which can cost accuracy. And it has to return words fast enough to feel live, which is a latency you cannot read off a batch benchmark. We measured both, on the same two test sets as part two, through each vendor's own streaming API.
The headline: streaming costs accuracy on clean speech and, at best, nothing on hard speech. On clean read audio every engine but one got worse streaming than batch, by 0.16 to 1.38 points. On hard conference call audio no engine got better: four engines tied with their batch score and the rest lost ground. First words arrive anywhere from 0.90s to 6.56s in, and the first word on screen and the first locked word are two different clocks. Two of the ten could not hold up a long sequential run at all: Gladia finished 71 of 500 hard clips, and AssemblyAI needed a retry with backoff to finish.
How we tested
We reused part two's English sets: LibriSpeech test-clean, the easy end of clean read audiobook speech, and Earnings-22, real earnings call audio built as an accent benchmark, the hard end. Streaming runs are slow by design, so we scored a 500-clip sample of each set rather than the full 2,620 and 2,741 utterances part two used. Some engines carry more than 500 clips in the tables below, from earlier runs we kept.
The runner does not hand the engine a whole file. It reads each clip as raw audio, splits it into 100-millisecond chunks, and feeds them to the streaming socket at true wall-clock pace, one chunk every 100 milliseconds, the way a microphone would. This is the part that makes the latency honest. If you push the whole clip at once, every engine looks instant, because you have removed the one constraint a live product actually has. Feeding at 1x also means we cannot run many streams at once per vendor without distorting the timing, so each engine streamed its clips one at a time.
We recorded four timings per clip, defined in the latency section below, and every one of them is measured from the moment the audio starts playing.
The transcript each stream produced was scored with part two's scorer: the OpenAI Whisper English text normalizer then corpus word error rate. Normalize first, then score, the convention the Open ASR Leaderboard uses. Because the scorer is identical to part two's, the streaming WER sits on the same scale as the batch WER, and subtracting one from the other gives the accuracy cost of going live.
The numbers below are matched: we score the batch and the streaming transcript on the exact same clips, the ones the engine streamed, then put a 95% interval on the gap with a paired bootstrap. Resample that shared clip set 1,000 times with a fixed seed, measure the gap between streaming and batch on each resample, and read the difference as real only when the whole interval stays on one side of zero. This is the paired test part two used for its tie claims.
Comparing against part two's batch number for the whole set instead would mislead. The clips we streamed are spaced evenly across each set, which covers every speaker but does not reproduce the set's mix of difficulty. Score the same batch transcripts on only those clips and the number moves.
No streaming in this chart. It is part two's batch transcripts on Earnings-22, scored on all 2,741 clips and then on only the clips the streaming runs used; the bar is how much lower the second score is. Google Chirp is the one engine that moves the other way, by 0.06 points, so it has no bar.
Seven of the eight engines score lower on the sample, so the clips we streamed are easier than the set they came from. The gap reaches 2.07 points on this hard set and 0.24 points on the clean one. None of that has anything to do with streaming. Matching each engine against itself on the same clips removes it, which is why the tables below do that. On the hard set it turns out to be the whole story.
Does streaming cost accuracy?
On clean speech, nearly everywhere. Eight of the nine engines with a batch baseline scored worse streaming than batch by the paired test. Only OpenAI's gpt-4o-transcribe pays no measurable cost: its streaming and batch WER differ by 0.09 points with an interval that spans zero, a tie. Every other engine loses real ground, though for all but Cartesia it stays under three quarters of a point.
| Engine | Streaming WER | Batch WER | Change | 95% CI on change | n |
|---|---|---|---|---|---|
 OpenAI gpt-4o-transcribe | 1.52% | 1.61% | -0.09 | -0.25 to +0.07 | 500 |
 AssemblyAI Universal | 1.74% | 1.55% | +0.19 | +0.09 to +0.31 | 627 |
 Amazon Transcribe | 1.96% | no batch | n/a | n/a | 666 |
 Gladia Solaria | 2.58% | 2.03% | +0.55 | +0.30 to +0.78 | 500 |
 Google Chirp | 2.72% | 2.56% | +0.16 | +0.04 to +0.31 | 500 |
 Speechmatics Enhanced | 2.78% | 2.39% | +0.39 | +0.19 to +0.60 | 500 |
 Azure AI Speech | 2.86% | 2.35% | +0.50 | +0.21 to +0.84 | 500 |
 Soniox | 3.17% | 2.88% | +0.29 | +0.01 to +0.59 | 666 |
 Deepgram Nova-3 | 3.25% | 2.58% | +0.67 | +0.48 to +0.86 | 758 |
 Cartesia Ink-Whisper | 3.43% | 2.04% | +1.38 | +1.01 to +1.85 | 500 |
Scored 2026-07-22. Batch and streaming are scored on the identical clips per engine; the interval is computed before rounding, so it can differ by 0.01 from the rounded columns.
Amazon Transcribe has no batch cell, and the reason is structural rather than a gap in this run. It offers no synchronous endpoint for pre-recorded audio, so part two streamed its clips too, on a different set. There is no batch number to subtract. Its 1.96% streaming result would place it third here on its own terms.
One caveat on that streaming column: the engines carry between 500 and 758 clips, because we kept some longer runs from earlier, so the column is not strictly like for like. Rescoring all ten on the 500 clips they share moves AssemblyAI to 1.82% and Soniox to 3.09%, leaves the other eight within 0.03 points, and changes nobody's rank.
On the hard set the matched comparison changes the answer. Measured against the batch score for the whole set, Deepgram looked 1.41 points better streaming than batch, and OpenAI 1.39. On the same clips they are not. Deepgram's change is -0.93 with an interval from -2.07 to +0.11, which crosses zero, so it is a tie. OpenAI's point estimate flips to +0.68 with an interval from -1.00 to +2.38, also a tie. These two are where the sampling gap above bites hardest, worth 2.07 points for OpenAI and 0.49 for Deepgram, which is enough on its own to turn a tie into an apparent gain. Gladia is not in the table below, for reasons the reliability section covers.
| Engine | Streaming WER | Batch WER | Change | 95% CI on change | n |
|---|---|---|---|---|---|
| Amazon Transcribe | 11.12% | no batch | n/a | n/a | 500 |
| AssemblyAI Universal | 11.23% | 10.26% | +0.97 | +0.47 to +1.58 | 500 |
| Azure AI Speech | 11.34% | 10.47% | +0.87 | +0.12 to +1.65 | 499 |
| Soniox | 11.69% | 11.17% | +0.52 | -0.14 to +1.16 | 500 |
| Google Chirp | 12.60% | 12.73% | -0.14 | -0.63 to +0.36 | 499 |
| Speechmatics Enhanced | 12.60% | 11.90% | +0.70 | +0.37 to +1.08 | 500 |
| Cartesia Ink-Whisper | 14.01% | 11.06% | +2.95 | +2.24 to +3.64 | 500 |
| Deepgram Nova-3 | 14.72% | 15.64% | -0.93 | -2.07 to +0.11 | 500 |
| OpenAI gpt-4o-transcribe | 17.16% | 16.48% | +0.68 | -1.00 to +2.38 | 500 |
Scored 2026-07-22 on matched clips; the change is computed before rounding.
Each row is one engine's streaming WER minus its batch WER on the same clips, so a point right of zero means streaming was worse. Bars are 95% intervals, and a bar crossing zero is a tie.
No engine on the hard set is measurably better streaming than batch. Four are ties: Deepgram, OpenAI, Google at -0.14, and Soniox at +0.52, each with an interval crossing zero. A tie is not proof of equal accuracy, only that any gap is smaller than this test can resolve, and OpenAI's interval still allows a two-point streaming penalty. For a live product a tie is still the good case, no measurable cost for going real time. The other four lost real ground: Speechmatics +0.70, Azure +0.87, AssemblyAI +0.97, and Cartesia +2.95, the largest streaming penalty in either set. AssemblyAI holds near the top of the hard table in absolute terms even so, 10.26% batch against 11.23% streaming, while taking one of the larger hits.
How fast do words appear?
Every engine here sells a real-time API. Fed the same audio at the same pace, the first word on screen arrived anywhere from 0.90s to 6.56s in.
A streaming engine gives you words twice. First it guesses, and it keeps changing its guess as more audio arrives. Then it commits, and the words stop moving. That gives four numbers worth timing, and the table uses them as its columns. Each one is measured from the moment the audio starts playing.
- First partial. When the first words appear on screen, in any form, including a guess the engine may still rewrite.
- First final. When the first words are locked, marked final by the engine and no longer subject to revision.
- Finalization lag. The gap between the audio ending and the transcript settling. This is the tail you feel after you stop talking.
- Revisions. How many times the on-screen partial changed before it settled, counting its first appearance. High counts are the visible flicker where text appears, changes, and changes again.
| Engine | First partial | First final | Finalization lag | Revisions |
|---|---|---|---|---|
| Soniox | 0.90s | 5.40s | 0.12s | 25.8 |
| Amazon Transcribe | 0.92s | 4.65s | 0.26s | 11.6 |
| AssemblyAI Universal | 0.94s | 5.23s | 0.16s | 5.2 |
| Speechmatics Enhanced | 0.96s | 1.25s | 0.36s | 17.5 |
| Deepgram Nova-3 | 0.96s | 2.10s | 0.07s | 5.4 |
| Azure AI Speech | 1.26s | 5.27s | 0.29s | 13.2 |
| Cartesia Ink-Whisper | 3.01s | 3.01s | 0.11s | 0.0 |
| Google Chirp | 5.63s | 6.04s | 0.67s | 0.7 |
| OpenAI gpt-4o-transcribe | 6.56s | 6.86s | 0.69s | 17.6 |
Medians over 500 clips per engine, measured 2026-07-22 from one location. Absolute seconds move with network and region; the gaps between engines are the durable part.
First partials keep a similar order on the clean set, most within a few tenths of a second of the hard set, with Cartesia moving most at about 0.6s. On LibriSpeech the first partial runs from 0.86s for AssemblyAI to 6.54s for OpenAI. First finals move further between the sets, and not all in the same direction: Deepgram's rises 1.14s on clean audio, 3.24s against 2.10s, Azure's 0.90s and Amazon's 0.79s, while AssemblyAI's falls 0.38s. Median clip length is within a tenth of a second across the two sets, so duration is not the explanation.
Five engines show a first partial under a second. Three of them then hold those words unlocked for several more: Soniox, Amazon, and AssemblyAI answer near 0.9s but do not reach a first final until 4.6 to 5.4 seconds. You see words immediately and they are not committed for another four to five seconds.
Speechmatics is the opposite: first partial at 0.96s, first final at 1.25s, so it commits text almost as fast as it shows it. For a caption you want to trust without waiting, that 1.25-second first final is the best number in the table. Deepgram is close behind at 2.10s.
At the other end, Google Chirp and OpenAI's gpt-4o-transcribe do not show a word for more than five seconds, and they get there differently. Google's first partial and first final nearly coincide and its revision count is near zero, 0.7 per clip, which reads as returning the transcript in large blocks rather than word by word. OpenAI does stream partials and updates them heavily, 17.6 per clip, it just does not emit the first one until 6.5 seconds in. Either way, their median first word lands at five to seven seconds, which is not the word-by-word live captioning the fast engines do.
The revision column is the flicker you would see on screen. Cartesia sits at exactly zero because it emits no interim guesses: its first result is already final, which is why its first partial and first final are the same 3.01s. Soniox rewrites its running transcript 25.8 times per clip before it settles. Each trade has a cost you have to design around: 25 rewrites per utterance is visible churn, and zero rewrites means the user stares at nothing for three seconds and then sees a finished line.
The finalization lag, the wait after the speaker stops, is short across the field. Deepgram settles 0.07s after the audio ends, and six of the nine engines land under three tenths of a second. Google and OpenAI trail again near 0.7s. This is the pause before your UI can mark a turn complete and act on it.
Which column is yours depends on what your interface does with unfinished text. If you render partials as they arrive and let them change on screen, first partial is your responsiveness number. First final barely matters then, and what you pay instead is revisions. If you show text only once it is committed, the two swap places. First final becomes your responsiveness number, and the sub-second first partials of Soniox, Amazon, and AssemblyAI buy you nothing, because their committed words are 4.6 to 5.4 seconds in. Read that way, only Speechmatics at 1.25s and Deepgram at 2.10s stay under three seconds. And if something fires when the speaker stops, a translation pass or an agent reply, finalization lag is the number you are waiting on: 0.07s for Deepgram against about 0.7s for Google and OpenAI.
The four columns move independently, so a fast first word tells you little about when the words settle. Five engines reach their first partial within six hundredths of a second of each other, and their first finals run from 1.25s to 5.40s. Cartesia shows the far end of the trade: it never guesses, so it never flickers, and nothing appears for 3.01s. Speechmatics takes the fastest commit in the table and pays 17.5 revisions on the way there. Deepgram is the only engine near the front of all four columns, so the trade is not forced on you.
Reliability is the third axis
Accuracy and speed both assume the run finishes. Two engines failed in ways a batch queue would never have surfaced, so opening and closing hundreds of streaming sessions in a row belongs in the comparison.
AssemblyAI returned "too many concurrent sessions" errors partway through, even though we streamed one clip at a time. It reached full 500-clip coverage only after we added a retry with backoff between sessions. The accuracy and latency numbers above are from the completed run, which we got only by writing that retry ourselves.
Gladia went further. On clean audio it finished cleanly, 500 of 500 clips. On the hard set the endpoint that opens a session would hang or error, and despite the same retry logic only 71 of 500 clips ever completed. We could not get a full Earnings-22 run out of it, so we left it out of the accuracy table for the hard set instead of reporting it at 14% coverage. Its 71-clip numbers are not a result we would stand behind. We cannot tell from outside whether this was a temporary capacity problem or a standing limit, and for a product that has to open a session every time someone starts talking, the distinction matters less than the failure rate.
What is missing, and what we will not claim
- The absolute percentages are the soft part. Matching each engine against itself keeps the sampling out of the deltas, but the absolute WERs rest on 500 clips or so instead of part two's full set. Read the deltas as the result and the percentages as approximate.
- English only, on public sets. Treat a low number on clean speech as a best case, the same caution part two put on its own clean column.
- On-device streaming is not here. The local models from part two run differently under streaming and are a separate measurement.
- Cost is not tabulated. Streaming pricing can differ from the batch rate. Amazon's Transcribe pricing bills streaming at $0.01/min ($0.60/hr) against $0.006/min ($0.36/hr) for batch, checked 2026-07-18. Cartesia's pricing runs real-time speech-to-text at 1 credit per second ($0.18/hr) against 1 credit per 2 seconds for batch ($0.09/hr), checked 2026-07-23. We confirmed only those two, so we leave streaming cost out until we have all ten rates.
If you are choosing a streaming engine, two separate tables matter. Pick for accuracy on audio like yours from the delta on the hard set. Then pick the latency column your interface actually depends on, and read the table on that column alone. An engine that shows its first word at six seconds and one that locks it at one second are not the same product, however close their WER. And run your own clips through it first.