Research

Which speech-to-text engine is most accurate in your language?

Every speech-to-text vendor publishes an accuracy number for English. Almost none publish one for Japanese, Arabic, or Hindi, and when they do, each uses its own audio and its own scoring, so the numbers do not compare. If you are building for users who do not speak English, the public record is close to empty.

So we filled it. We sent the same non-English audio through 16 speech-to-text engines, across 10 languages, and scored every transcript the same way. The short version: the engine that wins English is not the engine that wins Japanese, and two names you might not expect lead most of the non-English field.

This is the third part of our speech-to-text series. Part one covered how to choose an API, and part two benchmarked English accuracy on the same audio. This piece does the same for the languages part two left out.

How we tested

We used FLEURS, Google's open multilingual speech dataset (CC-BY), which carries the same sentences read aloud in each of its languages. For every language we took an even sample of 200 clips from the test split and used the raw transcription as the reference text. We sent each clip to every engine that supports the language, then scored the returned transcript against the reference.

The 10 languages are Spanish, French, German, Chinese (Mandarin), Japanese, Korean, Arabic, Hindi, Russian, and Vietnamese. We picked them deliberately. The three European languages are a sanity check against the Open ASR Leaderboard's multilingual track, which scores a handful of engines on German, French, Italian, Spanish, and Portuguese. The other seven, especially Chinese, Japanese, Korean, Arabic, and Hindi, are where the public record runs out.

Two scoring choices matter. First, we score most languages with word error rate (WER), the share of reference words an engine got wrong. For Chinese and Japanese we use character error rate (CER) instead, because those languages do not put spaces between words, so there are no word boundaries to count errors against. This is the standard convention for languages without word separators, which Thennal and colleagues document for Thai and Chinese in their NAACL 2025 paper on multilingual ASR metrics. Do not compare a CER number against a WER number; they are different measurements.

Second, we normalize before scoring, using Whisper's multilingual BasicTextNormalizer rather than the English one part two used. It lowercases, strips punctuation, and standardizes spacing, so an engine is not penalized for writing a comma where the reference has none.

Every number below is corpus WER or CER with a 95% confidence interval from a bootstrap over the clip set (1,000 resamples, fixed seed). We score each engine only on the clips it actually returned, and note coverage where it is below 100%. A clip an engine refused is a coverage gap, not a wrong transcript, so it does not count against the score.

Three of the 19 engines in our test set are absent. Resemble supports English only. Rev and Cartesia we could not get non-English transcripts from within our paid account limits, so they are excluded rather than shown on partial data.

The encoding trap that mis-ranks accented languages

If you benchmark non-English speech-to-text yourself, check the text encoding before you trust any ranking. Some transcript endpoints return UTF-8 without declaring it, and many HTTP clients then decode the response as Latin-1 by default. That silently turns every accented or CJK character into two or three garbage characters, so "Gegenüber" becomes "Gegenüber" and an otherwise accurate engine sinks to the bottom of the table. On Chinese, a full mojibake decode produces a character error rate above 100 percent. We score against the raw transcripts and read them by eye for exactly this reason, because a naive multilingual test will rank several engines dead last for a reason that has nothing to do with accuracy.

The results

The table shows WER for every language except Chinese and Japanese, which are CER. Lower is better. The final column is the mean across all ten languages, which mixes WER and CER and so is only a rough guide to overall standing; the per-language columns are the real comparison.

Engine↕	es↕	fr↕	de↕	zh*↕	ja*↕	ko↕	ar↕	hi↕	ru↕	vi↕	Mean↕
ElevenLabs Scribe	5.29	4.86	4.64	4.17	2.52	7.43	11.88	12.90	6.47	4.91	6.51
Soniox	5.04	5.72	4.41	4.71	2.92	9.62	13.96	6.23	6.52	6.70	6.58
OpenAI gpt-4o-transcribe	4.30	4.69	3.84	4.46	3.66	9.04	14.01	14.39	6.87	5.31	7.06
Fish Audio	5.62	5.76	4.74	4.10	5.96	10.75	16.16	7.75	7.74	7.32	7.59
Google Chirp 2	5.35	6.70	5.12	6.13	7.68	10.96	14.77	6.18	6.92	6.59	7.64
AssemblyAI Universal	5.31	4.56	3.88	6.29	3.93	n/a	17.63	12.55	n/a	9.44	7.95
Amazon Transcribe	6.20	7.91	6.82	6.43	4.20	13.21	14.22	6.46	9.39	9.36	8.42
Azure	5.93	6.11	5.11	4.07	3.75	16.63	13.79	6.42	10.88	12.33	8.50
Speechmatics	5.89	5.99	5.61	7.11	4.89	17.05	12.98	7.20	10.34	9.26	8.63
Lemonfox	5.29	6.15	4.86	5.63	4.47	13.63	16.74	18.77	7.16	10.27	9.30
Inworld STT-1	6.44	5.99	5.85	8.09	10.77	12.63	17.35	8.62	9.12	8.51	9.34
Gladia Solaria	5.46	6.59	5.31	6.49	4.54	12.91	16.71	18.75	7.27	11.23	9.53
Groq Whisper v3	5.68	6.82	5.33	6.01	5.11	13.66	17.14	20.56	7.48	11.30	9.91
Gemini 2.5 Flash	5.48	6.93	5.12	19.44	4.47	9.59	33.06	n/a	6.36	6.30	10.75
Deepgram Nova-3	8.50	9.71	7.91	5.75	5.80	19.00	29.80	18.35	7.29	13.25	12.54
Grok STT	5.85	11.83	6.66	n/a	21.72	12.80	23.52	16.54	9.12	10.14	13.13

FLEURS test split, 200 clips per language, accessed 2026-07-19. Columns zh and ja are character error rate; all others are word error rate. Lower is better.

A few cells need their caveat stated in full rather than buried. Gladia's German number rests on 55 percent coverage, because it returned nothing for nearly half the clips, so read it as provisional. Azure returned 94 to 97 percent of clips in every language, high enough to compare but not complete. AssemblyAI does not offer Korean or Russian, and Grok does not offer Chinese, which is why those cells read n/a. Gemini has no Hindi row because its output there was unusable, which the section below explains.

The two engines that lead outside English

ElevenLabs Scribe has the lowest mean error of any engine here, and it is the clear or joint leader in eight of the ten languages. OpenAI's gpt-4o-transcribe is the other pillar: lowest error on Spanish and German, and in the leading group everywhere except Hindi. Neither is the name most people reach for first when they think about transcription accuracy, and on English (part two) they were strong but not dominant. Outside English, they separate from the pack. The Open ASR Leaderboard's own multilingual table finds the same thing on its five European languages, with ElevenLabs Scribe and AssemblyAI on top, which is reassuring given that we ran different audio and a different sample.

The reshuffle is real and it is specific. Japanese shows it best: the engines that lead there are not the ones that lead the European languages, and two names that are competitive in English fall to the bottom.

On Japanese, ElevenLabs (2.52 CER) and Soniox (2.92) lead, while Google Chirp falls to 7.68 and Grok collapses to 21.72. On Hindi the leaders change again: Google Chirp (6.18) and Soniox (6.23) come first, while ElevenLabs and OpenAI, the overall front-runners, drop to 12.90 and 14.39. An engine's English rank tells you very little about its rank in a given other language.

Clear winners are rarer than the ranking suggests

A ranked table invites you to read the top row as the winner. Usually it is not, because the top two are within noise of each other. Comparing each engine's own confidence interval is a weak test; the interval is wide because clips vary in difficulty. The right test is a paired bootstrap: resample the same clips and measure the difference between two engines on each resample, which cancels the shared difficulty. If that difference interval excludes zero, one engine really is better.

Run that test and only three languages have a statistically clear winner. Spanish goes to OpenAI, Korean to ElevenLabs, and Arabic to ElevenLabs. The other seven are ties at the top, a leading cluster rather than a single winner. French, German, Chinese, Hindi, and Russian each have three engines the test cannot separate. Japanese has two (ElevenLabs and Soniox) and Vietnamese has two (ElevenLabs and OpenAI), and in both cases that pair clearly beats everything below it.

These are genuine ties, not a sample-size problem. The gaps between tied engines are 0.05 to 0.4 points, with intervals straddling zero, and the paired test already controls for clip difficulty. FLEURS test splits run from 382 to 945 clips per language, so there is a ceiling on how much more data could help, and where two engines differ by a tenth of a point, more clips will not manufacture a winner because they are genuinely equal.

Where the familiar names fall apart

Some engines that are competitive in English come undone outside it. Deepgram Nova-3 is last or near last in most languages and reaches 29.80 WER on Arabic, close to a third of every word wrong. Grok does not offer Chinese at all and posts 21.72 CER on Japanese, an outlier by a wide margin.

Gemini 2.5 Flash is the sharpest example, and it is a different kind of failure. Gemini has no dedicated transcription setting; you ask it to transcribe with a text prompt, and it decides what to do. That works acceptably on some languages (6.36 WER on Russian) and badly on others (19.44 CER on Chinese, 33.06 WER on Arabic). On Hindi it produced a 569 percent WER, which means the output was several times longer than the reference. It had returned commentary about the audio instead of a transcript. We excluded that cell rather than score it, but the lesson stands: a general model prompted to transcribe is not the same as a transcription engine, and the gap widens as the language gets further from English.

Arabic is the wall

Every engine is worse at Arabic than at any other language we tested. The best result is ElevenLabs at 11.88 WER, more than double its Spanish number, and most engines sit between 14 and 33. Two documented reasons help explain it. Arabic is morphologically complex, packing grammatical meaning into fused affixes rather than separate words, which the same ASR-metrics paper flags as a hard case. It also splits into many dialects that differ sharply from the written standard, the diglossia problem the Open Universal Arabic ASR Leaderboard was built to study. If your product depends on Arabic, budget for an error rate several times higher than your English baseline, and test on your own audio before committing.

What this means when you pick an engine

Pick per language, not once. The right engine for your Japanese users (ElevenLabs or Soniox) is not the strongest choice for your Hindi users (Google Chirp or Soniox), and the English benchmark that sold you an engine says nothing about either. If you serve more than a few languages, the safe general-purpose choices on this evidence are ElevenLabs and OpenAI, which stay in the leading group almost everywhere, but "almost everywhere" is doing real work in that sentence: check your specific languages against the table above.

We run the same kind of cascade behind Verli's live translation, so these are the tradeoffs we weigh for every language pair we add. The data here is from controlled test clips, not from anyone's calls; your audio, with its own accents and noise and vocabulary, will land somewhere different, and the only number that finally matters is the one you measure on it.