Research

Best LLM for translation: we tested 11 on live speech

Ask a standard translation benchmark which LLM translates best, and it will tell you they are all about the same. On FLORES-200, a set of sentences translated by professionals, the twelve LLM setups we ran scored within 0.021 of each other on COMET, a score trained to predict human judgments. Live speech is different. It arrives as half-finished sentences and misheard words, and on it the same setups ended up almost ten points apart.

We ran this test to choose the model for Verli's multilingual rooms, where one person speaks and every listener reads live captions in their own language. On one host, a model at half the leader's cost came within a point of it. Another finished the speaker's sentences in 15% of its updates. The two best-known translation services split, one close to the top and one below most of the LLMs, and the cost of an hour of captions varied fifteen-fold. Here is what we measured, and the model we run now.

How we tested

We picked seven public talks of 6 to 8 minutes each, all published on YouTube under a CC BY license: six sermons and a lecture. The speakers use English, Spanish, Portuguese, Korean, Chinese and Ukrainian. We translated each talk into three to five languages, 24 pairs. We chose them from the language pairs Verli users ran most in the 90 days to November 1, 2026, plus common church languages.

We streamed each talk through Soniox speech recognition once at normal speed, and we saved every response with its arrival time. We replay that one recording into every setup, so every model sees the same live transcript, with the same half-finished sentences and the same misheard words at the same moments. The replay runs the product's loop on a virtual clock and makes real API calls, so the delay and cost measurements are real. While someone is still speaking a sentence, the loop translates it again about every 1.5 seconds into all the room's languages in one call. When the sentence ends, the loop translates it once more and locks it.

Two AI reviewers scored quality. We ran Claude Opus 5.5 and a GPT model through their command line tools on subscriptions. We cut passages of about 45 seconds at the end of a sentence. For each passage the reviewers saw every setup's final translation side by side under shuffled letters and listed errors in the MQM style that professional translation reviewers use. The score is 100 minus 25 points per critical error, 5 per major and 1 per minor, per 100 source words (estimated from characters for Chinese). AI reviewers are not people. Their two rankings agreed at 0.96 (1 means the same order), so we read differences of a point or less as ties. We ran 100 FLORES sentences per pair through every model and scored them with chrF and COMET against the professional translations.

By our estimate, the AI review of the three LLM rounds would have cost about $130 at API prices, many times what their translation calls cost.

DeepL and Google Cloud Translation are translation services, so we ran them differently from the LLMs. They translated one finished sentence at a time, with one request per language. Re-translating every unfinished sentence would bill each character many times. We gave DeepL the sentence before as context. DeepL does not bill for it. Google's standard model takes no context. Both runs fit inside the free allowances: DeepL's free API plan and Google's first 500,000 characters a month.

The results

Quality is the mean of both reviewers. Relative cost counts billed tokens for the LLMs, characters for DeepL and Google, and audio and text for Soniox. Delay is the time from when a speaker says a word until a translation that covers the word appears on screen. We measured delay over every update. First words is the time from the start of a sentence until any translation of it appears on screen. Sentence locked is the time from the end of a sentence to its final translation. DeepL and Google show nothing until a sentence ends, so we counted their delay from the last word of the sentence. Their delay equals their locked time, and the first words column shows the wait. The Wafer row and the DeepL and Google rows come from later rounds scored the same way. We scored some setups again beside them as anchors, and those anchors landed within 0.9 points of their earlier scores. We scored Qwen 3.8 Flash and Hunyuan-MT2 in the first round only and then dropped them.

Sentence tests can't tell these models apart

On FLORES, eleven of the twelve LLM setups span about 0.012 COMET, and Google Cloud Translation tied the top one. On live speech the LLM setups are 9.8 quality points apart. A sentence benchmark gives every model a clean, complete, well-punctuated sentence. Live speech gives the model fragments.

We looked at one way models handle fragments. While someone is mid-sentence, the right translation of "When Paul writes to the church in Rome, he is not just" is a translation of those words. Some models finish the thought instead. We sampled 20 unfinished updates per setup and counted how often the translation added meaning the speaker had not said yet. Hunyuan-MT2 is a model built only for translation, and it did so in 15% of the updates. GPT-6 Luna and Gemma 4 26B never did so. DeepSeek V4.1 Flash did so in 0 to 5% across three samples. Other setups failed on hard passages in another way. Claude Haiku 5.5 sometimes returned the English untranslated. That is why it scored 84.0 to 86.3 on English into Russian, Polish, French and Persian while it scored above 98 into Arabic, Korean and Chinese.

If you are choosing a model for anything live, test it on your own recorded speech.

The fastest model costs the most

gpt-oss-120b was the fastest LLM in the table. Its median delay was 0.99 seconds, and it locked sentences in 0.91 seconds. It is the most expensive LLM per hour, at about six times the cost of GPT-6 Luna. The model thinks before it answers even at its lowest setting. It writes about 231 tokens per call. Most other models write 79 to 89 tokens. The host bills for every one of those tokens.

Its quality score is 93.8, close to Gemini 2.5 Flash-Lite. Gemini 2.5 Flash-Lite is cheaper and nearly as fast. We sent gpt-oss a prompt with its thinking step already closed to skip the thinking. On a cheaper host that change cut the cost to about an eighth. On Cerebras the model was faster, with a median delay of 0.90 seconds at a little over half the cost. Quality fell 1.6 to 1.8 points, and serious errors rose by about 40%. gpt-oss-20b was the weakest setup overall. It scored 75.4 into Persian and 72.5 from Chinese into Vietnamese.

The host matters as much as the model

Two rows in the table are the same model. DeepSeek V4.1 Flash on the Wafer host had a median delay 0.7 seconds shorter than on Morph and cost about twice as much. Wafer runs a 4-bit version of the model, and we measured no loss of quality. Qwen 3.8 Flash scored 96.3, but the host we used gave it a median delay of 20 seconds, which makes Qwen 3.8 Flash useless for captions. Gemma 4 26B was fast on average. Its host had slow stretches, so sentence locking reached 9.8 seconds at the 90th percentile.

When you buy access to an open model through a marketplace like OpenRouter, the same model name can mean a different delay and cost.

Pick by the speaker's language

DeepSeek V4.1 Flash was the best or close to it on most pairs, but it was weaker when the speaker spoke Korean or Chinese. On the Morph host, Korean into English scored 95.8 and Chinese into Korean 94.5, where GPT-6 Luna scored 99.0 and 98.2. Gemini 2.5 Flash-Lite, the cheapest Google model, dropped to 88.9 on English into Chinese while staying above 95 on most European pairs.

Average scores do not show this. If your speakers mostly use one or two languages, test those pairs before you choose.

DeepL and Google Translate

DeepL and Google Cloud Translation are what most people mean by a translation API, so we put them through the same talks. Neither takes a prompt or translates into several languages in one call. We made one request for each language, ran the requests for a room in parallel, and used finished sentences only. We scored both in one review round, side by side with GPT-6 Luna, DeepSeek V4.1 Flash on Wafer, Gemini 2.5 Flash-Lite and Soniox.

On FLORES, Google Cloud Translation scored 0.898 COMET, tied with Gemini 3.1 Flash-Lite for the top, and DeepL scored 0.897, the same as GPT-6 Luna. The median time was 0.18 seconds per sentence for Google and 0.46 seconds for DeepL.

On live speech, DeepL scored 96.2, against 97.2 for DeepSeek V4.1 Flash and 98.1 for GPT-6 Luna in the same round, with about twice the serious errors of DeepSeek (38 against 18 per 100 passages). The gap between DeepL and DeepSeek is small. The 95% range runs from 0.2 to 1.7 points in DeepSeek's favor. Google scored 92.3, with 79 serious errors per 100 passages. That score is 3.9 points below DeepL (95% range 2.9 to 5.0), 1.8 below Gemini 2.5 Flash-Lite and 2.1 above Soniox.

DeepL beat DeepSeek on Chinese into English and Chinese into Korean, but not on Chinese into Vietnamese. DeepL scored 97.2 into Korean, and DeepSeek scored 95.6. From Spanish, DeepL was weakest. It scored 92.0 into Portuguese, and DeepSeek scored 98.1. Google scored 98.3 from English into Chinese, close to the LLMs, but fell to 82.5 from Chinese into Korean and 83.4 from Chinese into Vietnamese. We gave DeepL the sentence before as context. The standard Google model takes no context. We did not test whether that difference explains the gap.

The quality score does not show the cost of waiting for finished sentences. The first words of each sentence appeared a median 6.1 seconds after the speaker started it with DeepL and 4.0 seconds with Google. The LLMs other than the slow Qwen 3.8 Flash took 1.1 to 3.4 seconds. Once a sentence ended, Google locked the sentence in 0.75 seconds at the median and 1.01 at the 90th percentile. DeepL took 2.05 seconds at the median and 9.75 at the 90th percentile. Our plan caused part of the DeepL delays. On the free DeepL API plan some requests came back as "too many requests". Our client waited and retried, and 61 of 567 calls took longer than 3 seconds.

The bigger difference is cost. At DeepL's rate of $27.50 per million characters beyond a plan's allowance, the room cost about 31 times as much as DeepSeek V4.1 Flash on Wafer. At Google's $20 per million, the room cost about 23 times as much.

Soniox, the fast baseline

Soniox can translate as it recognizes speech, word by word. Its captions appeared in 0.53 seconds and never changed. We cannot match its output word for word to the speech, so our method measures Soniox's delay slightly in its favor. Every LLM setup except the slow Qwen 3.8 Flash rewrote words on screen as the sentence grew. They rewrote 1.1 to 2.5 words per final word. We counted characters for Chinese and Japanese. A rule that shows only the words two consecutive updates share cuts that count to 0.12 to 0.19. That rule adds about 1.4 seconds of delay. We explain the loop and its trade-offs in how we translate live captions.

Soniox scored 91.0, below every LLM setup in the table except gpt-oss-20b, and it translates into one language per stream. One extra language can use the stream the host runs, at little extra cost. Each further language needs its own stream, so a room in three to five languages cost almost four times as much as DeepSeek V4.1 Flash on Wafer, more than most LLM setups.

What we run

Verli's multilingual rooms use DeepSeek V4.1 Flash on Wafer, with the same model on Morph and then Gemma 4 31B as backups. We chose the faster host over the cheaper Morph host because delay matters more than cost to a reader. Wafer costs about the same as GPT-6 Luna and shows the translation a second sooner (1.36 against 2.34 seconds). All three routes run with zero data retention, so no host keeps the text.

If your speakers mostly speak Korean or Chinese, GPT-6 Luna is the better choice on these results. If you need captions in exactly one extra language and speed matters most, Soniox's own translation is the better choice.