Engineering
WebSocket vs SSE for live captions: our numbers
A live caption feature has two connections to get right. One runs from your server to the model that translates. The other runs from your server to everyone reading. We measured both while building Verli's multilingual rooms, where one person speaks and every listener reads captions in their own language, and the right choice was different for each.
To the model, streaming the answer cut the wait for its first output by 0.4 to 1.4 seconds on three of the four setups we timed, several times what reusing a connection saved. To the viewers, server-sent events (SSE) delivered updates to 50 viewers with a p95 delay of 36 to 94 ms in three local load tests. In production each stream runs from a Vercel Function, which can stay open for at most 5 minutes, so the server moves every viewer to a fresh stream before then, with no gap on screen. Below are the numbers and how the handover works.
To the model: stream the answer
OpenRouter, Groq, Cerebras and Anthropic offer plain HTTP APIs. You send a request and read the answer, streamed as it is written if you ask for that. Two of the APIs we tested also offer a socket you keep open: OpenAI's Responses API has a WebSocket mode, and Gemini's Live API is a WebSocket.
We timed one caption translation call, a half-finished sentence into Spanish, Korean and Chinese, six times per setup on November 1, 2026, and took the median.
| Setup | New connection | Reused connection | Streamed, first output | Streamed, finished |
|---|---|---|---|---|
| Gemini 2.5 Flash-Lite | 0.81 s | 0.74 s | 0.36 s | 0.64 s |
| GPT-6 Luna, HTTP | 1.98 s | 1.74 s | 0.34 s | 1.48 s |
| GPT-6 Luna, WebSocket mode | not timed | not timed | 0.57 s | 1.29 s |
| gpt-oss-120b on Cerebras | 0.48 s | 0.34 s | 0.31 s | 0.34 s |
| Gemma 4 31B on Friendli | 0.84 s | 1.20 s | 0.18 s | 0.65 s |
Median of 6 calls per cell, run from our development Mac. Without streaming, a call's time is the time to the full answer.
Reusing a connection, which skips the DNS lookup and the TLS handshake, saved 0.07 to 0.24 seconds on three of the four setups. Gemma's host got 0.36 seconds slower on the reused connection, which we read as noise from that host rather than an effect of reuse.
OpenAI's WebSocket mode finished GPT-6 Luna's answer in 1.29 seconds, against 1.48 seconds for the same model streamed over HTTP. Its first output came later, though, at 0.57 seconds against 0.34.
Streaming did more than either. Over HTTP, the first streamed output of each setup arrived in 0.18 to 0.36 seconds, against 0.34 to 1.74 seconds for the whole answer on a reused connection. That cut the wait by 0.38 seconds on Gemini, 1.40 on GPT-6 Luna and 1.02 on Gemma. gpt-oss-120b on Cerebras was already fast and gained only 0.03 seconds.
"First output" is the first piece of streamed text, not a readable caption in every language. Our prompt asks for one JSON answer with all the room's languages, so they arrive one after another. Streaming would let each language show as soon as its text is complete. Our benchmark replays did not stream, so the delays in our LLM translation benchmark are upper bounds for the slower models. Verli's rooms do not stream the answer yet either.
To the viewers: one way, to many
Viewers in a room only listen. Their phones receive captions and send only a few small requests back: picking an extra language sends one, repeated every 30 seconds while it stays picked. That is the case SSE was made for. The browser's built-in EventSource opens one long HTTP response and reads events from it as the server writes them.
WebSockets now run on Vercel Functions too. Vercel's guide says WebSocket support is in public beta on all plans. The same guide recommends SSE when data only flows from the server to the client. It also gives the reason that mattered to us. EventSource reconnects by itself after a drop, while WebSocket reconnection and catching up on missed messages are code you write.
Either way, the connection lives inside a function, so it ends when the function hits its time limit. Vercel's function limits page gives 300 seconds by default and up to 800 seconds on Pro, with an 1,800-second limit in beta for some runtimes. Every client has to reconnect regularly, whichever protocol you pick. We accept that because the stream then ships with the rest of the app, and there is no separate server to run, watch or scale.
What it took to make SSE reliable
Our viewer stream is a single API route. These are the parts that took work:
- One database watcher per server process. Every accepted update from the speaker is written to MongoDB, and a single change stream per process feeds all the viewers connected to it. A second watcher carries translated captions. In normal delivery nothing polls the database per viewer; only a viewer recovering from an outage fetches a fresh copy, at most every 15 seconds.
- A keepalive every second. If a viewer hears nothing for three seconds, it opens a second stream in the background and keeps the first one until the new one has caught up. This catches a connection that stays open while bytes stop arriving.
- Leaving before the deadline. Our route's limit is 300 seconds. Between 210 and 225 seconds, the server tells each viewer to open a replacement, spread out so the whole room does not reconnect at once. Any stream still open closes itself at 270 seconds.
- Catching up on reconnect. Every new stream starts with the full current transcript, then live events. A viewer that missed something while reconnecting never needs the server to remember what it sent before.
Browsers allow only six SSE connections per domain over HTTP/1.1, across all tabs. Over HTTP/2 the limit is negotiated and defaults to 100 streams, according to MDN's EventSource page. verli.app answers over HTTP/2, so a viewer with several rooms open in tabs does not hit the low limit.
The load tests
We test the stream with two real server processes and a disposable MongoDB replica set on one machine. The speaker writes to one process, and 50 viewers read from the other, through the same client code as the hosted viewer page.
| Test | Duration | Updates checked | Delivery p95 | Notes |
|---|---|---|---|---|
| Normal handover | 300 s at 4 updates/s | 60,000 | 94 ms | all 50 viewers switched to a new stream once |
| Slow handover | 300 s at 4 updates/s | 60,000 | 93 ms | each new stream held back 13 seconds on purpose; slowest update 904 ms |
| With the caption watcher | 60 s at 4 updates/s | 12,000 | 36 ms | no handover in this run; p50 19 ms, slowest update 88 ms |
Local runs on one machine, not production. Updates checked counts each update for each of the 50 viewers, including updates covered by a catch-up copy.
In the normal handover test, 59,750 updates arrived as live events and 250 were covered by the catch-up after the switch, which is how the design is meant to recover. The caption watcher test checks that a second database watcher, the one for translated captions, does not slow delivery. It timed transcript updates. It also sent a translated caption and checked that every viewer received it, but did not time it. None of these runs cover real phones on mobile networks or a whole day of streaming.
Where we do use a WebSocket
The speaker's app sends its audio to Soniox speech recognition over a WebSocket, straight from the device. That is a continuous stream from the client, which is the case Vercel's guide gives for WebSockets. Everything after it, from our server to the model and from our server to the room, is plain HTTP.