Speech-to-Text Benchmark Results - Word Error Rate (WER %, lower is better)
Methodology: benchmarks run on the same framework as the Solaria-1 campaign. Solaria-3 and Solaria-1 compared against 8 speech-to-text providers. Each provider tested on identical audio files via production APIs with default settings. Real customer audio = Gladia internal production dataset, annotated by humans. Soniox and Pipecat STT Benchmark excluded on some datasets pending data availability. Transcriptions normalized with Gladia's open-source normalization library (https://github.com/gladiaio/normalization), based on the OpenAI Whisper text normalizer and extended to cover more languages, before WER computation. Diarization Error Rate (DER) measured on DIHARD III using standard protocols. 74+ hours of audio across 8 evaluation datasets.
Columns: WER = Word Error Rate (%, lower is better) · Perfect = number of files with 0% WER · High WER = files with WER above 50% · RTFx = Real-Time Factor multiplier (higher = faster than real-time)
Real Customer Audio - English (Gladia Internal Dataset)
| Provider | WER (%) |
| Solaria-3 | 9.6 |
| ElevenLabs Scribe v2 | 9.9 |
| AssemblyAI | 10.0 |
| Deepgram Nova-3 | 10.7 |
| Mistral Voxtral | 12.2 |
| Solaria-1 | 12.9 |
Switchboard - Conversational Speech
| Provider | WER (%) | Perfect transcriptions | High WER files | RTFx |
| Solaria-3 | 33.9 | 27 | 3 | 0 |
| Solaria-1 | 37.3 | 27 | 3 | 0 |
| AssemblyAI | 42.3 | 28 | 4 | 1 |
| Speechmatics | 46.0 | 31 | 7 | 0 |
| Mistral Voxtral | 48.1 | 25 | 6 | 6 |
| Deepgram Nova-3 | 49.8 | 26 | 7 | 1 |
| ElevenLabs Scribe v2 | 55.2 | 31 | 6 | 0 |
Common Voice 24 - Clean Audio (Multilingual)
| Provider | WER (%) | Perfect transcriptions | High WER files | RTFx |
| Speechmatics | 3.8 | 1067 | 2 | 2 |
| AssemblyAI | 3.9 | 1041 | 1 | 0 |
| ElevenLabs Scribe v2 | 3.9 | 1059 | 3 | 6 |
| Mistral Voxtral | 5.1 | 984 | 2 | 6 |
| AssemblyAI Universal v2 | 5.2 | 968 | 2 | 1 |
| Solaria-3 | 6.9 | - | - | - |
| Soniox v4 | 7.2 | 902 | 7 | 1 |
| Deepgram Nova-3 | 7.9 | 808 | 0 | 2 |
| Solaria-1 | 8.2 | 904 | 2 | 1 |
VoxPopuli - Formal Discourse (European Parliament)
| Provider | WER (%) | Perfect transcriptions | High WER files | RTFx |
| ElevenLabs Scribe v2 | 1.7 | 418 | 0 | 5 |
| AssemblyAI | 2.1 | 396 | 0 | 2 |
| Mistral Voxtral | 2.1 | 394 | 0 | 5 |
| Solaria-1 | 2.2 | 393 | 0 | 3 |
| AssemblyAI Universal v2 | 2.2 | 377 | 0 | 3 |
| Solaria-3 | 2.9 | - | - | - |
| Speechmatics | 3.0 | 326 | 0 | 6 |
| Deepgram Nova-3 | 3.2 | 363 | 0 | 7 |
Earnings22 Full - Financial Calls (Long-form, single file)
| Provider | WER (%) | RTFx |
| ElevenLabs Scribe v2 | 9.4 | 35 |
| Speechmatics | 10.0 | 17 |
| AssemblyAI Universal v3 | 11.0 | 71 |
| AssemblyAI Universal v2 | 11.1 | 82 |
| Mistral STT | 11.6 | 135 |
| Solaria-1 | 11.8 | 28 |
| Deepgram v3 | 14.5 | 348 |
Earnings22 Cleaned AA - Financial Calls (Curated by Artificial Analysis)
| Provider | WER (%) | RTFx |
| Solaria-3 | 6.4 | - |
| AssemblyAI | 6.9 | 64 |
| ElevenLabs Scribe v2 | 7.7 | 32 |
| Speechmatics | 7.8 | 24 |
| Mistral Voxtral | 7.9 | 57 |
| Solaria-1 | 8.1 | 39 |
| Deepgram Nova-3 | 12.0 | 234 |
Multilingual LibriSpeech - Audiobooks (5 languages, average)
| Provider | WER (%) | Perfect transcriptions | High WER files | RTFx |
| ElevenLabs Scribe v2 | 3.7 | 565 | 3 | 15 |
| AssemblyAI | 4.7 | 508 | 3 | 1 |
| Soniox v4 | 5.6 | 378 | 3 | 6 |
| Solaria-1 | 5.9 | 367 | 3 | 3 |
| AssemblyAI Universal v2 | 6.2 | 369 | 2 | 3 |
| Deepgram Nova-3 | 7.5 | 270 | 4 | 7 |
| Solaria-3 | 8.0 | - | - | - |
Multilingual LibriSpeech - WER by Language (%)
| Provider | German (DE) | Spanish (ES) | French (FR) | Italian (IT) | Portuguese (PT) |
| Solaria-1 | 5.0 | 4.0 | 4.8 | 9.9 | 5.3 |
| AssemblyAI Universal v3 | 3.5 | 3.2 | 2.6 | 9.7 | 4.4 |
| ElevenLabs Scribe v2 | 3.1 | 3.2 | 2.9 | 6.1 | 3.0 |
| Soniox v4 | 5.4 | 4.4 | 5.0 | 8.8 | 4.3 |
| AssemblyAI Universal v2 | 3.4 | 4.0 | 5.8 | 11.9 | 5.9 |
| Deepgram v3 | 6.9 | 4.6 | 6.2 | 8.8 | 11.3 |
Pipecat STT Benchmark - Real-Time Streaming
| Provider | WER (%) | Perfect transcriptions | High WER files | RTFx |
| AssemblyAI Universal v3 | 2.0 | 531 | 0 | 2 |
| ElevenLabs Scribe v2 | 2.2 | 512 | 0 | 8 |
| AssemblyAI Universal v2 | 2.5 | 494 | 0 | 1 |
| Mistral STT | 2.6 | 485 | 0 | 5 |
| Solaria-1 | 2.7 | 482 | 0 | 4 |
| Speechmatics | 2.7 | 476 | 0 | 0 |
| Soniox v4 | 2.9 | 480 | 0 | 2 |
| Deepgram v3 | 3.1 | 449 | 0 | 8 |
Speaker Diarization Benchmark - DIHARD III - Diarization Error Rate (DER %, lower is better)
| Provider | Broadcast | Meeting | Web Video | Socio Field | Court | Clinical | Restaurant | Socio Lab | CTS | Maptask | Simple Avg | Weighted Avg |
| Solaria-1 | 9.4 | 29.9 | 44.4 | 12.3 | 3.9 | 13.3 | 41.3 | 5.5 | 7.7 | 4.5 | 17.2 | 16.6 |
| NVIDIA NeMo Sortformer | 10.3 | 33.0 | 43.5 | 13.0 | 24.1 | 14.4 | 50.9 | 8.6 | 14.1 | 8.2 | 22.0 | 20.4 |
| pyannoteAI Community-1 | 10.5 | 35.8 | 48.7 | 17.9 | 11.6 | 23.8 | 49.9 | 13.9 | 12.3 | 10.2 | 23.5 | 23.0 |
| Speechmatics | 17.2 | 55.6 | 55.6 | 28.9 | 15.0 | 24.9 | 58.4 | 18.6 | 20.1 | 23.4 | 31.8 | 30.1 |
| AWS Transcribe | 16.4 | 51.4 | 60.3 | 25.2 | 16.7 | 27.3 | 63.1 | 20.2 | 31.2 | 22.9 | 33.5 | 33.8 |
| Soniox STT-async-preview-v1 | 24.8 | 58.3 | 57.5 | 30.1 | 39.3 | 35.1 | 67.4 | 28.0 | 29.2 | 27.6 | 39.7 | 37.8 |
| ElevenLabs Scribe-v1 | 25.6 | 50.5 | 63.4 | 29.7 | 23.1 | 47.7 | 57.4 | 30.3 | 22.9 | 45.2 | 39.6 | 39.5 |
| OpenAI GPT-4o Transcribe | 26.4 | 57.8 | 64.1 | 28.8 | 30.0 | 40.8 | 59.7 | 26.5 | - | 34.8 | 41.0 | 42.8 |
| AssemblyAI Universal | 30.9 | 46.4 | 68.4 | 33.1 | 24.5 | 51.4 | 59.4 | 33.1 | 33.1 | 42.1 | 42.2 | 43.9 |
| Deepgram v3 | 27.0 | 59.7 | 83.0 | 35.5 | 25.6 | 44.8 | 75.2 | 32.2 | 35.5 | 45.9 | 46.4 | 46.9 |
Speech-to-text benchmark FAQs
Which speech-to-text API has the lowest word error rate?
It depends on the audio. In this benchmark, Gladia's Solaria-3 has the lowest word error rate on real customer calls (9.6%), Switchboard telephone conversations (33.9%) and Earnings22 cleaned financial calls (6.4%). Speechmatics leads on Common Voice 24 (3.8%), ElevenLabs Scribe v2 leads on VoxPopuli (1.7%) and Multilingual LibriSpeech (3.7%), and AssemblyAI Universal v3 leads on the Pipecat real-time streaming benchmark (2.0%). Test on audio that matches your production use case.
How is word error rate measured in this benchmark?
Word error rate is the share of words a model substitutes, deletes or inserts compared with a human reference transcript. Each audio file is sent to every provider's production API with default settings. Transcriptions normalized with Gladia's open-source normalization library (https://github.com/gladiaio/normalization), based on the OpenAI Whisper text normalizer and extended to cover more languages, before WER computation.
What does RTFx mean in speech-to-text benchmarks?
RTFx is the real-time factor multiplier: how many times faster than real time a provider transcribes audio. Higher is faster. On the Earnings22 cleaned dataset, for example, Deepgram Nova-3 runs at 234x and AssemblyAI at 64x. RTFx measures throughput, not the delay before the first words appear in a live stream.
Which speech-to-text API is most accurate for speaker diarization?
Diarization accuracy is measured with diarization error rate (DER), where lower is better. On the DIHARD III benchmark, Gladia's Solaria-1 has a weighted average DER of 16.6%, compared with 20.4% for NVIDIA NeMo Sortformer, 23.0% for pyannoteAI Community-1 and 30.1% for Speechmatics.
SUMMARY
Average word error rate by provider
Solaria-3 has the lowest word error rate on real customer calls (9.6%), conversational telephone speech (Switchboard, 33.9%) and financial calls (Earnings22 cleaned, 6.4%). On clean read speech it trails: Speechmatics leads Common Voice at 3.8% to Solaria-3's 6.9%, and ElevenLabs Scribe v2 leads VoxPopuli at 1.7% to 2.9%.
Across the four datasets where all six providers were tested, average word error rate is Solaria-3 12.5%, AssemblyAI 13.8%, Speechmatics 15.2%, Mistral Voxtral 15.8%, ElevenLabs Scribe v2 17.1% and Deepgram Nova-3 18.2%. Conversational telephone audio drives the gap: excluding Switchboard, AssemblyAI leads at 4.3% and Solaria-3 is fifth at 5.4%. Data as of March 2026.
| Provider and model |
Average WER, 4 shared datasets |
Real customer calls WER |
Lowest WER on |
Price per 1,000 min |
| Gladia Solaria-3 | 12.5% | 9.6% | Real customer calls, Switchboard, Earnings22 cleaned | €2.67 async (Self-serve) |
| AssemblyAI | 13.8% | 10.0% | Pipecat streaming (Universal v3) | $3.50 async (Universal-3.5 Pro) |
| Speechmatics | 15.2% | Not tested | Common Voice 24 | $4.00 async (Melia 1 base) |
| Mistral Voxtral | 15.8% | 12.2% | None | $3.50 async (Voxtral Mini) |
| ElevenLabs Scribe v2 | 17.1% | 9.9% | VoxPopuli, Earnings22 full, Multilingual LibriSpeech | $3.67 async (Scribe v2) |
| Deepgram Nova-3 | 18.2% | 10.7% | None | $4.30 async (Nova-3) |
Average WER is the simple mean of Switchboard, Common Voice 24, VoxPopuli and Earnings22 cleaned, the four datasets where all six providers were tested. Per-dataset results are below. Prices are public list rates converted to per 1,000 minutes from each vendor’s pricing page (Gladia €0.16/hr Solaria-3 Self-serve; AssemblyAI $0.21/hr; Speechmatics Melia 1 $0.24/hr; Mistral Voxtral Mini $0.0035/min; Deepgram Nova-3 $0.0043/min; ElevenLabs Scribe v2 $0.00367/min). Checked October 2026.