Benchmark
How well Lipik hears Indian English.
On 150 clips of Indian-accented English from AI4Bharat's Svarah test set, Lipik's default speech model, Parakeet, got 8.5% of words wrong. Whisper large-v3-turbo got 7.5%, Qwen3-ASR 1.7B 12.7% and Apple Dictation (English, India) 18.3%. Parakeet needed the least model time per clip.
Results published 28 Oct 2026. A lower word error rate is better.
Results on 150 clips
Word error rate is the share of words that had to be changed, added or removed to match what was said.
| Speech to text | Word error rate | Seconds per clip | Model time | Peak memory |
|---|---|---|---|---|
| Lipik, final text (Parakeet + clean-up) | 8.5% | 0.46 s | 0.06 s | 934 MB |
| Lipik, Parakeet text before clean-up | 8.5% | 0.46 s | 0.06 s | 934 MB |
| Whisper large-v3-turbo | 7.5% | 1.77 s | 0.69 s | 1,173 MB |
| Qwen3-ASR 1.7B | 12.7% | 1.23 s | 0.24 s | 3,690 MB |
| Apple Dictation, English (India) | 18.3% | 0.15 s | 0.15 s | 22 MB* |
Seconds per clip is the median for the whole run of one clip. For Lipik, Whisper and Qwen3-ASR it includes loading the model each time (the app keeps it loaded) and, for Lipik's final text, the clean-up. Model time is speech to text only. *Apple's model runs inside a macOS system service, so its memory number leaves the model out.
What the numbers mean
- Lipik uses Parakeet Unified English 0.6B by default. It got about 1 word in 12 wrong on this set.
- Whisper large-v3-turbo made fewer mistakes, 7.5%, but needed about 11 times the model time per clip (0.69 s against 0.06 s). That is why Parakeet is the default. Whisper models are a free download in Lipik's Settings if you prefer them.
- Clean-up did not change the word error rate. It removes fillers and fixes punctuation and formatting, which this score ignores.
- Apple Dictation with English (India) made about twice as many mistakes as Parakeet on this set. Lipik uses it only for Hinglish, where it writes Hindi words in Roman letters.
By the speaker's first language
Svarah's speakers come from many parts of India. Each group here has only 4 to 13 clips, so treat these numbers as rough.
| First language | Clips | Lipik (Parakeet) | Whisper large-v3-turbo | Qwen3-ASR 1.7B | Apple, English (India) |
|---|---|---|---|---|---|
| Assamese | 4 | 8.0% | 8.0% | 52.0% | 44.0% |
| Bengali | 10 | 4.2% | 1.2% | 3.6% | 12.5% |
| Bodo | 10 | 7.9% | 7.9% | 7.9% | 30.2% |
| Dogri | 9 | 10.4% | 13.5% | 6.2% | 17.7% |
| Gujarati | 5 | 3.1% | 3.1% | 1.6% | 6.2% |
| Hindi | 7 | 5.8% | 4.7% | 7.0% | 10.5% |
| Kannada | 11 | 5.9% | 5.4% | 10.4% | 12.2% |
| Kashmiri | 8 | 10.6% | 11.4% | 17.9% | 25.2% |
| Konkani | 8 | 12.8% | 10.5% | 29.1% | 14.0% |
| Maithili | 7 | 9.1% | 6.5% | 2.6% | 11.7% |
| Malayalam | 9 | 15.3% | 16.5% | 10.6% | 15.3% |
| Marathi | 7 | 8.5% | 8.5% | 8.5% | 19.7% |
| Nepali | 13 | 6.3% | 3.4% | 5.7% | 20.6% |
| Odia | 6 | 3.9% | 2.6% | 2.6% | 17.1% |
| Punjabi | 5 | 1.7% | 5.2% | 3.4% | 24.1% |
| Sindhi | 5 | 12.0% | 6.0% | 54.0% | 6.0% |
| Tamil | 8 | 5.6% | 1.4% | 11.3% | 15.5% |
| Telugu | 6 | 5.5% | 8.2% | 30.1% | 19.2% |
| Urdu | 12 | 24.0% | 19.4% | 27.9% | 34.1% |
How it was measured
- Clips: 150 clips from Svarah, AI4Bharat's test set of Indian-accented English (CC BY 4.0), from speakers of 19 first languages.
- Score: both the output and the reference are lower-cased, punctuation is dropped and numbers are written as digits before counting word errors.
- How each ran: Parakeet, Whisper and Qwen3-ASR through Lipik's own speech engine on a Mac with Apple silicon; Apple Dictation through Apple's on-device speech service. Lipik's final text used clean-up with Qwen3 8B.
- Not measured here: how close the text is to what you'd send. Svarah's references are plain transcripts, so only the word error rate is meaningful on this set.
- Limits: one test set of 150 clips of Indian-accented English; results on your voice may differ.
The scripts, the clip list and the full results are on GitHub: RESULTS.md and how to run it yourself.
What about Hinglish?
This test is English only. Hinglish is harder, and Hinglish dictation is still in beta. How it works is on the Hinglish speech to text page.
Questions
What is Svarah?
A test set of Indian-accented English speech from AI4Bharat, released under CC BY 4.0. Its speakers come from many parts of India.
Which speech model does Lipik use by default?
Parakeet Unified English 0.6B, a one-time 731 MB download that runs on your Mac. On 150 Svarah clips it got 8.5% of words wrong.
Is Whisper more accurate than Parakeet?
On this set, Whisper large-v3-turbo got 7.5% of words wrong against Parakeet's 8.5%, but it needed about 11 times the model time per clip. You can switch to a Whisper model in Lipik's Settings.
How did Apple Dictation do?
Apple Dictation with English (India) got 18.3% of words wrong on the same 150 clips.
Can I run the benchmark myself?
Yes. The scripts and instructions are in training/benchmark in Lipik's GitHub repository. Svarah needs a free Hugging Face login to download.
Get Lipik for Mac.
Free, for Macs with Apple silicon. Open the download, drag Lipik into Applications, and hold fn to start.
curl -fsSL
https://lipik.pages.dev/install.sh | sh
Pasting the line above into Terminal installs Lipik with no security warning. If you use the download instead, macOS asks you to confirm the first time: go to System Settings, then Privacy & Security, and click Open Anyway. Install help