Benchmark
Phoneme error rate (PER) on held-out sets: reciters, phones and a professional the model never heard. Lower is better.
Zipformer Quran 3.1, the current model
All 600 clips of quranic-asr-benchmark v1.1, every one scored, with the corrected references. Greedy decoding in the streaming setting, no language model.
- All 600 clips
- 4.13%
- 25,989 reference phonemes
- Held-out studio reciters
- 2.06%
- 200 / 200 clips
- Real phone recordings
- 5.96%
- 200 / 200 clips
- A professional reciter it never heard
- 5.09%
- 200 / 200 clips
How the score works
Phoneme error rate counts what the model got wrong, one sound at a time.
Every clip comes with a reference: the canonical phonemes of what was recited. The scorer lines up what the model wrote against the reference and counts each phoneme that was swapped, dropped or added. Phoneme error rate (PER) is that count divided by the number of phonemes in the reference, 25,989 across the 600 clips.
A phoneme score sees what a word score misses. A short madd, a missing ghunnah or a wrong vowel changes one phoneme, but the word looks the same once diacritics are stripped, as they are for the word scores on the leaderboard.
Lower is better. 4.13% means that about 4 in every 100 reference phonemes came out differently.
- Clips
- 600, in three sets of 200
- Audio
- 16 kHz mono, about 128 minutes
- Reference phonemes
- 25,989
- Decoding
- Greedy, in the streaming setting, no language model
- References
- v1.1, corrected on 15 August 2026
About this benchmark
quranic-asr-benchmark v1.1 is Quran Lab's held-out test: 600 clips in three sets of 200, none of them used in training. Every model is scored on the same clips, so versions can be compared.
Clean studio recitation the model was not trained on.
Recitation on phone microphones, held out from training.
One professional voice absent from every training source.
What it cannot tell you
- 1
It scores the written text, not tajweed judgement
PER compares the output to the canonical phonemes of each ayah. It does not measure whether a reciter's mistake is caught. On a separate set of 159 recordings with human-marked tajweed deviations, 3.1 kept 22.7% of them and wrote the canonical form for the rest.
- 2
The Quran is one closed text
Every ayah in the benchmark also occurs in training, recited by other people. What is held out is the recording and the voice, never the words.
- 3
The audit checks recordings, not voices
No benchmark recording appears in training. That rules out the same file, not an acoustically identical copy under another name.
- 4
Three sets, one professional
The unseen-reciter set is a single voice. It shows how the model meets a new reciter, not how it meets every reciter.
Zipformer Quran 2 and 3
On quranic-asr-benchmark v1.1, as corrected on the model card.
Phoneme error rate. Lower is better. No language model is used: every phoneme comes from the sound, so a mistake in recitation stays visible instead of being corrected away.
Emphatic letters
ص ض ط ظ used to fail at a multiple of the rate of plain letters. In v3 the gap is statistically zero on all three held-out sets.
Run your own model
The clips, the references and the scorer are public, so any model can be scored the same way.
- 1
Ask for access to quranic-asr-benchmark on Hugging Face.
- 2
Transcribe every clip listed in benchmark.jsonl, and write one line per clip: its id and your text.
- 3
Score the file with score.py, which ships with the dataset and applies the leaderboard's normalization.
- 4
Compare with the models on the leaderboard.
# one line per clip id, in jsonl:
# {"id": "everyayah_heldout/xxx", "text": "..."}
pip install rapidfuzz
python score.py --hyps my_hyps.jsonlThe leaderboard ranks speech recognizers by word error on the same 600 clips, with the same greedy decoding and the same normalization for every model: diacritics, Quranic annotation signs and tatweel removed, hamza forms and alef maqsura unified. A second column also ignores alef, so the Uthmani spelling and the everyday spelling of a word count as the same.
Changes to the benchmark
v1.1 · 15 August 2026
Ten clips of the professional reciter held more than their file names said: in nine the ayah is recited twice, and one goes on to the next ayah and back. Each was checked against the audio and its reference corrected. No audio changed and no clip was removed. Models that wrote the whole recording faithfully gained; for most, the two versions differ by about one point of word error or less.
Behind the numbers
Quranic ASR leaderboard
The leaderboard scores any model on the same held-out sets. It runs on Hugging Face.