whisper-tiny-grpo-5e2f49e-cv22-tiny-sftadaptive

A Whisper model fine-tuned with GRPO (Group Relative Policy Optimization) using a blended error-rate reward. Trained with whisper-rl.

Best validation (overall across 78 languages): WER 0.671, CER 0.290

Training data

Fine-tuned on cv22_index — all 78 Common Voice locales, streamed and decoded on the fly from the full training split. Each clip's language is pinned from its Common Voice locale during training.

Performance per language

Validation WER and CER at the best checkpoint, per Common Voice locale (also in the Evaluation Results metadata above):

Language	WER	CER
af	0.867	0.360
am	1.000	0.814
ar	0.588	0.232
as	0.594	0.307
az	0.860	0.257
ba	0.844	0.359
be	0.815	0.277
bg	0.750	0.206
bn	0.469	0.242
br	0.829	0.376
ca	0.660	0.218
cs	0.656	0.130
cy	0.650	0.240
da	0.615	0.264
de	0.306	0.103
el	0.409	0.101
en	0.293	0.134
es	0.587	0.221
et	0.981	0.296
eu	0.708	0.184
fa	0.967	0.456
fi	0.577	0.153
fr	0.448	0.141
gl	0.480	0.124
ha	0.409	0.135
he	0.882	0.366
hi	0.321	0.154
ht	0.905	0.411
hu	1.056	0.746
hy-AM	1.000	0.480
id	0.846	0.486
is	0.935	0.419
it	0.553	0.153
ja	1.000	0.396
ka	1.000	0.822
kk	0.700	0.290
ko	0.522	0.244
lo	0.909	0.755
lt	1.000	0.368
lv	0.750	0.190
mk	0.727	0.194
ml	0.535	0.337
mn	0.853	0.330
mr	0.200	0.091
mt	0.812	0.254
ne-NP	0.480	0.241
nl	0.476	0.167
nn-NO	0.833	0.336
oc	0.720	0.270
pa-IN	0.400	0.195
pl	0.615	0.215
ps	0.833	0.529
pt	0.333	0.155
ro	0.167	0.046
ru	0.636	0.221
sk	0.826	0.281
sl	0.583	0.167
sq	0.963	0.450
sr	0.444	0.226
sv-SE	0.483	0.157
sw	0.767	0.214
ta	0.400	0.223
te	0.679	0.356
tg	0.800	0.270
th	0.655	0.248
tk	1.095	0.469
tr	0.750	0.276
tt	1.000	0.378
uk	0.625	0.205
ur	0.680	0.280
uz	0.818	0.260
vi	0.522	0.315
yi	0.810	0.439
yo	0.935	0.440
yue	1.000	0.275
zh-CN	1.000	0.271
zh-HK	1.000	0.321
zh-TW	0.667	0.250

How it was trained

Instead of cross-entropy against a single reference, for each audio clip the policy samples a group of num_generations transcriptions, scores each by a negated blend of word error rate, character error rate, and length / repetition penalties, and is nudged toward the better candidates with a clipped policy-gradient objective regularized by a per-token KL penalty to the frozen base model. Advantages are the group-relative, standardized rewards (A = (r - mean) / (std + eps)), so no value network is needed. The clip's language is pinned from its Common Voice locale, and the policy's own greedy transcriptions are scored as validation WER and CER.

Hyperparameters

Field	Value
Base model	openai/whisper-tiny
Dataset	/data/cv22_index
Learning rate	1e-05
Sampling temperature	0.7
Group size (generations/clip)	8
Reward weights	{'cer': 1, 'wer': 1, 'length': 0.5, 'repetition': 0.5}
KL penalty (β)	0.04
Batch size (clips/step)	16
Max optimizer steps	1000000
Warmup steps	20

Training curves

Pulled from the Weights & Biases run (static snapshot):

Usage

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="wrice/whisper-tiny-grpo-5e2f49e-cv22-tiny-sftadaptive")
print(asr("audio.wav")["text"])

Limitations

A proof-of-concept GRPO recipe, not a tuned production system. WER and CER are reported on a held-out Common Voice validation slice after text normalization; real-world performance varies by domain, accent, language, and audio quality.