capacit-ai/saga-2-m

saga-2

Danish automatic speech recognition. Ranked #1 on the Danish ASR Leaderboard.

thorhojhus/whisper-large-v3-turbo-danish's encoder paired with a Qwen3-0.6B-Base decoder through a frame-stacking projector. 1.24 B parameters, bf16. Output is cased and punctuated Danish.

Try out saga-2 here

Usage

pip install -r requirements.txt

from saga2 import load

model = load("capacit-ai/saga-2-m")          # CUDA if available, else CPU
print(model.transcribe("meeting.mp3"))
print(model.transcribe(["a.wav", "b.flac"], batch_size=16))

From the command line:

python run.py clip.wav                                 # weights from this directory
python run.py --repo capacit-ai/saga-2-m a.wav b.mp3   # or fetch from the hub
python run.py --json out.jsonl *.flac                  # one {"file","text"} line per file

Any sample rate or container librosa can read. Audio longer than 30 s is split at silences into pieces of at most 28 s and rejoined automatically.

Results

Measured with the leaderboard's own evaluation harness and normaliser (NFKC, spoken-number canonicalisation, hesitation fillers removed from reference and hypothesis alike), greedy decoding, on the five public Danish test sets. Percentages, lower is better.

split	WER	CER
CoRal conversation	18.38	10.67
CoRal read-aloud	11.10	4.54
Common Voice Danish	7.14	2.44
FLEURS da_dk	8.07	3.41
FTSpeech	6.08	3.34
mean	10.15	4.88

268x real time on one RTX 5090 at batch 16.

Checkpoint selection used an in-training validation mix that includes a slice of the FLEURS da_dk test split, so the FLEURS column is mildly optimistic. The other four columns are untouched by selection.

Why these weights changed

An earlier release of saga-2 used an encoder we had fine-tuned ourselves from stock openai/whisper-large-v3-turbo. Swapping in thorhojhus/whisper-large-v3-turbo-danish as the encoder and continuing training scored better on four of the five test sets and on the mean, so those are the weights published here. There is no reason to ship the weaker model when the better one exists, and the repository name and usage stay the same so existing code keeps working.

Architecture

16 kHz audio -> log-mel (128 bins)
  -> Whisper-large-v3-turbo encoder                   32 layers, d_model 1280, 20 heads
  -> frame stack x4                                   one token per 80 ms, 375 per 30 s
  -> RMSNorm(5120) + Linear(5120 -> 1024)             projector, 5.25 M params
  -> Qwen3-0.6B-Base decoder                          28 layers, hidden 1024, 16 heads / 8 KV
  -> cased, punctuated Danish

Parameters	1.238 B (encoder 0.635 B, decoder 0.596 B, projector 5.25 M)
Precision	bf16
Encoder window	30 s
Released checkpoint	EMA (decay 0.9999)

The encoder keeps Whisper's absolute sinusoidal position embeddings, frozen throughout. The position table is sliced to the actual window rather than requiring a padded 30 s block, so short clips cost proportionally less compute.

The decoder sees audio embeddings spliced into its token stream as [audio][<|im_start|>] text [<|im_end|>], and generation stops on <|im_end|> (151645), not the tokenizer's <|endoftext|>. The bundled saga2 package handles this; calling generate directly requires setting eos_token_id=151645.

Whisper turbo's 4-layer decoder loops on short conversational clips. Replacing it with a full 28-layer Qwen3-0.6B language model is the main quality change in saga-2.

Training data

Training datasets used on the final run, no more than 2 epochs were used of each.

corpus	description
FTSpeech	Danish parliament proceedings
CoRal read-aloud	Prompted read speech, broad dialect coverage
CoRal conversation	Spontaneous conversational Danish
NST Danish	Read speech, Norwegian Language Bank
FLEURS da_dk train	Read sentences
Common Voice 17 Danish	Crowd-sourced read speech

Augmentation

Channel augmentation on 60 % of rows (room impulse response, MUSAN noise, or codec simulation) at 0 to 20 dB SNR, VTLP on 35 %, speed perturbation, gain ±10 dB, SpecAugment. Trailing non-speech tails on 25 % of clips and 2.5 % of rows replaced with non-speech carrying an empty transcript, which is why the model returns an empty string on ring tones or dead air instead of hallucinating.

Training recipe

The encoder is thorhojhus's work. Both his released weights and his published recipe are used here:

• thorhojhus/whisper-large-v3-turbo-danish The encoder weights, and the Muon recipe below. Whisper-turbo fine-tuning.
• thorhojhus/multimodal-alignment The projector recipe.

His hyperparameters, used essentially unchanged:

Muon, hidden matrices	peak lr 5e-6, RMS-matched to AdamW, momentum 0.95 Nesterov, 5 Newton-Schulz steps, weight decay 0.01
AdamW, everything else	lr 2.5e-6, β (0.9, 0.98), ε 1e-6, weight decay 0.01
Gradient clipping	1.0
Numerics	bf16 parameters with compensated fp32 optimizer updates
Schedule	WSD: warmup 1000, stable, linear decay to 0.1x
EMA	0.9999, fp32

The compensated update matters: plain bf16 write-back discards most of a 5e-6 Muon step, because the update is below the representable gap at typical parameter magnitudes. Kahan compensation preserves it (This is absolutely essential for MUON).

Changes from his setup: split learning rates (encoder Muon 5e-6, Qwen3 decoder 2e-5, projector AdamW 1e-4), the decoder replacement described above, and a projector-alignment phase that trains the projector alone against the frozen encoder and decoder before unfreezing all 1.236 B parameters. Trained on a single RTX 5090.

Inference details

Decoding uses a per-clip token cap of ceil(seconds × 8) + 12, a repeated-token loop stop, and n-gram collapse on the output.

Clips over 30 s are split at the midpoints of detected silences into pieces of at most 28 s, and the hypotheses are joined. Context does not cross a chunk boundary, so a name established early in a long recording will not inform a later chunk, and capitalisation restarts per chunk.

transcribe length-sorts chunks before batching, which matters because a batch decodes until its longest member finishes and pads mel to its longest member.

Limitations

• Danish only. Other languages were not trained or evaluated.
• No word-level timestamps, only per-chunk boundaries.
• No speaker diarization.
• No streaming; decoding is offline.
• Conversational speech remains the hardest case at 18.38 WER, and overlapping speech, crosstalk and far-field microphones degrade it further.
• Context is not shared across chunks in long-form audio.
• FTSpeech carries 57 % of the sampling weight, so formal monologue Danish is the strongest domain.
• Numbers and rare proper nouns are the most common residual error class.

Users deploying in high-stakes settings should measure accuracy on their own population rather than assuming it is uniform across accents, ages and recording conditions.

Files

model.safetensors        1.24 B parameters, bf16
config.json              nested whisper + qwen configs, frame stack, encoder position scheme
variant.json             checkpoint variant and step
saga2/model.py           encoder, projector, decoder, generation
saga2/encoder_pos.py     encoder position handling and the variable-length window forward
saga2/gpu_mel.py         log-mel on GPU, matching WhisperFeatureExtractor
saga2/decode.py          chunking, guards, batching
run.py                   command-line transcription

Citation

@misc{saga2-2026,
  title  = {saga-2: Danish speech recognition with a Whisper encoder and a Qwen3 decoder},
  author = {capacit},
  year   = {2026},
  url    = {https://huggingface.co/capacit-ai/saga-2-m}
}

Acknowledgements

thorhojhus, whose whisper-large-v3-turbo-danish the encoder in this model comes from, and whose Muon recipe and projector recipe the training follows. Alexandra Institute for FTSpeech and CoRal, Norwegian Language Bank for NST, and Rye-AI for the Danish ASR leaderboard.

License

Apache 2.0. The Whisper encoder is MIT and Qwen3 is Apache 2.0. Training corpora carry their own licenses.