What didn't work
Failed experiments, dead ends, and retracted features. With root-cause analysis and what I did instead. The point of this doc is that you don't repeat my dead ends — they cost about 6 weeks total across the project.
0. The Falklands → Fucklands obscenity bug (v1/v2 tokenizer)
What I tried: Train ControlMT v1 with a 64K SentencePiece Unigram vocabulary, joint KN+EN.
What happened: During v2 production testing, the model output Fucklands for the input Falklands. Same root cause hit other inputs: PyTorch → 4 mangled subwords, backpropagation → back/prop/ag/ation, dark matter → "ಗಾಢ ಪಥ" (dark path), Shika (Japanese place) → "Chicago". Each of these is a tokenizer fault. The model did exactly what it should given the broken tokens.
Root cause: 64K vocab + corpus-driven Unigram training without curated user_defined_symbols. Falklands got split into subword pieces, one of which collided with a piece that the model strongly associated with completing an obscene English bigram. No amount of retraining the model fixes this — the obscene completion is encoded into the vocabulary. It's a foundation crack.
What I did instead: Built a new 128K SentencePiece Unigram tokenizer with force-included single-token coverage of top-50K English words, top-30K Kannada words, top-1K named entities (Falklands, Modi, Bangalore, ISRO, ...), top-500 tech terms, and currency/units. Ran a 10-gate audit script (scripts/tokenizer_audit.py) — pass criteria includes "no obscene tokens in vocab," "no bigram pairs that form slurs," "char/token ratio ≥ 3.0 for English." This audit became "Rule Zero" in PROTOCOLS.md — non-negotiable for every retrain.
Cost: 14 days of training on the bad vocabulary in v1/v2, then ~3 days to retrain the tokenizer + reaudit + bump v2.1 from 106M to 139M params (the vocab change alone added the parameters). Plus the reputational risk if the obscene output had reached an actual user.
Permanent lesson (saved in PROTOCOLS.md): a broken tokenizer is unfixable by retraining the model. Always audit before training, never assume defaults are safe.
1. The style-token claim (v2.2 → retracted same day)
What I tried: Add a 6-token style/control embedding to the input prefix (STRICT, NATURAL, FORMAL, CASUAL, JSON, TEXT). Same translation pair, different stylistic registers selectable at inference. The pitch was "compact translator that also exposes register control."
What happened: Model trained fine. Val loss 2.19 (best ever at that point). Label-prediction accuracy for the style token was high. But at inference, when I generated outputs for the same 100 source sentences across all 4 styles:
- FORMAL produced visibly different outputs on ~5% of sentences (e.g. "ಪ್ರಯಾಣ" instead of "ಸಂಚಾರ" for "travel")
- STRICT / NATURAL / CASUAL produced bit-for-bit identical outputs on 95% of sentences
The style-control was hollow. I had a model that could predict the style label (because the input prefix carried it) but didn't use the style to actually shape outputs.
Root cause: At 139M parameters, the model splits its capacity. ~95% of training-corpus labels were NATURAL (default). The other 5% spread across 3+ styles. With such an imbalanced distribution, the model learned that "given any source sentence, the most likely output is the NATURAL-register version" — the style token was a high-bias hint the model could safely mostly ignore. The few stylistically distinct examples weren't enough to teach the model what to do differently for each non-NATURAL label.
What I did instead: v2.3 — retrained the same architecture (kept the style embedding tensor in place to maintain weight-shape compatibility with v2.2's modeling_controlmt.py) but forced style_id = DEFAULT_NATURAL on every training row. This recovered the parameter capacity that was being spent on the unused control signal and gave v2.3 strictly better translation quality than v2.2.
Cost: ~2 weeks of corpus prep + training time was specifically for style control. The corpus itself (master_v22.jsonl) survived as v2.3's training set. The eval-curated set with style-tagged outputs got repurposed as a regression test.
What I'd do differently: At project start I had ~5K explicitly style-tagged pairs and 6.6M default-style pairs. That ratio (0.075%) was never going to teach style control. If I'd done the math then, I would have either (a) deferred style control until I had ≥10% per-style coverage, or (b) generated synthetic style-paired data (same source, multiple register-variant outputs from a teacher model). Instead I assumed "model will figure it out from the few examples." Models don't figure it out from a few examples — not at 139M params.
2. Gradio's flex-wrap battle (release day, 2 hours wasted)
What I tried: Build a polished Gradio Space demo. Two columns on desktop (Settings | Translate), single column on mobile.
What happened: On mobile (~360px viewport), the two columns stayed side-by-side, each squeezed to ~150px wide. Settings labels got truncated to single characters ("S", "D"). Translate textboxes were unusable. I spent ~2 hours iterating on CSS — @media queries, :has() selectors, display: block !important, fill_width=True, custom elem_classes on the Row and Columns. Nothing worked.
Root cause (eventually): My custom CSS had .main-row { display: flex; gap: 16px; } — without flex-wrap: wrap. Flex defaults to nowrap. Gradio's own default Row CSS does include flex-wrap: wrap, so the built-in min_width=320 parameter on Columns would normally trigger column wrapping when total width < ~660px. But my custom rule was overriding Gradio's wrap with nowrap, which is why min_width=320 silently did nothing. All my @media-block "block-out-the-row-on-mobile" overrides were fighting my own CSS, not Gradio's.
What I did instead: After two hours of pointless CSS, switched the Space from Gradio SDK to Docker SDK + FastAPI + vanilla HTML/CSS/JS. Mobile responsiveness then became "normal CSS for a normal HTML page" — single-column layout always, max-width 760px centered on desktop, full-width on mobile. Solved in 30 minutes of clean implementation.
What I'd do differently: When a framework's mobile output looks broken, before writing custom CSS, read the framework's default CSS. Gradio's defaults are sensible. I was solving a problem I'd created myself by overriding them. The lesson is broader: when you're three hours into fighting a framework, the right move is often "stop fighting the framework, use a different framework or write the layer yourself" — not "more CSS hacks."
3. bitsandbytes int8 GPU quantization
What I tried: Apply BitsAndBytesConfig(load_in_8bit=True) to ControlMT to get GPU int8 inference. The plan was a controlmt-v2.3-int8-gpu HF repo alongside the CPU dynamic int8 one.
What happened:
RuntimeError: self and mat2 must have the same dtype, but got Half and Char
Failure on the first translation call.
Root cause: bitsandbytes replaces nn.Linear layers with Int8Params versions. These expect specific dtype-routing assumptions about the surrounding model code — input activations stay in fp16, the linear layer dequantizes to fp16 internally, output stays fp16. Our custom model.py does its own matmul operations (not through nn.Linear for some paths), and those matmuls received Char (int8) tensors mixed with Half (fp16) activations, which torch can't multiply.
What I did instead: Documented it as "Not directly supported" in release/DEPLOYMENT.md §9. Made the CPU int8 dynamic path (via torch.quantization.quantize_dynamic) the recommended low-memory deployment, since it's already 1.8× faster than CPU bf16 with no quality regression — and crucially, GPU fp16 at 0.19s/pair already beats int8 on every metric except memory, which we don't actually need (the model is 280 MB at bf16).
Cost: ~1 hour to test + 30 min to write the honest "not supported" doc.
What I'd do differently: Custom architectures + quantization libraries that assume standard layouts is a common landmine. If you must have int8-bnb support, structure your modeling code as nn.Linear calls exclusively (no manual matmuls), and route all activations through standard dtypes. ControlMT's model.py was written for clarity over framework compatibility — the trade-off was right for the project, but it forecloses some standard tooling.
4. ONNX export (deferred indefinitely)
What I tried: Briefly investigated using optimum-onnx to export the model for cross-platform inference (Windows native, browser via ONNX Runtime Web, no Python required).
What happened: optimum-onnx's auto-exporter doesn't know about custom seq2seq architectures with trust_remote_code=True. It only handles known classes (T5, BART, mT5, etc.). Manual export requires:
- Separately exporting the encoder forward pass
- Separately exporting the decoder step (one autoregressive step)
- Reimplementing beam search in the target runtime (Python+onnxruntime, JS, or C++)
- Handling the anti-LM contrastive decoding (which needs two encoder masks alternated)
I estimated 3-4 hours of focused work to get a basic encoder+decoder-step ONNX export, plus another 2-3 hours to write a working beam-search loop around it.
What I did instead: Marked it as experimental in release/DEPLOYMENT.md §10, with a sketch of the manual export pattern. Pointed contributors at the GitHub repo. At 0.19s/pair on a $300 GPU, ONNX's main use case (CPU edge deployment) is already covered by our int8-dynamic recipe at 0.28s/pair on CPU.
Cost: ~20 minutes of investigation, 15 minutes of writing the experimental doc note.
What I'd do differently: Nothing. This was a correct deferral decision. ONNX export is a real cross-platform win for some use cases (browser inference, Windows native without Python), but the cost-benefit at our model size is bad. If someone in the community wants this, they can build it — the architecture is open-source.
4a. v2 DataLoader OOM crisis at step 8500
What I tried: Scale the v1 dataset from 877K to 8M pairs, train v2 with the same DataLoader(num_workers=4) configuration that worked fine in v1.
What happened: At step ~8500 of epoch 1, training crashed with RuntimeError: DataLoader worker (pid 2185568) is killed by signal: Killed. The Linux OOM killer had reaped a DataLoader worker after exhausting 15 GB RAM + 4 GB swap. The 7000 steps before the crash were the early warning I missed — throughput silently dropped from 15 batch/s → 1.5 batch/s as swap thrashing built up.
Root cause: load_pairs() built a Python list of 16M (src, tgt, direction_id) tuples in memory. Each Python string carries a refcount header in the same memory page as the data. When DataLoader forks workers, pages are shared via copy-on-write (CoW) — but the CoW breaks every time Python increments a refcount, which happens just by iterating the list. Over thousands of steps, workers progressively duplicate the whole dataset. 4 workers × 5GB dataset = 20GB extra → OOM on a 15GB machine.
v1 didn't hit this because 877K pairs ≈ 500MB in RAM, so 4 workers × 500MB = 2GB extra fit easily. The issue is the dataset size × num_workers product, not either alone.
What I did instead (the numpy memory-mapped dataset migration):
- Immediate fix:
num_workers=4 → 0. Stopped OOM but throughput stayed at 1.5 batch/s (no parallel data loading). - Proper fix: pre-tokenize the corpus once with
model/preprocess_pairs.pyinto flat int32 numpy arrays + offset arrays. Load withnp.load(path, mmap_mode='r'). Numpy arrays have no per-element Python refcount — workers share the mmap'd pages via the kernel page cache.num_workers=2+is now safe.
Result: back to ~15 batch/s, stable RAM at ~3-4 GB, no OOM.
Cost: 2-3 days of debug + recovery (running checkpoints from before the crash, validating the new numpy pipeline, re-tokenizing the corpus).
What I'd do differently: Sanity-check RAM/dataset_size × num_workers ratio before launching any long training. If raw_data_size × num_workers × 1.5 > available_RAM, switch to numpy mmap or reduce workers. Detection signal: if batch/s drops over hours with no other config change, suspect memory growth — check free -h regularly. dmesg | grep "killed process" shows OOM killer events after the fact.
4b. v2 eval loop ate 80% of wall time
What I tried: Carry the v1 EVAL_EVERY=1000 setting into v2.
What happened: With 5% val ratio on 16M bidirectional examples (805K val pairs) at batch 24 = 33,580 val batches per eval. Each eval took ~11 minutes. With 638 evals per epoch, val time = 638 × 11 min = 117 hours per epoch. Training itself was ~80 sec per 1000-step cycle. We spent more time evaluating than training.
Root cause: v1 had 44K val examples (1830 batches × 30 sec = quick). v2 had 805K val examples — same EVAL_EVERY setting, 20× longer per eval, didn't notice until throughput felt off.
What I did instead: Added EVAL_MAX_BATCHES=300 and made evaluate() take an optional max_batches arg. Periodic eval calls with max_batches=300 (capped trajectory signal). End-of-epoch eval calls with max_batches=None (full val, for checkpoint decisions). Single-line change, 10× wall-clock recovery.
Cost: ~1 hour to diagnose + write the fix.
What I'd do differently: At project setup, define "periodic eval must complete in <10% of the training cycle it interrupts." If EVAL_EVERY=1000 steps takes 80 sec training, periodic val must take <8 sec. Capped val on a subset is fine for trajectory signal; reserve full val for epoch boundaries when you make checkpoint decisions.
4c. v2.1 decoder pollution — code-mix Kannada as a training target
What I tried: Include code-mix Kannada sentences (Kannada matrix with embedded English-script tokens) as both KN→EN sources AND EN→KN targets during v2.1 training. The implicit assumption: "if it's in the dataset, the model learns to handle it both directions."
What happened: After v2.1 training completed (val_loss ≈ 2.38), I noticed the model producing outputs like:
catch → ಕ್ಯಾಚ್(transliterated, should be ಹಿಡಿ)later → ಲೇಟರ್(transliterated, should be ನಂತರ)run → ರನ್(transliterated, should be ಓಡು)
Common English verbs were being transliterated into Kannada instead of translated. The decoder had learned that "outputting a few Latin words in the middle of Kannada is normal" — because in training, that was a normal pattern in the EN→KN targets.
Root cause: Training had no constraint preventing code-mix Kannada from being a target. When the dataset loader emitted bidirectional swaps for every pair (KN→EN and EN→KN both trained), code-mix KN rows trained the decoder to output code-mix patterns. With enough exposure (~5 epochs), the decoder generalized this to "output Latin-script tokens for common English words" regardless of source.
What I did instead: Built v2.2 with a kn_is_mixed flag in model/preprocess_pairs.py that fires when KN side has ≥3 consecutive Latin words. The dataset loader uses this flag to gate the EN→KN swap — mixed-KN rows are trained KN→EN only, never as EN→KN targets. This is the explicit "decoder hygiene rule" baked into v2.2.
The HARDER decision: warm-start v2.2 from v2.1 best.pt, or retrain from scratch? I tested warm-start. Several thousand fine-tune steps later, the model was still occasionally producing catch → ಕ್ಯಾಚ್. Once a decoder has learned a wrong signal across multiple epochs, warm-starting will fight you forever. v2.2 trained from scratch — 3.5 days of GPU time, no fine-tune shortcut.
Cost: 2 weeks of v2.1 work that couldn't be salvaged for the v2.2 weights. The corpus and tokenizer survived. The model weights had to be retrained.
What I'd do differently: When training data has any asymmetric pattern (something only valid in one direction), encode the asymmetry in the dataset code, not in the training-loop hopes. The kn_is_mixed gate is 10 lines of Python — it should have been there from the start of v2.
5. The controlmt-v2.1 release that didn't happen
What I tried: After v2.1 finished training (early June 2026, val loss 2.38, COMET-DA 0.85 both directions), I considered releasing it publicly. I had eval numbers I was proud of, a working architecture, and a clean codebase.
What happened: I didn't release it. Instead I decided to keep v2.1 internal and immediately start work on v2.2 (with style tokens), because I (incorrectly) believed v2.2 would be a strict superset of v2.1 — same translation quality + a new feature. I treated v2.1 as a stepping-stone.
Two months later: v2.2 had to be retracted (see #1 above), v2.3 took another two weeks. v2.1 was never released. v2.3 is what shipped — eight months after starting the project — when v2.1 was already shipworthy at month 3.
Root cause: Optimism bias. I assumed v2.2 would be a clean upgrade. It wasn't. The combinatorial risk of "add a new feature while also retraining" is real — when something goes wrong, you don't know if it's the feature or the retraining, and you've lost ground on the previous-known-good.
What I did instead (now, in hindsight): nothing — v2.1 stayed internal. Public artifacts (model card, blog, banner) reference v2.2 as the first public version (memory note: [[project-controlmt-v21-unreleased]]).
What I'd do differently: Ship the stepping stones. If you have a working version that hits your eval gate, release it before starting the next ambitious experiment. The release process itself surfaces real-world issues (mobile UX, install paths, "vLLM doesn't work") that you want to discover with a known-good model, not while you're also debugging a new training run. If v2.1 had shipped in June, the deployment work for v2.3 in late June would have been incremental polish, not first-time discovery.
5a. Gemini bulk-synthesis produced ~70% duplicate outputs
What I tried: Use Gemini API to bulk-generate ~25K KN↔EN tech-domain pairs (programming, networking, hardware) to fix the v2.1 "PyTorch fragmentation" gap.
What happened (logged in CONTROLMT.md §4.1): Gemini bulk-generation had massive duplication rates:
- Web/networking domain: 2.5% unique (126 unique pairs out of 5K generated)
- Programming domain: 18% unique
- Hardware domain (generated via local Gemma instead): 84% unique
Total: 24,963 raw pairs → 9,069 after deduplication (63.7% duplicates).
Root cause: Bulk-prompted Gemini converges on a small set of canonical examples. Given a prompt like "generate 100 KN↔EN pairs about networking," the model picks the most-frequent-in-training-data networking concept (probably IP addresses or routers) and produces close-to-identical outputs across batches. Without explicit per-call seeding (different concept anchors per request), the diversity collapses.
What I did instead:
- Local Gemma with explicit per-call seed terms got 84% uniqueness. Slower per call but produces real diversity. Pattern: pass an explicit term list ("aspirin, paracetamol, ibuprofen, ...") as the seed and constrain "use only these terms; vary sentence structure."
- IT2 retranslation as a cleaning pass — take generated EN pairs, retranslate to KN using IndicTrans2 1.1B, compare to original KN. Caught script leaks (one row had Japanese script in KN — Gemini hallucination).
- Standing quality filter (
scripts/quality_filter.py) — applies to all generated/corrected data. Confidence score 0-1 per row stored as_quality. Drops below threshold 0.7. Catches: foreign-script leaks, true word triplet repetitions, length-ratio outliers, exact duplicates.
Cost: ~3 days of effort generating bulk synthetic data that I had to drop or heavily filter. Eventually got 9K usable tech pairs vs the 25K target.
What I'd do differently: For future synthesis, default to local Gemma with explicit per-call seeds. Reserve Gemini for one-off corrections (where each pair is unique by construction), not bulk generation. And budget the dedup pass into the time estimate — assume 50-70% raw duplicates and plan accordingly.
6. CPU int8 quantized weights as a separate file
What I considered: Pre-quantize the v2.3 weights to int8 and upload as model.int8.pt alongside the bf16 model.safetensors. Users get smaller download (140 MB instead of 280 MB) and skip the runtime quantization step.
What I learned before doing it: torch.quantization.quantize_dynamic returns a quantized model object, not a quantized state dict — and the underlying storage uses packed _packed_params blobs that torch.save serializes but that aren't portable across all PyTorch versions. The "pre-quantized file" would be brittle: it'd work on the exact torch version it was saved on, possibly break on newer/older versions, and definitely couldn't be loaded via safetensors.
What I did instead: Built anandkaman/controlmt-v2.3-int8 as a separate repo with the same bf16 weights (280 MB safetensors), but with a custom modeling_controlmt.py that contains a ControlMTForSeq2SeqLMInt8 subclass which auto-applies torch.quantization.quantize_dynamic in from_pretrained(). Users load it the standard HuggingFace way (AutoModelForSeq2SeqLM.from_pretrained("anandkaman/controlmt-v2.3-int8", trust_remote_code=True)) and get a CPU-int8 model with zero quantization code.
Pros of this approach over a pre-quantized file:
- Same weights file as the main repo (no version skew)
- Quantization happens fresh each load using the current torch version
- Standard HF auto_map dispatch — no custom load functions
Cons:
- Slight model-load overhead (~3s for the quant pass)
- Storage isn't actually reduced — both repos have 280 MB safetensors
Cost: ~1 hour to build the int8 repo with subclass approach. The pre-quantized-file approach would have been ~3 hours including the portability testing.
What I'd do differently: Nothing. The subclass approach is strictly better at this scale. Pre-quantization makes sense at scale (deployment-time quant cost matters when you're loading hundreds of models per second), but for a one-shot model load, runtime quantization wins.
7. Multi-token code-mix (the known v2.3 weakness — diagnosed, fix locked for v2.4)
What I observed during the competitor benchmark: ControlMT v2.3 translates ನಾನು ಬೆಂಗಳೂರಿನ Manyata Tech Park ನಲ್ಲಿ Software Engineer ಆಗಿ ಕೆಲಸ ಮಾಡುತ್ತೇನೆ. (three Latin-script tokens in a Kannada matrix) as "I will work as a software engineer at Girinagar Tech Park in Bangalore." — substituting "Manyata" with "Girinagar" (another Bangalore tech park's name). IndicTrans2 1.1B and Sarvam-Translate 4B both handle the same input correctly.
The single-token version (ನಾನು ಬೆಂಗಳೂರಿನ Manyata Tech Park ನಲ್ಲಿ ಕೆಲಸ ಮಾಡುತ್ತೇನೆ. — just one Latin entity) works fine.
Root cause hypothesis: The training corpus has lots of Kannada + 1 English entity pairs but few Kannada + 2+ English code-mix tokens pairs. At inference, when the source has high English-token density, the decoder's Kannada-language prior overwhelms the source-attention signal for the foreign tokens, and it substitutes nearest-by-phonetic-similarity from its Kannada-place-name distribution.
This isn't a "what didn't work" in the sense of a retracted experiment — it's a known limitation that's correctly attributed and has a planned fix. I'm listing it here for completeness: a public v2.3 ships with this gap, and if you're reading this thinking of using ControlMT in production, you should know about it before you hit it yourself.
The v2.4 plan: 50k+ synthetic + scraped pairs of the form Kannada matrix sentence + 2-4 Latin-script English tokens → English target preserving every Latin-script token verbatim. Locked in ../CHANGELOG.md under [Unreleased].
Pattern across all seven failures
Reading these back, the common thread is: most of my dead ends came from skipping the cheap empirical check.
- Style-tokens: didn't measure inference-time style differences during training (cheap qualitative check, would have caught the hollow-claim two weeks earlier)
- Gradio CSS battle: didn't read Gradio's default Row CSS before writing my own (cheap 5-min docs read, would have saved 2 hours)
- bitsandbytes int8: didn't think about custom-matmul incompatibility before trying (cheap mental check)
- v2.1 unreleased: didn't ask "is v2.1 already shipworthy?" before starting v2.2 (cheap framing question)
The pattern that breaks this: before committing to N hours of building Feature X, spend 30 minutes confirming the cheap-to-verify assumptions Feature X depends on. In every one of my failures above, a 30-minute check would have saved 2-100x its time.
This isn't unique to me. But noticing the pattern means the next time I'm about to commit hours, I default-stop and ask "what assumption am I about to commit on without checking?"