Upload fine-tuned XTTS-v2 for Wolof (3 epochs, A100)

README.md

@@ -0,0 +1,202 @@
+---
+language:
+  - wo
+  - fr
+  - en
+license: other
+license_name: coqui-public-model-license
+license_link: https://coqui.ai/cpml
+base_model: coqui/XTTS-v2
+tags:
+  - text-to-speech
+  - tts
+  - wolof
+  - xtts
+  - coqui
+  - fine-tuned
+  - african-languages
+  - senegal
+datasets:
+  - WaxalNLP/wolof_speech
+  - google/fleurs
+  - keithito/lj_speech
+library_name: coqui-tts
+pipeline_tag: text-to-speech
+---
+
+# XTTS-v2 Fine-tuned for Wolof 🇸🇳
+
+A fine-tuned version of [Coqui XTTS-v2](https://huggingface.co/coqui/XTTS-v2) with improved Wolof language support. This model was fine-tuned to bring high-quality text-to-speech synthesis to the **Wolof language** — one of the most widely spoken languages in Senegal and West Africa.
+
+## Model Description
+
+XTTS-v2 is a multilingual text-to-speech model that supports voice cloning. This fine-tuned version enhances performance on Wolof while retaining capabilities in French and English.
+
+| Property | Value |
+|----------|-------|
+| **Base Model** | [coqui/XTTS-v2](https://huggingface.co/coqui/XTTS-v2) |
+| **Architecture** | GPT-2 based encoder + HiFi-GAN decoder |
+| **Parameters** | ~467M |
+| **Audio Sample Rate** | 24,000 Hz |
+| **Model Size** | 1.7 GB |
+
+## Training Details
+
+### Datasets
+
+The model was fine-tuned on a curated multilingual dataset of **10,861 audio samples**:
+
+| Dataset | Language | Samples | Description |
+|---------|----------|---------|-------------|
+| [WaxalNLP/wolof_speech](https://huggingface.co/datasets/WaxalNLP/wolof_speech) | Wolof | 1,042 | Wolof speech corpus |
+| [google/fleurs](https://huggingface.co/datasets/google/fleurs) (wo) | Wolof | 2,819 | FLEURS Wolof split |
+| [google/fleurs](https://huggingface.co/datasets/google/fleurs) (fr) | French | 2,000 | FLEURS French (subset) |
+| [google/fleurs](https://huggingface.co/datasets/google/fleurs) (en) | English | 2,000 | FLEURS English (subset) |
+| [keithito/lj_speech](https://huggingface.co/datasets/keithito/lj_speech) | English | 3,000 | LJSpeech (subset) |
+
+French and English data was included to prevent catastrophic forgetting of multilingual capabilities during fine-tuning.
+
+### Fine-tuning Technique
+
+- **Method**: GPT encoder fine-tuning (Stage 1) using `GPTTrainer` from Coqui TTS 0.22.0
+- **All 898 parameters** were trained (full fine-tuning, no freezing)
+- **Optimizer**: AdamW with learning rate 5e-6
+- **Batch size**: 4 (effective batch size 16 with gradient accumulation of 4)
+- **Epochs**: 3 (7,740 total steps)
+- **Precision**: FP32 (mixed precision disabled — FP16 caused NaN losses on A100)
+- **Hardware**: NVIDIA A100-SXM4-40GB
+- **Training loss**: 0.924 → 0.767
+- **Best eval loss**: 3.049
+
+### Key Training Decisions
+
+1. **FP32 over FP16**: Mixed precision training produced NaN losses with TTS 0.22.0 on A100 GPUs. FP32 training was stable and produced valid gradients throughout.
+2. **Multilingual data mix**: Including French, English, and LJSpeech alongside Wolof prevented the model from losing its multilingual voice cloning ability.
+3. **Low learning rate (5e-6)**: A conservative learning rate preserved the pre-trained model's strengths while allowing adaptation to Wolof phonology.
+
+## Evaluation Results
+
+Comparison against the original XTTS-v2 base model on 9 test sentences (5 Wolof, 2 French, 2 English):
+
+### Overall
+
+| Metric | Fine-tuned | Original | Δ |
+|--------|-----------|----------|---|
+| Speaker Similarity (↑) | **0.8273** | 0.8175 | +0.0099 |
+| Wins (speaker match) | **7/9** | 2/9 | — |
+
+### Per-Language Speaker Similarity
+
+| Language | Fine-tuned | Original | Δ | FT Win Rate |
+|----------|-----------|----------|---|-------------|
+| **Wolof** 🟢 | **0.8396** | 0.8193 | **+0.0203** | 5/5 (100%) |
+| **French** 🟢 | **0.8266** | 0.8187 | +0.0080 | 1/2 |
+| **English** 🔴 | 0.7974 | **0.8116** | -0.0142 | 1/2 |
+
+**Key finding**: The fine-tuned model shows a **+2% improvement in speaker similarity for Wolof** while maintaining competitive performance in French and English.
+
+## Usage
+
+### With Coqui TTS
+
+```python
+from TTS.api import TTS
+import torch
+
+# Load model
+tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2", gpu=False)
+
+# Override with fine-tuned weights
+model_path = "path/to/model.pth"
+checkpoint = torch.load(model_path, map_location="cpu", weights_only=False)
+tts.synthesizer.tts_model.load_state_dict(checkpoint["model"], strict=True)
+
+# Generate Wolof speech
+tts.tts_to_file(
+    text="Jàmm nga fanaan. Nanga def?",
+    speaker_wav="reference_audio.wav",
+    language="fr",  # Wolof maps to French in XTTS-v2
+    file_path="output.wav"
+)
+```
+
+### Direct with XttsModel
+
+```python
+from TTS.tts.configs.xtts_config import XttsConfig
+from TTS.tts.models.xtts import Xtts
+import torch
+
+config = XttsConfig()
+config.load_json("config.json")
+model = Xtts.init_from_config(config)
+model.load_checkpoint(config, checkpoint_path="model.pth", vocab_path="vocab.json")
+model.eval()
+
+outputs = model.synthesize(
+    text="Jàmm nga fanaan.",
+    config=config,
+    speaker_wav="reference.wav",
+    language="fr",
+)
+```
+
+## Files
+
+| File | Size | Description |
+|------|------|-------------|
+| `model.pth` | 1.7 GB | Fine-tuned model weights (wrapped with `{"model": state_dict}`) |
+| `config.json` | 4.3 KB | Model configuration |
+| `vocab.json` | 353 KB | Tokenizer vocabulary |
+| `speakers_xtts.pth` | 7.4 MB | Speaker embeddings |
+
+## Limitations
+
+- **Wolof is mapped to French** (`language="fr"`) since XTTS-v2 has no native Wolof language token. This works well because Wolof and French share phonological similarities in the Senegalese context.
+- English speaker similarity shows a slight regression (-1.4%) compared to the base model.
+- The model was fine-tuned for 3 epochs only — longer training or larger Wolof datasets could yield further improvements.
+- Voice quality depends on the reference audio provided for cloning.
+
+## Citation
+
+If you use this model, please cite the original XTTS-v2 and the datasets:
+
+```bibtex
+@misc{xtts-v2-wolof,
+  title={XTTS-v2 Fine-tuned for Wolof},
+  author={Muhamad Ul},
+  year={2026},
+  url={https://huggingface.co/muhamadul/xtts-v2-wolof}
+}
+
+@misc{casanova2024xtts,
+  title={XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model},
+  author={Casanova, Edresson and others},
+  year={2024},
+  publisher={Coqui AI}
+}
+
+@misc{waxalnlp-wolof,
+  title={Wolof Speech Dataset},
+  author={WaxalNLP},
+  url={https://huggingface.co/datasets/WaxalNLP/wolof_speech}
+}
+
+@inproceedings{conneau2023fleurs,
+  title={FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech},
+  author={Conneau, Alexis and others},
+  booktitle={IEEE SLT},
+  year={2023}
+}
+```
+
+## Acknowledgments
+
+- **[Coqui AI](https://coqui.ai/)** for the XTTS-v2 base model and TTS framework
+- **[WaxalNLP](https://huggingface.co/WaxalNLP)** for the Wolof speech dataset
+- **[Google FLEURS](https://huggingface.co/datasets/google/fleurs)** for multilingual speech data
+- **[Daande](https://daande.aliune.com)** — The TTS studio powered by this model, built for Wolof speakers
+
+---
+
+*Daande (ދައންދެ) means "voice" in Wolof. This project aims to bring modern AI voice technology to West African languages.*

config.json

@@ -0,0 +1,159 @@
+{
+    "output_path": "output",
+    "logger_uri": null,
+    "run_name": "run",
+    "project_name": null,
+    "run_description": "\ud83d\udc38Coqui trainer run.",
+    "print_step": 25,
+    "plot_step": 100,
+    "model_param_stats": false,
+    "wandb_entity": null,
+    "dashboard_logger": "tensorboard",
+    "save_on_interrupt": true,
+    "log_model_step": null,
+    "save_step": 10000,
+    "save_n_checkpoints": 5,
+    "save_checkpoints": true,
+    "save_all_best": false,
+    "save_best_after": 10000,
+    "target_loss": null,
+    "print_eval": false,
+    "test_delay_epochs": 0,
+    "run_eval": true,
+    "run_eval_steps": null,
+    "distributed_backend": "nccl",
+    "distributed_url": "tcp://localhost:54321",
+    "mixed_precision": false,
+    "precision": "fp16",
+    "epochs": 1000,
+    "batch_size": 32,
+    "eval_batch_size": 16,
+    "grad_clip": 0.0,
+    "scheduler_after_epoch": true,
+    "lr": 0.001,
+    "optimizer": "radam",
+    "optimizer_params": null,
+    "lr_scheduler": null,
+    "lr_scheduler_params": {},
+    "use_grad_scaler": false,
+    "allow_tf32": false,
+    "cudnn_enable": true,
+    "cudnn_deterministic": false,
+    "cudnn_benchmark": false,
+    "training_seed": 54321,
+    "model": "xtts",
+    "num_loader_workers": 0,
+    "num_eval_loader_workers": 0,
+    "use_noise_augment": false,
+    "audio": {
+        "sample_rate": 22050,
+        "output_sample_rate": 24000
+    },
+    "use_phonemes": false,
+    "phonemizer": null,
+    "phoneme_language": null,
+    "compute_input_seq_cache": false,
+    "text_cleaner": null,
+    "enable_eos_bos_chars": false,
+    "test_sentences_file": "",
+    "phoneme_cache_path": null,
+    "characters": null,
+    "add_blank": false,
+    "batch_group_size": 0,
+    "loss_masking": null,
+    "min_audio_len": 1,
+    "max_audio_len": Infinity,
+    "min_text_len": 1,
+    "max_text_len": Infinity,
+    "compute_f0": false,
+    "compute_energy": false,
+    "compute_linear_spec": false,
+    "precompute_num_workers": 0,
+    "start_by_longest": false,
+    "shuffle": false,
+    "drop_last": false,
+    "datasets": [
+        {
+            "formatter": "",
+            "dataset_name": "",
+            "path": "",
+            "meta_file_train": "",
+            "ignored_speakers": null,
+            "language": "",
+            "phonemizer": "",
+            "meta_file_val": "",
+            "meta_file_attn_mask": ""
+        }
+    ],
+    "test_sentences": [],
+    "eval_split_max_size": null,
+    "eval_split_size": 0.01,
+    "use_speaker_weighted_sampler": false,
+    "speaker_weighted_sampler_alpha": 1.0,
+    "use_language_weighted_sampler": false,
+    "language_weighted_sampler_alpha": 1.0,
+    "use_length_weighted_sampler": false,
+    "length_weighted_sampler_alpha": 1.0,
+    "model_args": {
+        "gpt_batch_size": 1,
+        "enable_redaction": false,
+        "kv_cache": true,
+        "gpt_checkpoint": null,
+        "clvp_checkpoint": null,
+        "decoder_checkpoint": null,
+        "num_chars": 255,
+        "tokenizer_file": "",
+        "gpt_max_audio_tokens": 605,
+        "gpt_max_text_tokens": 402,
+        "gpt_max_prompt_tokens": 70,
+        "gpt_layers": 30,
+        "gpt_n_model_channels": 1024,
+        "gpt_n_heads": 16,
+        "gpt_number_text_tokens": 6681,
+        "gpt_start_text_token": null,
+        "gpt_stop_text_token": null,
+        "gpt_num_audio_tokens": 1026,
+        "gpt_start_audio_token": 1024,
+        "gpt_stop_audio_token": 1025,
+        "gpt_code_stride_len": 1024,
+        "gpt_use_masking_gt_prompt_approach": true,
+        "gpt_use_perceiver_resampler": true,
+        "input_sample_rate": 22050,
+        "output_sample_rate": 24000,
+        "output_hop_length": 256,
+        "decoder_input_dim": 1024,
+        "d_vector_dim": 512,
+        "cond_d_vector_in_each_upsampling_layer": true,
+        "duration_const": 102400
+    },
+    "model_dir": null,
+    "languages": [
+        "en",
+        "es",
+        "fr",
+        "de",
+        "it",
+        "pt",
+        "pl",
+        "tr",
+        "ru",
+        "nl",
+        "cs",
+        "ar",
+        "zh-cn",
+        "hu",
+        "ko",
+        "ja",
+        "hi"
+    ],
+    "temperature": 0.75,
+    "length_penalty": 1.0,
+    "repetition_penalty": 5.0,
+    "top_k": 50,
+    "top_p": 0.85,
+    "num_gpt_outputs": 1,
+    "gpt_cond_len": 30,
+    "gpt_cond_chunk_len": 4,
+    "max_ref_len": 30,
+    "sound_norm_refs": false
+}

model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43f387a3bd6d2dadcb61c641fe6c7aa528e667d670ed01e9953c589e2012c72d
+size 1867854631

speakers_xtts.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0f6137c19a4eab0cbbe4c99b5babacf68b1746e50da90807708c10e645b943b
+size 7754818