vietnamese-correction-lora-v2

This model is a fine-tuned version of vinai/bartpho-syllable on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1776123046875
• Sacrebleu: 25.07128550273525
• Precision: 0.9230769230769231
• Recall: 0.6
• F1 Score: 0.7272727272727274
• eval_samples_per_second: 6.776

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: 800000
    })
    val: Dataset({
        features: ['input', 'output'],
        num_rows: 200000
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: 40000
    })
})

Training procedure

The following bitsandbytes quantization config was used during training:

• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: float16

Training hyperparameters

The following hyperparameters were used during training:

• trainable params: 25,165,824 || all params: 326,801,408 || trainable%: 7.700647360735974
• Num examples = 800,000
• Num Epochs = 2
• Instantaneous batch size per device = 12
• Total train batch size (w. parallel, distributed & accumulation) = 72
• Gradient Accumulation steps = 6
• Total optimization steps = 22,222
• Number of trainable parameters = 25,165,824

Framework versions

• PEFT 0.4.0
• PEFT 0.14.0
• Transformers 4.47.0
• Pytorch 2.5.1+cu121
• Datasets 3.3.1
• Tokenizers 0.21.0

---

library_name: peft

Training procedure

The following bitsandbytes quantization config was used during training:

• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: float16

Framework versions

• PEFT 0.4.0