IndicBERT_WR

Model Description

IndicBERT_WR is a Telugu sentiment classification model built on IndicBERT (ai4bharat/indicBERTv2-MLM-only), a BERT-style Transformer model developed by AI4Bharat for Indian languages.

IndicBERT is pretrained on OSCAR and AI4Bharat-curated corpora spanning 12 Indian languages, including Telugu and English. The model is trained exclusively using the Masked Language Modeling (MLM) objective, focusing on learning high-quality, language-aware representations for Indian languages rather than cross-lingual alignment.

The suffix WR denotes With Rationale supervision. This model is fine-tuned using both sentiment labels and human-annotated rationales, enabling the model to align its predictions and explanations with human-identified evidence spans.

---

Pretraining Details

• Pretraining corpora:
  • OSCAR
  • AI4Bharat-curated Indian language corpora
• Training objective:
  • Masked Language Modeling (MLM)
• Language coverage: 12 Indian languages, including Telugu and English
• Code-mixed support: Not supported

---

Training Data

• Fine-tuning dataset: Telugu-Dataset
• Task: Sentiment classification
• Supervision type: Label + rationale supervision
• Rationales: Token-level human-annotated evidence spans

---

Rationale Supervision

During fine-tuning, human-provided rationales are incorporated alongside sentiment labels. In addition to the standard classification loss, an auxiliary rationale loss encourages the model’s attention or explanation scores to align with annotated rationale tokens.

This supervision improves:

• Alignment between model explanations and human judgment
• Plausibility of generated explanations
• Interpretability of sentiment predictions

---

Intended Use

This model is intended for:

• Explainable Telugu sentiment classification
• Rationale-supervised learning experiments
• Monolingual Telugu NLP research
• Comparative evaluation against label-only (WOR) baselines

IndicBERT_WR is particularly well-suited for monolingual Telugu tasks, where language-aware tokenization and embeddings are critical.

---

Performance Characteristics

Compared to label-only training, rationale supervision typically improves explanation plausibility while maintaining competitive sentiment classification performance.

Strengths

• Strong Telugu-specific representations
• Faster training compared to large multilingual models
• Human-aligned explanations through rationale supervision

Limitations

• Not designed for cross-lingual transfer learning
• Does not support code-mixed data
• Requires human-annotated rationales, increasing annotation cost

---

Use in Explainability Evaluation

IndicBERT_WR is suitable for evaluation with explanation frameworks such as FERRET, enabling:

• Faithfulness evaluation: How well explanations support the model’s predictions
• Plausibility evaluation: How closely explanations align with human rationales

---

References

• Joshi et al. (2022). IndicBERT. EMNLP.
• Marreddy et al. (2022).
• Duggenpudi et al. (2022).
• Rajalakshmi et al. (2023).