mBERT_WR

Model Description

mBERT_WR is a Telugu sentiment classification model built on Google’s BERT-base-multilingual-cased (mBERT) architecture. The base model consists of 12 Transformer encoder layers with approximately 110 million parameters and is pretrained on Wikipedia text from 104 languages, including Telugu, using Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) objectives.

The suffix WR stands for With Rationale supervision. This model is trained using both sentiment labels and human-annotated rationales, enabling the model to align its internal attention and predictions with human-identified evidence spans.

---

Pretraining Details

• Pretraining corpus: Multilingual Wikipedia (104 languages)
• Training objectives:
  • Masked Language Modeling (MLM)
  • Next Sentence Prediction (NSP)
• Language coverage: Telugu is included, but the model is not exclusively trained on Telugu data

---

Training Data

• Fine-tuning dataset: Telugu-Dataset
• Task: Sentiment classification
• Supervision type: Label + rationale supervision
• Rationales: Token-level human-annotated evidence spans

---

Rationale Supervision

During fine-tuning, the model incorporates human-provided rationales to guide learning. In addition to the standard classification loss, an auxiliary rationale loss is applied to encourage the model’s attention or explanation scores to align with annotated rationale tokens.

This supervision improves:

• Interpretability of predictions
• Alignment between model explanations and human judgment
• Plausibility of generated explanations

---

Intended Use

This model is intended for:

• Explainable Telugu sentiment classification
• Rationale-supervised learning experiments
• Cross-lingual explainability research
• Comparison with label-only (WOR) baselines

The model is especially suitable for studies that require both predictive performance and human-aligned explanations.

---

Performance Characteristics

Compared to label-only training, rationale supervision typically improves explanation plausibility while maintaining competitive classification performance.

Strengths

• Human-aligned explanations via rationale supervision
• Improved plausibility scores in explanation evaluation
• Suitable for explainable AI research

Limitations

• Requires annotated rationales, increasing data and annotation cost
• Classification gains may be modest compared to WOR models
• Not specifically optimized for Telugu morphology or syntax

---

Use in Explainability Evaluation

mBERT_WR is designed for evaluation with frameworks such as FERRET, enabling:

• Faithfulness analysis: How well explanations support the model’s own predictions
• Plausibility analysis: How closely explanations match human rationales

This makes the model suitable for rigorous explainability benchmarking in low-resource Telugu NLP.

---

References

• Devlin et al., 2019
• Hedderich et al., 2021
• Wu and Dredze, 2020
• Kalyan et al., 2021
• Marreddy et al., 2021, 2022
• Duggenpudi et al., 2022
• Rajalakshmi et al., 2023