---
library_name: transformers
license: apache-2.0
---

# BERnaT: Basque Encoders for Representing Natural Textual Diversity

Submitted to LREC 2026

## Model Description

BERnaT is a family of monolingual Basque encoder-only language models trained to better represent linguistic variation—including standard, dialectal, historical, and informal Basque—rather than focusing solely on standard textual corpora. Models were trained on corpora that combine high-quality standard Basque with varied sources such as social media and historical texts, aiming to enhance robustness and generalization across natural language understanding (NLU) tasks.

- **Developed by:** HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU)
- **Funded by:** Ikergaitu and ALIA projects (Basque and Spanish Government)
- **License:** Apache 2.0
- **Model Type**: Encoder-only Transformer models (RoBERTa-style)
- **Languages**: Basque (Euskara)


## Getting Started

You can either use this model directly as the example below, or fine-tune it to your task of interest.

```python
>>> from transformers import pipeline
>>> pipe = pipeline("fill-mask", model='HiTZ/BERnaT-base')
>>> pipe("Kaixo! Ni <mask> naiz!")
[{'score': 0.022003261372447014,
  'token': 7497,
  'token_str': ' euskalduna',
  'sequence': 'Kaixo! Ni euskalduna naiz!'},
 {'score': 0.016429167240858078,
  'token': 14067,
  'token_str': ' Olentzero',
  'sequence': 'Kaixo! Ni Olentzero naiz!'},
 {'score': 0.012804778292775154,
  'token': 31087,
  'token_str': ' ahobizi',
  'sequence': 'Kaixo! Ni ahobizi naiz!'},
 {'score': 0.01173020526766777,
  'token': 331,
  'token_str': ' ez',
  'sequence': 'Kaixo! Ni ez naiz!'},
 {'score': 0.010091394186019897,
  'token': 7618,
  'token_str': ' irakaslea',
  'sequence': 'Kaixo! Ni irakaslea naiz!'}]
```

## Training Data

The BERnaT family was pre-trained on a combination of:
- Standard Basque corpora (e.g., Wikipedia, Egunkaria, EusCrawl).
- Diverse corpora including Basque social media text and historical Basque books.
- Combined corpora for the unified BERnaT models.

Training objective is masked language modeling (MLM) on encoder-only architectures across medium (51M), base (124M), and large (355M) sizes.

## Evaluation

|                     | **AVG standard tasks** | **AVG diverse tasks** | **AVG overall** |
|---------------------|:----------------------:|:---------------------:|:---------------:|
| **BERnaT_standard** |                        |                       |                 |
| medium              |          74.10         |         70.30         |      72.58      |
| base                |          75.33         |         71.26         |      73.70      |
| large               |          76.83         |         73.13         |      75.35      |
| **BERnaT_diverse**  |                        |                       |                 |
| medium              |          71.66         |         69.91         |      70.96      |
| base                |          72.44         |         71.43         |      72.04      |
| large               |          74.48         |         71.87         |      73.43      |
| **BERnaT**          |                        |                       |                 |
| medium              |          73.56         |         70.59         |      72.37      |
| base                |          75.42         |         71.28         |      73.76      |
| large               |        **77.88**       |       **73.77**       |    **76.24**    |

## Acknowledgments

This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project). The project also received funding from the European Union’s Horizon Europe research and innovation program under Grant Agreement No 101135724, Topic HORIZON-CL4-2023-HUMAN-01-21 and DeepKnowledge (PID2021-127777OB-C21) founded by MCIN/AEI/10.13039/501100011033 and FEDER. Jaione Bengoetxea, Julen Etxaniz and Ekhi Azurmendi hold a PhD grant from the Basque Government (PRE_2024_1_0028, PRE_2024_2_0028 and PRE_2024_1_0035, respectively). Maite Heredia and Mikel Zubillaga hold a PhD grant from the University of the Basque Country UPV/EHU (PIF23/218 and PIF24/04, respectively). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2024E01-042.

## Citation:

To cite our work, please use:

```bibtex
@misc{azurmendi2025bernatbasqueencodersrepresenting,
      title={BERnaT: Basque Encoders for Representing Natural Textual Diversity}, 
      author={Ekhi Azurmendi and Joseba Fernandez de Landa and Jaione Bengoetxea and Maite Heredia and Julen Etxaniz and Mikel Zubillaga and Ander Soraluze and Aitor Soroa},
      year={2025},
      eprint={2512.03903},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.03903}, 
}
```