itzune/berteus-onnx

BERTeus ONNX (int8 + int4 quantized)

⚠️ This is an unofficial community conversion. This is not the original BERTeus repository. The original model was created by Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre at the IXA NLP Group (University of the Basque Country, UPV/EHU).

🔗 Original model: ixa-ehu/berteus-base-cased

📄 Paper: Give your Text Representation Models some Love: the Case for Basque (LREC 2020)

---

What is this?

This repository provides ONNX-quantized versions of BERTeus, a monolingual BERT model pre-trained on Basque (Euskara). Two quantization levels are available:

• int8 (model_quantized.onnx, 119 MB) — dynamic int8 quantization
• int4 (model_q4.onnx, 85 MB) — weight-only int4 (MatMul) + int8 (embeddings)

Both enable efficient inference in:

• JavaScript / TypeScript via @huggingface/transformers (Transformers.js) — runs in the browser via WebAssembly
• Python via optimum with ONNX Runtime
• Any runtime that supports the ONNX format

Why use this instead of the original?

	Original (ixa-ehu/berteus-base-cased)	This repo (itzune/berteus-onnx)
Format	PyTorch / Safetensors	ONNX (int8 or int4 quantized)
Size	~470 MB	85 MB (int4) or 119 MB (int8) + 74 MB (embeddings)
Python (PyTorch)	✅	—
Python (ONNX Runtime)	—	✅
Browser (WASM)	❌	✅ via Transformers.js
Speed (CPU)	Baseline	~2–4× faster (quantized + ONNX Runtime)

The original PyTorch model remains the best choice for training, fine-tuning, or GPU inference. Use this ONNX version when you need browser deployment, small download size, or fast CPU inference.

int4 vs int8 — which should I use?

	int4 (model_q4.onnx)	int8 (model_quantized.onnx)
File size	85 MB	119 MB
Transformers.js dtype	'q4'	'q8'
Quality (benchmark)	+110 net (85.5% accuracy)	+105 net (85.0% accuracy)
Ranking agreement vs fp32	96.7%	96.7%
Mean score diff vs fp32	0.011	0.013
Browser load time	~2.8s	~3.1s

Recommendation: Use int4 (dtype: 'q4') for browser deployment — it is smaller, faster to load, and produces equivalent or better quality on the re-ranking benchmark. Use int8 if you need maximum backward compatibility with older Transformers.js versions.

---

Model details

Property	Value
Architecture	BERT (encoder only, BertModel)
Parameters	~110M (encoder) + ~39M (embeddings)
Hidden size	768
Layers	12
Attention heads	12
Vocab size	50,099
Max sequence length	512
Training corpus	224.6M tokens (Basque news + Wikipedia)
Quantization	int4 (weight-only, MatMul) + int8 (embeddings), or int8 (dynamic)

Note: The original BERTeus checkpoint was saved as BertModel (encoder only) — it does not include the masked LM (MLM) prediction head. This means BertForMaskedLM will produce random MLM outputs. For masked LM tasks, use the static word embeddings approach described below, or fine-tune an MLM head on the original model.

Quantization details

int4 (model_q4.onnx): Two-step quantization using ONNX Runtime's MatMulNBitsQuantizer:

1. Encoder MatMul ops → int4 (weight-only, block_size=128, asymmetric) via MatMulNBits operator
2. Embedding Gather ops → int8 (QDQ: DequantizeLinear) via dynamic quantization

This approach keeps the embedding table in int8 (not int4) to avoid GatherBlockQuantized, which is not implemented on the WASM backend. The MatMulNBits operator IS supported on WASM in Transformers.js v3+.

int8 (model_quantized.onnx): Standard dynamic int8 quantization using MatMulInteger + DynamicQuantizeLinear operators.

---

Files

File	Size	Description
onnx/model_q4.onnx	85 MB	int4 quantized ONNX model (recommended for browser)
onnx/model_quantized.onnx	119 MB	int8 quantized ONNX model
word_embeddings_f16.bin	74 MB	Word embedding matrix (50,099 × 768) in float16
config.json	578 B	Model configuration
tokenizer.json	1.2 MB	Fast tokenizer (Transformers.js compatible)
tokenizer_config.json	1.3 KB	Tokenizer configuration
special_tokens_map.json	125 B	Special tokens mapping
vocab.txt	413 KB	WordPiece vocabulary
embedding_meta.json	187 B	Embedding matrix metadata (vocab size, token IDs)

Why a separate embedding file?

The ONNX model outputs contextual hidden states, but the static word embedding matrix (the input embedding layer) is useful for tasks like lexical similarity and candidate scoring. We extract it as a separate float16 binary file (74 MB) to avoid loading the full model when only embeddings are needed.

---

Usage

Python (ONNX Runtime via Optimum)

from optimum.onnxruntime import ORTModelForFeatureExtraction
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("itzune/berteus-onnx")
model = ORTModelForFeatureExtraction.from_pretrained("itzune/berteus-onnx")

inputs = tokenizer("Euskara munduko hizkuntzarik ederrena da.", return_tensors="pt")
outputs = model(**inputs)

# Contextual embeddings: [1, seq_len, 768]
hidden_states = outputs.last_hidden_state

JavaScript / Browser (Transformers.js)

int4 (recommended — smaller, faster):

import { AutoModel, AutoTokenizer, env } from '@huggingface/transformers';

const model = await AutoModel.from_pretrained('itzune/berteus-onnx', {
  dtype: 'q4',    // int4 quantization — 85 MB
  device: 'wasm',
});

const tokenizer = await AutoTokenizer.from_pretrained('itzune/berteus-onnx');

const inputs = tokenizer('Euskara mundiko hizkuntzarik ederrena da.', {
  truncation: true,
  max_length: 512,
  padding: false,
});

const outputs = await model(inputs);
// outputs.last_hidden_state.data → Float32Array of shape [1, seq_len, 768]

int8 (alternative — slightly larger):

const model = await AutoModel.from_pretrained('itzune/berteus-onnx', {
  dtype: 'q8',    // int8 quantization — 119 MB
  device: 'wasm',
});

Using the embedding matrix (lexical similarity)

import numpy as np
import struct

# Load float16 embeddings
with open("word_embeddings_f16.bin", "rb") as f:
    raw = np.frombuffer(f.read(), dtype=np.float16)

embeddings = raw.reshape(50099, 768).astype(np.float32)

# Get embedding for a token ID
token_id = tokenizer.convert_tokens_to_ids("euskara")
word_embedding = embeddings[token_id]  # 768-dim vector

// Browser: load and convert float16 → float32
const response = await fetch('word_embeddings_f16.bin');
const buffer = await response.arrayBuffer();
const uint16 = new Uint16Array(buffer);
const float32 = new Float32Array(uint16.length);
// Convert each float16 → float32 (bit manipulation, see source)

---

Use cases

✅ What this model is good for

Use case	How
Contextual embeddings	Run the encoder, extract last_hidden_state for any token
Sentence embeddings	Mean-pool the last_hidden_state across tokens
Lexical similarity	Compare static word embeddings (from the .bin file)
Candidate re-ranking	Mask a position, compare [MASK] hidden state against candidate embeddings (cosine similarity)
Feature extraction	Feed hidden states to a downstream classifier (NER, POS, sentiment)
Text classification	Use [CLS] token output as a sentence representation

❌ What this model cannot do

Limitation	Why
Masked LM fill-in-the-blank	The checkpoint has no MLM head (cls weights missing). Use the embedding similarity approach instead.
Text generation	BERT is encoder-only, not a generative model.
Fine-tuning	ONNX is an inference format. Fine-tune the original PyTorch model instead.

Masked embedding similarity (no MLM head needed)

Since the checkpoint lacks an MLM head, the recommended approach for "which word fits here?" tasks is:

1. Replace the target position with [MASK] (token ID 4)
2. Run the BERT encoder on the full sentence (bidirectional context)
3. Extract the [MASK] position's hidden state (768-dim)
4. For each candidate word, compute its static embedding (mean of subword piece embeddings)
5. Score by cosine similarity between the [MASK] hidden state and the candidate embedding

This is based on the lexical substitution approach from Paetzold & Specia (2017) and works well for spell-checker candidate re-ranking.

---

Performance

int4 vs int8 vs fp32 (933-case spell-correction re-ranking benchmark)

Model	Size	Best weight	Accuracy	Net improvement	Mean score diff vs fp32
PyTorch fp32	496 MB	w=12	84.2%	+99	—
ONNX int8	119 MB	w=15	85.0%	+105	0.013
ONNX int4	85 MB	w=18	85.5%	+110	0.011

Both quantized variants slightly outperform fp32 PyTorch on this benchmark — quantization noise occasionally helps break ties in the re-ranking task. The int4 model is the best overall: smallest, fastest, and highest accuracy.

Browser validation (30-case subset, Transformers.js WASM)

Variant	Model load	T2 accuracy	Match Python ranking	Mean score diff
int4 (q4)	2.8s	80.0%	96.7%	0.011
int8 (q8)	3.1s	73.3%	96.7%	0.013

The float16 embedding matrix has negligible precision loss compared to float32.

---

Citation

If you use this model, please cite the original BERTeus paper:

@inproceedings{agerri2020give,
  title     = {Give your Text Representation Models some Love: the Case for Basque},
  author    = {Rodrigo Agerri and Iñaki San Vicente and Jon Ander Campos and Ander Barrena and Xabier Saralegi and Aitor Soroa and Eneko Agirre},
  booktitle = {Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC)},
  year      = {2020}
}

---

License

The original BERTeus model is released under the Apache 2.0 license. This ONNX conversion is provided under the same license.

Acknowledgements

All credit for the BERTeus model goes to the IXA NLP Group at the University of the Basque Country (UPV/EHU). This repository is merely a format conversion for deployment convenience.