# Sarvam-30B 4-Bit (BitsAndBytes)

This repository provides a **4-bit NF4 quantized version** of the base model **`sarvamai/sarvam-30b`** using **bitsandbytes**.
The quantization significantly reduces GPU memory usage while preserving strong inference performance.

Base model
`sarvamai/sarvam-30b`

Architecture
`SarvamMoEForCausalLM`

---

# Quantization Details

Quantization method: **BitsAndBytes 4-bit (NF4)**

Configuration used:

* load_in_4bit = True
* bnb_4bit_quant_type = nf4
* bnb_4bit_compute_dtype = float16
* bnb_4bit_use_double_quant = True

Approximate GPU memory usage:

| Model         | GPU VRAM  |
| ------------- | --------- |
| FP16 original | ~60 GB    |
| 4-bit NF4     | ~16-18 GB |

This version is recommended for most users who want to run the model with reduced hardware requirements.

---

# Installation

Install the required libraries.

```bash
pip install transformers accelerate bitsandbytes torch safetensors
```

---

# Loading the Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "neuralnets/sarvam-30b-4bit",
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(
    "neuralnets/sarvam-30b-4bit",
    trust_remote_code=True
)
```

---

# Example Inference

```python
prompt = "Explain mixture of experts in simple terms."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

# Hardware Requirements

Recommended GPUs:

* A100 40GB or 80GB
* RTX 4090
* RTX 3090 (with offloading)

CPU RAM recommendation:

* 32 GB or higher

---

# Notes

* This model uses **bitsandbytes quantization** integrated into Hugging Face Transformers.
* The Sarvam architecture requires `trust_remote_code=True`.
* Designed primarily for **inference workloads**.

---

# Base Model

Original model:

`sarvamai/sarvam-30b`

Please refer to the base repository for model training details and benchmarks.

---

# License

This repository distributes a **quantized derivative** of the original model.

Users must follow the license of the upstream model:

`sarvamai/sarvam-30b`