Configuration Parsing Warning:In config.json: "quantization_config.bits" must be an int
Inkling-2-Small EXL3 3.48 bpw (experts only)
An EXL3 quantization of thinkingmachines/Inkling-2-Small (594B total, 17B active). Only the routed MoE experts are quantized. Every other tensor is copied unchanged from the source checkpoint: attention, shared weights, router, embeddings, vision tower and the MTP head.
| Routed experts | EXL3 trellis, mul1 codebook, mixed 2.5–5 bits per layer, 3.48 bpw average |
| Everything else | Unchanged from source (BF16) |
| MTP / speculative head | Included unchanged (mtp.safetensors) |
| Total size | 270 GB, against 393.5 GB for the source |
Newer version: Inkling-2-Small-EXL3-4bpw is uniform 4.0 bpw with the mcg codebook, for Local Inference Lab's fused MoE kernels. It scores lower KLD (0.0213 vs 0.0240) and higher top-1 agreement (97.35% vs 96.82%), at 313 GB.
Loader status: Inkling-2 is not yet supported by stock exllamav3 or TabbyAPI. A model implementation is in progress and this card will be updated with run instructions once it lands. Until then these weights are only usable with that in-progress implementation.
Quality
KL divergence and next-token top-1 agreement against the source checkpoint, with its NVFP4/MXFP8 experts dequantized to BF16 weights. Measured on a 218K-token held-out set that does not overlap the calibration data. Lower KLD is better.
| Domain | Tokens | KLD (nats) | Top-1 agree | PPL source → quant |
|---|---|---|---|---|
| Agentic (multi-turn tool use) | 30,695 | 0.0518 | 96.35% | 3.042 → 3.056 |
| Tool calling | 18,420 | 0.0304 | 96.34% | 3.920 → 3.932 |
| Code | 15,388 | 0.0091 | 97.58% | 2.692 → 2.694 |
| Thinking / reasoning | 18,238 | 0.0085 | 96.08% | 3.204 → 3.201 |
| Chat | 7,019 | 0.0100 | 96.62% | 4.573 → 4.559 |
| Multilingual | 17,507 | 0.0273 | 94.59% | 5.708 → 5.742 |
| Wiki / general | 12,909 | 0.0258 | 95.10% | 5.578 → 5.604 |
| Long context (32K rows) | 98,301 | 0.0196 | 97.71% | 2.195 → 2.208 |
| All | 218,477 | 0.0240 | 96.82% | 2.951 → 2.963 (+0.4%) |
Most of the measured KLD is a numerical floor rather than quantization damage:
- Shrinking this quant's expert error 32-fold still leaves about 85% of the measured KLD.
- Running the source model with the NVFP4/MXFP8 activation quantization it was designed for shifts it by a similar amount relative to the BF16 reference (KLD ≈ 0.037).
- Compare quants on the same eval protocol; numbers from different eval sets are not comparable.
Bit allocation
Bits are set per layer and per projection, from measured sensitivity. Gate/up and down projections get separate widths. The full per-layer table, with per-layer reconstruction error, is in quantization_details.json.
| Layers | Gate / up bpw | Down bpw |
|---|---|---|
| 0–16 | 4 (3.5 on 5, 11) | 5 |
| 17–23 | 3.5 | 5 → 4 |
| 24–28 | 3 | 4 → 3.5 |
| 29–39 | 2.5 | 3.5 → 2.5 |
| 40–47 | 2.5 | 2.5–3 |
How it was made
- Source: the official checkpoint, where layers 0–39 experts are NVFP4 and layers 40–47 are MXFP8. Both were dequantized exactly, bit-checked against an independent decoder.
- Calibration: 4.17M tokens in 916 rows.
- About 60% is real multi-turn agentic and coding sessions with tool use.
- The rest is multilingual text, Wikipedia, reasoning, math, code, structured and function-calling data, and chat.
- 12% of tokens are in 16K–32K long-context rows.
- All secrets were redacted before use.
- Procedure: layer by layer on the quantized model's own activations, so each layer is calibrated on inputs that already include upstream quantization error. Each expert's Hessian comes only from the tokens the router actually sends to it, which averages about 80K tokens per expert. Rarely routed experts are shrunk toward the layer-wide covariance.
- Model math: verified against the reference implementation to about 1e-7 relative error in FP32, with 100% router top-8 agreement.
Running it
Coming soon. Inkling-2 support for exllamav3 is being written now, including batched decoding, multi-GPU, a sliding-window-aware quantized KV cache and MTP speculative decoding.
The weights are sized for 4× 96 GB GPUs. Only the 8 global-attention layers need a full-length KV cache; the 40 sliding-window layers keep a 1024-token ring. That leaves room for long contexts at high concurrency.
Credits
- Thinking Machines for Inkling-2-Small (Apache-2.0).
- turboderp for EXL3 / exllamav3.
- Downloads last month
- 107
Model tree for 0xSero/Inkling-2-Small-EXL3-3.48bpw
Base model
thinkingmachines/Inkling-2-Small