README.md

---
license: apache-2.0
base_model: google/gemma-4-12B-it
library_name: peft
language:
- en
tags:
- lora
- peft
- base_model:adapter:google/gemma-4-12B-it
- decision-making
- calibration
datasets:
- stanfordnlp/snli
- allenai/ai2_arc
- hotpotqa/hotpot_qa
- rajpurkar/squad_v2
- theatticusproject/cuad
- PolyAI/banking77
- allenai/qasc
- Salesforce/xlam-function-calling-60k
- clinc/clinc_oos
- tasksource/esci
- glaiveai/glaive-function-calling-v2
- NousResearch/hermes-function-calling-v1
- Team-ACE/ToolACE
- liminghao1630/API-Bank
- tau/commonsense_qa
---

# Hopper 12B (trained)

This branch (`hopper-12b`) of `HopitAI/hopper` holds **Hopper 12B (trained)**: a PEFT LoRA adapter for the frozen
[`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it) base at revision
`707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7`, and the readout map fitted for it. The repository's `main` branch holds
a different model, the Qwen3.5-4B Hopper adapter, which is unchanged.

The model scores the options of a structured decision request: a state, a question and a fixed set of options go in,
and a probability distribution over the options comes out. It does not generate text. It is served by the Hopper
serving code, [github.com/hopit-ai/hopper](https://github.com/hopit-ai/hopper), tag `12b-1.1.0` (package 1.2.0), which
speaks the JevBench `/v1/systemone` wire format.

## Files

| file | what it is |
| --- | --- |
| `adapter_config.json` | PEFT LoRA configuration (rank 32, alpha 64, dropout 0) |
| `adapter_model.safetensors` | adapter weights, 524,648,560 bytes |
| `hopper-12b-trained-readout.json` | the sealed `readout-bias/readout-seal-v2` readout map (seal `65c72f717a74eccfc38e3906345927765a00d8e0fdd8c61e8658c55eec34c818`); byte-identical to the copy shipped in the serving package |
| `SHA256SUMS` | SHA-256 checksums of the three files above |

## How to run it

```sh
pip install "git+https://github.com/hopit-ai/hopper@12b-1.1.0"
hopper-serve --base-loader gemma-4-12b-it \
  --adapter HopitAI/hopper@4b605f25eb903c2cf4d121040ffb097240b3bc37 \
  --readout-map "$(python -c 'import hopper_decisions, pathlib; print(pathlib.Path(hopper_decisions.__file__).parent / "maps/hopper-12b-trained-readout.json")')" \
  --port 8080
```

Always pin the revision: `HopitAI/hopper` without one resolves to `main`, the Qwen3.5-4B adapter, which does not fit
this base. To verify the files and serve a local copy instead:

```sh
huggingface-cli download HopitAI/hopper --revision 4b605f25eb903c2cf4d121040ffb097240b3bc37 --local-dir hopper-12b
(cd hopper-12b && shasum -a 256 -c SHA256SUMS)
hopper-serve --base-loader gemma-4-12b-it --adapter ./hopper-12b \
  --readout-map ./hopper-12b/hopper-12b-trained-readout.json --port 8080
```

## Architecture and serving

- **Base:** `google/gemma-4-12B-it` at revision `707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7`, frozen, bf16.
- **Adapter:** LoRA rank 32, alpha 64, on the seven text projections `q_proj`, `k_proj`, `v_proj`, `o_proj`,
  `gate_proj`, `up_proj` and `down_proj`.
- **Keep it unmerged.** A bf16 merge of the adapter changed 2.1% of top choices on 1,932 decisions, so the serving code
  loads it as a separate low-rank path, as it was evaluated. A merged copy is not decision-equivalent and is not
  supported.
- **Readout:** one forward pass per decision; the answer is a softmax over the option letters. The readout map then
  applies position priors for original two- and four-option choice menus (fitted on 240 and 74 menus) and one
  temperature, 1.65. Priors do not apply to `noul` or score questions or to other menu sizes.
- **Long menus** (more than 26 options) use the serving code's seeded tournament shortlist (`k=10`, residual mass
  `0.05`): qualification passes use raw temperature-1 scores, and the final pass uses the fitted temperature without a
  position prior.
- **Measured** on one L40S (48 GB), 50 serial requests through the serving package: p50 0.136 s, p95 0.469 s with this
  adapter, against p50 0.107 s, p95 0.300 s for the frozen base with its own map in the same run.

## Training

Two stages, each one epoch at effective batch eight, with the base frozen:

- **B-2:** 13,674 requests (1,710 updates) across language (SNLI, MAMS, CUAD, and 697 stance/sarcasm items from
  LLM-based generation), intent (BANKING77, CLINC150), knowledge/retrieval (CommonsenseQA, QASC, ARC, ESCI, HotpotQA),
  tool choice from a catalogue built on Glaive-v2 and Hermes-FC schemas, and executable long-policy decisions.
- **B-3:** continued from the B-2 adapter on 8,313 requests (1,040 updates): 788 large-catalogue tool choices, 342
  tool-set rows, 243 tool-relevance rows and 600 legacy tool rows, derived from ToolACE, xLAM/APIGen and API-Bank
  training data; 540 executable record joins; 540 executable numeric/threshold rows; 260 SQuAD2 rows; and 5,000 replay
  requests sampled from B-2.

Before training, rows failing manual review, automated correctness checks, shortcut checks, component-level
deduplication or held-out exclusions were removed. No evaluation-benchmark item or output was used for training,
readout fitting, or checkpoint selection.

## Evaluation

All checks use our own held-out sets and compare the B-3 endpoint with the B-2 endpoint, with paired bootstrap
intervals: harder tool-set and tool-relevance rows; document/rule rows (record joins, numeric/threshold, SQuAD2); and
retention.

| Check (B-3 minus B-2, our held-out sets) | Change | 95% interval lower bound | Registered rule |
|---|---:|---:|---|
| Harder tools: exact tool-set accuracy (tool-set and tool-relevance rows) | +6.7 points | +1.7 | ≥ +5 and lower bound > 0 |
| Document/rule decisions: accuracy (record joins, numeric/threshold, SQuAD2) | +16.1 points | +10.6 | lower bound > −3 |
| Retention: pooled accuracy on 1,800 earlier-skill rows | +0.8 points | −0.3 | lower bound > −1.5 |
| B-2 development set, primary score | +0.3 points | — | ≥ −1 |

All three registered checks passed (10,000 paired bootstrap draws; rules fixed before training).

**Decision Index:** a complete run of edition 0.2.1 scores **47.52** (raw 59.99), submitted as its own row ([PR #63](https://github.com/apolinario/decision-index/pull/63)); per-row results and scores at [`HopitAI/hopper-12b-decision-index-results`](https://huggingface.co/datasets/HopitAI/hopper-12b-decision-index-results).

These figures do not establish performance outside the named held-out sets. They are not benchmark results.

## Intended use

Scoring supplied options for structured decision requests, in evaluation, research and controlled applications where
callers can inspect the offered choices and keep an independent fallback. It is not intended for open-ended
generation or as the sole decision-maker in high-impact settings.

## Limitations

- Document/rule reasoning, abstention and missing-information cases remain known weak areas.
- Accuracy and calibration can shift with task family, menu construction, language and serving hardware. Training
  and evaluation were in English.
- The model can select a plausible but incorrect option. Applications should keep domain review and a safe fallback.
- Long menus need several forward passes, and the unmerged adapter path is slower than a merged load would be.

## Licence and attribution

- **Adapter and readout map:** [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).
- **Base model:** `google/gemma-4-12B-it`, Apache-2.0, subject to Gemma's upstream terms and prohibited-use policy.
  The base weights are not included here.
- **Training data:** the adapter is distributed without its training data. It was trained on rows converted from the
  datasets below, each used under its own licence, which continues to govern that data. Every row was modified:
  reformatted into an option-choice decision request, usually with constructed alternative options.

| dataset | licence |
| --- | --- |
| [SNLI](https://huggingface.co/datasets/stanfordnlp/snli) (Stanford NLP) | [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |
| [ARC](https://huggingface.co/datasets/allenai/ai2_arc) (Allen Institute for AI) | [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |
| [HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) | [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |
| [SQuAD2](https://huggingface.co/datasets/rajpurkar/squad_v2) (Stanford) | [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |
| [CUAD](https://huggingface.co/datasets/theatticusproject/cuad) (The Atticus Project) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [BANKING77](https://huggingface.co/datasets/PolyAI/banking77) (PolyAI) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [QASC](https://huggingface.co/datasets/allenai/qasc) (Allen Institute for AI) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [xLAM function-calling 60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) (Salesforce, APIGen) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [CLINC150](https://huggingface.co/datasets/clinc/clinc_oos) | [CC BY 3.0](https://creativecommons.org/licenses/by/3.0/) |
| [MAMS](https://github.com/siat-nlp/MAMS-for-ABSA) | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| [ESCI](https://huggingface.co/datasets/tasksource/esci) (Amazon Science, `amazon-science/esci-data`) | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| [Glaive function calling v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) (schemas) | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| [Hermes function calling v1](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) (schemas) | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| [ToolACE](https://huggingface.co/datasets/Team-ACE/ToolACE) | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| [API-Bank](https://huggingface.co/datasets/liminghao1630/API-Bank) (training data only) | MIT (Hub mirror); the upstream subtree carries Apache-2.0, and we follow both sets of notice conditions |
| [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) | MIT |

xLAM/APIGen is used under its CC BY 4.0 terms with the citation its authors request:

```bibtex
@article{liu2024apigen,
  title   = {APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author  = {Liu, Zuxin and others},
  journal = {arXiv preprint arXiv:2406.18518},
  year    = {2024}
}
```

The executable rows and the stance/sarcasm rows are this project's own. Distributions of data adapted from the
sources above must carry those sources' licences and notices; the adapter's licence does not replace them.