README.md

metadata

license: apache-2.0
base_model: google/gemma-4-12B-it
library_name: peft
language:
  - en
tags:
  - lora
  - peft
  - base_model:adapter:google/gemma-4-12B-it
  - decision-making
  - calibration
datasets:
  - stanfordnlp/snli
  - allenai/ai2_arc
  - hotpotqa/hotpot_qa
  - rajpurkar/squad_v2
  - theatticusproject/cuad
  - PolyAI/banking77
  - allenai/qasc
  - Salesforce/xlam-function-calling-60k
  - clinc/clinc_oos
  - tasksource/esci
  - glaiveai/glaive-function-calling-v2
  - NousResearch/hermes-function-calling-v1
  - Team-ACE/ToolACE
  - liminghao1630/API-Bank
  - tau/commonsense_qa

Hopper 12B (trained)

This branch (hopper-12b) of HopitAI/hopper holds Hopper 12B (trained): a PEFT LoRA adapter for the frozen google/gemma-4-12B-it base at revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, and the readout map fitted for it. The repository's main branch holds a different model, the Qwen3.5-4B Hopper adapter, which is unchanged.

The model scores the options of a structured decision request: a state, a question and a fixed set of options go in, and a probability distribution over the options comes out. It does not generate text. It is served by the Hopper serving code, github.com/hopit-ai/hopper, tag 12b-1.1.0 (package 1.2.0), which speaks the JevBench /v1/systemone wire format.

Files

file	what it is
adapter_config.json	PEFT LoRA configuration (rank 32, alpha 64, dropout 0)
adapter_model.safetensors	adapter weights, 524,648,560 bytes
hopper-12b-trained-readout.json	the sealed readout-bias/readout-seal-v2 readout map (seal 65c72f717a74eccfc38e3906345927765a00d8e0fdd8c61e8658c55eec34c818); byte-identical to the copy shipped in the serving package
SHA256SUMS	SHA-256 checksums of the three files above

How to run it

pip install "git+https://github.com/hopit-ai/hopper@12b-1.1.0"
hopper-serve --base-loader gemma-4-12b-it \
  --adapter HopitAI/hopper@4b605f25eb903c2cf4d121040ffb097240b3bc37 \
  --readout-map "$(python -c 'import hopper_decisions, pathlib; print(pathlib.Path(hopper_decisions.__file__).parent / "maps/hopper-12b-trained-readout.json")')" \
  --port 8080

Always pin the revision: HopitAI/hopper without one resolves to main, the Qwen3.5-4B adapter, which does not fit this base. To verify the files and serve a local copy instead:

huggingface-cli download HopitAI/hopper --revision 4b605f25eb903c2cf4d121040ffb097240b3bc37 --local-dir hopper-12b
(cd hopper-12b && shasum -a 256 -c SHA256SUMS)
hopper-serve --base-loader gemma-4-12b-it --adapter ./hopper-12b \
  --readout-map ./hopper-12b/hopper-12b-trained-readout.json --port 8080

Architecture and serving

• Base: google/gemma-4-12B-it at revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, frozen, bf16.
• Adapter: LoRA rank 32, alpha 64, on the seven text projections q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj.
• Keep it unmerged. A bf16 merge of the adapter changed 2.1% of top choices on 1,932 decisions, so the serving code loads it as a separate low-rank path, as it was evaluated. A merged copy is not decision-equivalent and is not supported.
• Readout: one forward pass per decision; the answer is a softmax over the option letters. The readout map then applies position priors for original two- and four-option choice menus (fitted on 240 and 74 menus) and one temperature, 1.65. Priors do not apply to noul or score questions or to other menu sizes.
• Long menus (more than 26 options) use the serving code's seeded tournament shortlist (k=10, residual mass 0.05): qualification passes use raw temperature-1 scores, and the final pass uses the fitted temperature without a position prior.
• Measured on one L40S (48 GB), 50 serial requests through the serving package: p50 0.136 s, p95 0.469 s with this adapter, against p50 0.107 s, p95 0.300 s for the frozen base with its own map in the same run.

Training

Two stages, each one epoch at effective batch eight, with the base frozen:

• B-2: 13,674 requests (1,710 updates) across language (SNLI, MAMS, CUAD, and 697 stance/sarcasm items from LLM-based generation), intent (BANKING77, CLINC150), knowledge/retrieval (CommonsenseQA, QASC, ARC, ESCI, HotpotQA), tool choice from a catalogue built on Glaive-v2 and Hermes-FC schemas, and executable long-policy decisions.
• B-3: continued from the B-2 adapter on 8,313 requests (1,040 updates): 788 large-catalogue tool choices, 342 tool-set rows, 243 tool-relevance rows and 600 legacy tool rows, derived from ToolACE, xLAM/APIGen and API-Bank training data; 540 executable record joins; 540 executable numeric/threshold rows; 260 SQuAD2 rows; and 5,000 replay requests sampled from B-2.

Before training, rows failing manual review, automated correctness checks, shortcut checks, component-level deduplication or held-out exclusions were removed. No evaluation-benchmark item or output was used for training, readout fitting, or checkpoint selection.

Evaluation

All checks use our own held-out sets and compare the B-3 endpoint with the B-2 endpoint, with paired bootstrap intervals: harder tool-set and tool-relevance rows; document/rule rows (record joins, numeric/threshold, SQuAD2); and retention.

Check (B-3 minus B-2, our held-out sets)	Change	95% interval lower bound	Registered rule
Harder tools: exact tool-set accuracy (tool-set and tool-relevance rows)	+6.7 points	+1.7	≥ +5 and lower bound > 0
Document/rule decisions: accuracy (record joins, numeric/threshold, SQuAD2)	+16.1 points	+10.6	lower bound > −3
Retention: pooled accuracy on 1,800 earlier-skill rows	+0.8 points	−0.3	lower bound > −1.5
B-2 development set, primary score	+0.3 points	—	≥ −1

All three registered checks passed (10,000 paired bootstrap draws; rules fixed before training).

Decision Index: a complete run of edition 0.2.1 scores 47.52 (raw 59.99), submitted as its own row (PR #63); per-row results and scores at HopitAI/hopper-12b-decision-index-results.

These figures do not establish performance outside the named held-out sets. They are not benchmark results.

Intended use

Scoring supplied options for structured decision requests, in evaluation, research and controlled applications where callers can inspect the offered choices and keep an independent fallback. It is not intended for open-ended generation or as the sole decision-maker in high-impact settings.

Limitations

• Document/rule reasoning, abstention and missing-information cases remain known weak areas.
• Accuracy and calibration can shift with task family, menu construction, language and serving hardware. Training and evaluation were in English.
• The model can select a plausible but incorrect option. Applications should keep domain review and a safe fallback.
• Long menus need several forward passes, and the unmerged adapter path is slower than a merged load would be.

Licence and attribution

• Adapter and readout map: Apache-2.0.
• Base model: google/gemma-4-12B-it, Apache-2.0, subject to Gemma's upstream terms and prohibited-use policy. The base weights are not included here.
• Training data: the adapter is distributed without its training data. It was trained on rows converted from the datasets below, each used under its own licence, which continues to govern that data. Every row was modified: reformatted into an option-choice decision request, usually with constructed alternative options.

dataset	licence
SNLI (Stanford NLP)	CC BY-SA 4.0
ARC (Allen Institute for AI)	CC BY-SA 4.0
HotpotQA	CC BY-SA 4.0
SQuAD2 (Stanford)	CC BY-SA 4.0
CUAD (The Atticus Project)	CC BY 4.0
BANKING77 (PolyAI)	CC BY 4.0
QASC (Allen Institute for AI)	CC BY 4.0
xLAM function-calling 60k (Salesforce, APIGen)	CC BY 4.0
CLINC150	CC BY 3.0
MAMS	Apache-2.0
ESCI (Amazon Science, amazon-science/esci-data)	Apache-2.0
Glaive function calling v2 (schemas)	Apache-2.0
Hermes function calling v1 (schemas)	Apache-2.0
ToolACE	Apache-2.0
API-Bank (training data only)	MIT (Hub mirror); the upstream subtree carries Apache-2.0, and we follow both sets of notice conditions
CommonsenseQA	MIT

xLAM/APIGen is used under its CC BY 4.0 terms with the citation its authors request:

@article{liu2024apigen,
  title   = {APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author  = {Liu, Zuxin and others},
  journal = {arXiv preprint arXiv:2406.18518},
  year    = {2024}
}

The executable rows and the stance/sarcasm rows are this project's own. Distributions of data adapted from the sources above must carry those sources' licences and notices; the adapter's licence does not replace them.