Terminal-Bench Harness — Benchmark Runs

1. Launch the model server (HF Jobs)

hf jobs run \
  --flavor h200 \
  --expose 30000 \
  --secrets HF_TOKEN \
  --timeout 4h \
  --label project=terminal-bench-harness \
  --label model=qwen3_8-27b-fp8 \
  -d \
  lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code \
    --model-path Qwen/Qwen3.8-27B-FP8 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.85 \
    --attention-backend flashinfer \
    --chunked-prefill-size 32768 \
    --max-prefill-tokens 32768 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 3.67 \
    --host 0.0.0.0 \
    --port 30000 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype float32

This exposes an OpenAI-compatible endpoint at https://<job_id>--30000.hf.jobs/v1, reachable with an HF token that has read access to the job's namespace.

2. Run the benchmark (Harbor + terminal-bench 3.0)

Prerequisite: git-lfs must be installed (brew install git-lfs && git lfs install) or task binary fixtures fail to download correctly.

OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-3.0 \
  --include-task-name atrx-vep-crispr \
  --n-tasks 1 \
  --job-name qwen3_8-27b-fp8-terminus2-smoketest-v3 \
  --n-concurrent 1 \
  -y

Notes:

• OPENAI_API_KEY must be exported into the shell running harbor, not passed via --ae (that flag only injects env vars into the sandboxed task container, not the host process making the LLM call).
• Drop --include-task-name and raise --n-tasks to run more of the dataset.

Result: single-task smoke test (terminal-bench 3.0)

atrx-vep-crispr__DvYyCpF: completed with 0 exceptions, reward 0 (task itself unsolved — it's a 7h expert-time-estimate task). Runtime 3h6m, ~5.68M input / ~681K output tokens.

3. Full run (terminal-bench 2.1, all 89 tasks, HF Sandbox environments)

Running all 89 tasks locally via Docker is impractical (each task's environment container competes for local CPU/RAM). Instead use --env hf-sandbox, which runs each task's environment as its own HF Jobs sandbox instead of a local container — every task in terminal-bench 2.1 already references a prebuilt public docker_image in its task.toml, so no local builds are needed.

OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --ak reasoning_effort=low \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --agent-timeout-multiplier 4 \
  --job-name qwen3_8-27b-fp8-terminus2-tb21-full-v2 \
  --n-concurrent 3 \
  -y

Notes:

• --agent-timeout-multiplier 4: this model is a heavy reasoner (burns lots of tokens/time per turn); the terminal-bench default per-task agent timeout (often 900s) is far too tight and causes widespread AgentTimeoutError. 4x (60 min/task) balances fairness against runtime.
• --ak reasoning_effort=low: keeps the model's thinking budget in check so it doesn't blow through the timeout on easy tasks.
• --n-concurrent: at 8, ~70% of trials errored out with SandboxError/InternalServerError/ReadTimeout (HF Sandbox infra strain, not model failures). Dropping to 3 brought the error rate down substantially. Lower is more reliable but slower.
• Mop up leftover infra errors without re-running everything:

  # first, if you changed --n-concurrent, also edit it in the job's config.json (and lock.json to match) before resuming
  OPENAI_API_KEY="$(hf auth token)" harbor job resume \
    -p jobs/<job-name> \
    --filter-error-type SandboxError \
    --filter-error-type InternalServerError \
    --filter-error-type ReadTimeout \
    --filter-error-type ReadError \
    --filter-error-type ConnectTimeout \
    --filter-error-type RemoteProtocolError
    # deliberately NOT AgentTimeoutError — that's a legitimate result, not infra flakiness
  

  Repeat as needed; each pass only touches the filtered trials, everything else is preserved.
• Harbor's own printed end-of-run summary double-counts across resume passes (it doesn't dedupe eval stats between attempts) — don't trust the final Trials/Exceptions/Mean table it prints after a multi-pass run. Get the real per-task outcome by taking, for each task name, the trial directory with the latest finished_at in result.json (reward at verifier_result.rewards.reward, or exception_info.exception_type if it errored).

Final result (89/89 tasks, after 3 resume/mop-up passes)

Pass rate among the 69 genuinely-scored tasks: 76.8%. Of the 20 errors, 16 are legitimate AgentTimeoutError (task too slow even at 4x budget); the remaining 4 (circuit-fibsqrt, feal-linear-cryptanalysis, query-optimize, schemelike-metacircular-eval) failed with SandboxError/RemoteProtocolError on every one of 3 separate retry attempts (across 2 different model-serving sessions) and were accepted as final errors — see the pitfall below before trying a 4th time.

Pitfall: resuming a job after the model server URL changes

If the model-serving job dies (it will — HF Jobs --timeout is enforced eventually, just unpredictably; ours ran ~91h against a 24h setting before getting killed) and you relaunch it, do not harbor job resume the original job just because you updated api_base in its config.json/lock.json. Job.create()'s resume path matches each on-disk completed trial against a freshly-resolved TrialConfig via exact equality — which includes the agent's kwargs. Editing only the job-level files doesn't touch the ~85 individual completed trials' own per-trial config.json, so none of them match anymore and harbor concludes the entire job is still pending, silently re-running everything (including tasks you already solved) instead of just the ones you meant to retry. This wasted ~8 hours (a 2h + a 6h job) for us with zero output, because the runs actually landed under an unexpected jobs/<job-name>/ bucket path — the default harbor run/resume subdirectory — not the flat job path we were watching, so it looked idle rather than busy-doing-the-wrong-thing.

Do this instead: launch a small standalone harbor run (not resume) into a new job name, scoped to just the tasks you need via repeated --include-task-name <name>, pointed at the new endpoint. Merge results at final-tally time by scanning every relevant job directory and, per task name, keeping whichever trial has the latest finished_at.

---

Run 2: gpt-oss-20b (terminal-bench 2.1)

1. Launch the model server

The model card's own recommended command (uv pip install --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ ...) pins a launch-day (Aug 2025) PyTorch nightly build that has since been pruned from the nightly index — it fails with No solution found when resolving dependencies. Used current stable vLLM instead:

hf jobs run \
  --flavor l40sx1 \
  --expose 8000 \
  --secrets HF_TOKEN \
  --timeout 12h \
  --label project=terminal-bench-harness \
  --label model=gpt-oss-20b \
  -d \
  vllm/vllm-openai:latest \
  vllm serve openai/gpt-oss-20b --host 0.0.0.0 --port 8000

Note: hf jobs run <image> <command> passes <command> as a literal argv, not through the image's shell/ENTRYPOINT — spell out the full command (vllm serve openai/gpt-oss-20b ...), don't rely on bare flags assuming an entrypoint wraps them (--model ... alone fails with exec: "--model": executable file not found in $PATH).

2. Run the benchmark

Same pattern as Run 1, orchestrator on its own HF Job with the bucket mounted — but written to land flat at the bucket root by passing --jobs-dir . after cd /data (the default --jobs-dir jobs would otherwise nest output at <bucket>/jobs/<job-name>, inconsistent with every other job dir here):

# inside the job container, after cd /data
OPENAI_API_KEY="$HF_TOKEN" harbor run \
  --agent terminus-2 \
  --model "openai/openai/gpt-oss-20b" \
  --ak api_base="https://<job_id>--8000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --agent-timeout-multiplier 4 \
  --jobs-dir . \
  --job-name gpt-oss-20b-terminus2-tb21-full \
  --n-concurrent 3 \
  -y

(--model "openai/openai/gpt-oss-20b" is correct, not a typo — the LiteLLM openai/ provider prefix plus the HF repo id openai/gpt-oss-20b just happen to both start with "openai".)

Ran a 1-task smoke test first (--n-tasks 1, plain local Docker, no --env hf-sandbox needed for a single task) to confirm the endpoint/harness wiring before committing to the full run.

Final result (89/89 tasks, single clean pass, no resume needed)

Pass rate excluding infra errors: 4.71% (4/85) — spot-checked the 4 passes for real multi-episode agent activity (not degenerate zero-effort completions): confirmed genuine. HF Sandbox reliability was strong this run (only 4/89 excluded: 2 SandboxError, 1 RemoteProtocolError, 1 CancelledError from the outer job hitting its 12h timeout mid-trial at the very end — harbor's own process had already finished cleanly and printed its 89/89 Mean: 0.045 summary, but a lingering unclosed aiohttp session kept the container alive past that point until the timeout force-killed it). Only 3 genuine AgentTimeoutErrors — this smaller/faster model wasn't timeout-bound the way the 27B reasoning model was; the whole 89-task run finished in ~13h vs. days for Run 1.

For comparison: Qwen3.8-27B-FP8 scored 62.35% on the same dataset/harness (Run 1) — gpt-oss-20b genuinely struggled with terminal-bench 2.1's agentic coding tasks.

---

Run 3: Qwen3.8-27B-FP8 with mini-swe-agent (terminal-bench 2.1)

Same model as Run 1, different harness agent, to compare terminus-2 vs mini-swe-agent on identical model/dataset. claude-code was considered first but ruled out: it requires an ANTHROPIC_BASE_URL speaking the Anthropic Messages API, incompatible with sglang's OpenAI-format endpoint without a LiteLLM proxy bridge — not worth building for this comparison, so mini-swe-agent was used instead (its CLI runs inside the sandbox, so wiring is via --ae, not --ak, unlike terminus-2 — see agent-wiring notes below).

1. Launch the model server

hf jobs run \
  --flavor h200 --expose 30000 --secrets HF_TOKEN --timeout 12h \
  --label project=terminal-bench-harness --label model=qwen3_8-27b-fp8 --label agent=mini-swe-agent \
  -d lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code --model-path Qwen/Qwen3.8-27B-FP8 \
    --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.85 \
    --attention-backend flashinfer --chunked-prefill-size 32768 --max-prefill-tokens 32768 \
    --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 3.67 --host 0.0.0.0 --port 30000 \
    --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype float32 \
    --default-chat-template-kwargs '{"enable_thinking": false}'

Note the added --default-chat-template-kwargs '{"enable_thinking": false}' vs. Run 1 — Qwen3's reasoning/thinking mode was on by default and caused runaway generation (AgentTimeoutErrors with suspiciously low output-token counts, e.g. 650 tokens over 3600s — confirmed via the server's own decode-batch logs showing #full token climbing continuously, and via non-zero reasoning_tokens in the trajectory JSON). Disabling it server-side fixed this (verified: reasoning_content: null, reasoning_tokens: 0 on a test completion).

A newer sglang cookbook variant adding speculative decoding (--speculative-algorithm DFLASH + draft model, --mamba-full-memory-ratio 11.01) was tried and OOM'd on H200 (confirmed CUDA OOM traceback) — reverted to the proven config above (no speculative decoding, ratio 3.67).

HF Inference Providers were tried as a self-hosting alternative for a bit (Qwen/Qwen3.8-27B:deepinfra) but produced repeated 504 Gateway Time-outs mid-run under sustained agentic load (LiteLLM retry-looping) — reverted to self-hosting.

2. Run the benchmark

# orchestrator, as its own HF Job with the bucket mounted at /data (git-lfs required, jobs-dir . for flat bucket layout)
pip install -q --pre "harbor[hf-sandbox]==0.22.1.dev202609020403"   # see pipefail/dash fix note below
harbor run \
  --agent mini-swe-agent \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ae OPENAI_API_KEY="$HF_TOKEN" \
  --ae OPENAI_BASE_URL="https://<job_id>--30000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --ek job_timeout=2h \
  --agent-timeout-multiplier 4 \
  --jobs-dir . \
  --job-name qwen3_8-27b-fp8-mswea-tb21-full \
  --n-concurrent 3 \
  -y

Notes:

• mini-swe-agent's CLI runs inside the sandbox (unlike terminus-2's LiteLLM, which runs on the host) — so it's wired via --ae OPENAI_API_KEY=... --ae OPENAI_BASE_URL=... (env vars forwarded into the sandboxed exec), not --ak api_base=... + a host-exported key.
• set -o pipefail/dash bug: hit NonZeroAgentExitCodeError: set: Illegal option -o pipefail on all 89 trials instantly (6 min) — harbor's ensure_system_dependencies prepends bash-only set -o pipefail;, but --env hf-sandbox execs via POSIX /bin/sh (dash), not bash (local Docker wraps in bash explicitly, hf-sandbox didn't). Fixed upstream in harbor-framework/harbor#3016; pin harbor[hf-sandbox]==0.22.1.dev202609020403 (a --pre PyPI dev build) until it lands in a stable release.
• Sandbox idle-timeout bug: RemoteProtocolError/SandboxError recurring on the same step even after retries — root cause found by checking the underlying ephemeral sandbox job's own logs (hf jobs ps -a to find it, then hf jobs logs), not harbor's wrapper: sbx-server: idle for 601917ms, shutting down. hf-sandbox's default 600s idle timeout was killing the sandbox mid-task because mini-swe-agent runs its whole agent loop as one long single exec call (no intermediate keep-alive activity harbor can see). Fixed with --ek job_timeout=2h (maps to the sandbox's own idle_timeout).

Final result (89/89 tasks, single pass, no resume needed)

Pass rate excluding infra errors: 51.9% (42/81). Full per-task breakdown: qwen3_8-27b-fp8-mswea-tb21-full/README.md in this bucket.

Of the 11 initial NonZeroAgentExitCodeError trials, only 3 were genuine infra (qemu-startup, qemu-alpine-ssh: apt-get failing on an expired Debian bullseye-security archive mirror; make-doom-for-mips: /bin/sh: ps: not found in a minimal task image) — the other 8 were real model/agent capability limits: 7 ContextWindowExceededError (mini-swe-agent resends full growing conversation history every turn; hit Qwen's 262144-token context limit on long tasks) and 1 exec-level timeout kill (query-optimize, agent launched a long-running background shell command and got cut off). None were a recurrence of the pipefail/dash bug above — that fix held across the full run.

Comparison: terminus-2 vs mini-swe-agent, same model (Qwen3.8-27B-FP8, terminal-bench 2.1)

terminus-2 substantially outperformed mini-swe-agent on the same model/dataset — plausibly related to mini-swe-agent's full-history-resend pattern (causing the context-window failures above) vs. terminus-2's context handling, though this wasn't isolated further.