Buckets:

SaylorTwift
/

harness-evals

Files

xet

SaylorTwift/harness-evals / README.md

SaylorTwift

about 1 month ago

preview code

download

raw

16.6 kB

# Terminal-Bench Harness — Benchmark Runs

## 1. Launch the model server (HF Jobs)

```bash
hf jobs run \
  --flavor h200 \
  --expose 30000 \
  --secrets HF_TOKEN \
  --timeout 4h \
  --label project=terminal-bench-harness \
  --label model=qwen3_8-27b-fp8 \
  -d \
  lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code \
    --model-path Qwen/Qwen3.8-27B-FP8 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.85 \
    --attention-backend flashinfer \
    --chunked-prefill-size 32768 \
    --max-prefill-tokens 32768 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 3.67 \
    --host 0.0.0.0 \
    --port 30000 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype float32
```

This exposes an OpenAI-compatible endpoint at `https://<job_id>--30000.hf.jobs/v1`, reachable with an HF token that has read access to the job's namespace.

## 2. Run the benchmark (Harbor + terminal-bench 3.0)

Prerequisite: `git-lfs` must be installed (`brew install git-lfs && git lfs install`) or task binary fixtures fail to download correctly.

```bash
OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-3.0 \
  --include-task-name atrx-vep-crispr \
  --n-tasks 1 \
  --job-name qwen3_8-27b-fp8-terminus2-smoketest-v3 \
  --n-concurrent 1 \
  -y
```

Notes:
- `OPENAI_API_KEY` must be exported into the shell running `harbor`, not passed via `--ae` (that flag only injects env vars into the sandboxed task container, not the host process making the LLM call).
- Drop `--include-task-name` and raise `--n-tasks` to run more of the dataset.

## Result: single-task smoke test (terminal-bench 3.0)

`atrx-vep-crispr__DvYyCpF`: completed with 0 exceptions, reward 0 (task itself unsolved — it's a 7h expert-time-estimate task). Runtime 3h6m, ~5.68M input / ~681K output tokens.

## 3. Full run (terminal-bench 2.1, all 89 tasks, HF Sandbox environments)

Running all 89 tasks locally via Docker is impractical (each task's environment container competes for local CPU/RAM). Instead use `--env hf-sandbox`, which runs each task's environment as its own HF Jobs sandbox instead of a local container — every task in terminal-bench 2.1 already references a prebuilt public `docker_image` in its `task.toml`, so no local builds are needed.

```bash
OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --ak reasoning_effort=low \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --agent-timeout-multiplier 4 \
  --job-name qwen3_8-27b-fp8-terminus2-tb21-full-v2 \
  --n-concurrent 3 \
  -y
```

Notes:
- `--agent-timeout-multiplier 4`: this model is a heavy reasoner (burns lots of tokens/time per turn); the terminal-bench default per-task agent timeout (often 900s) is far too tight and causes widespread `AgentTimeoutError`. 4x (60 min/task) balances fairness against runtime.
- `--ak reasoning_effort=low`: keeps the model's thinking budget in check so it doesn't blow through the timeout on easy tasks.
- `--n-concurrent`: at `8`, ~70% of trials errored out with `SandboxError`/`InternalServerError`/`ReadTimeout` (HF Sandbox infra strain, not model failures). Dropping to `3` brought the error rate down substantially. Lower is more reliable but slower.
- Mop up leftover infra errors without re-running everything:
  ```bash
  # first, if you changed --n-concurrent, also edit it in the job's config.json (and lock.json to match) before resuming
  OPENAI_API_KEY="$(hf auth token)" harbor job resume \
    -p jobs/<job-name> \
    --filter-error-type SandboxError \
    --filter-error-type InternalServerError \
    --filter-error-type ReadTimeout \
    --filter-error-type ReadError \
    --filter-error-type ConnectTimeout \
    --filter-error-type RemoteProtocolError
    # deliberately NOT AgentTimeoutError — that's a legitimate result, not infra flakiness
  ```
  Repeat as needed; each pass only touches the filtered trials, everything else is preserved.
- **Harbor's own printed end-of-run summary double-counts across resume passes** (it doesn't dedupe eval stats between attempts) — don't trust the final `Trials`/`Exceptions`/`Mean` table it prints after a multi-pass run. Get the real per-task outcome by taking, for each task name, the trial directory with the latest `finished_at` in `result.json` (reward at `verifier_result.rewards.reward`, or `exception_info.exception_type` if it errored).

### Final result (89/89 tasks, after 3 resume/mop-up passes)

| Outcome | Count | % of 89 |
|---|---|---|
| Passed | 53 | 59.6% |
| Failed (genuine, reward 0) | 16 | 18.0% |
| Errored (unscored) | 20 | 22.5% |

Pass rate among the 69 genuinely-scored tasks: **76.8%**. Of the 20 errors, 16 are legitimate `AgentTimeoutError` (task too slow even at 4x budget); the remaining 4 (`circuit-fibsqrt`, `feal-linear-cryptanalysis`, `query-optimize`, `schemelike-metacircular-eval`) failed with `SandboxError`/`RemoteProtocolError` on every one of 3 separate retry attempts (across 2 different model-serving sessions) and were accepted as final errors — see the pitfall below before trying a 4th time.

### Pitfall: resuming a job after the model server URL changes

If the model-serving job dies (it will — HF Jobs `--timeout` is enforced eventually, just unpredictably; ours ran ~91h against a 24h setting before getting killed) and you relaunch it, **do not `harbor job resume` the original job** just because you updated `api_base` in its `config.json`/`lock.json`. `Job.create()`'s resume path matches each on-disk completed trial against a freshly-resolved `TrialConfig` via exact equality — which includes the agent's `kwargs`. Editing only the job-level files doesn't touch the ~85 individual completed trials' own per-trial `config.json`, so none of them match anymore and harbor concludes the *entire job* is still pending, silently re-running everything (including tasks you already solved) instead of just the ones you meant to retry. This wasted ~8 hours (a 2h + a 6h job) for us with zero output, because the runs actually landed under an unexpected `jobs/<job-name>/` bucket path — the default `harbor run`/`resume` subdirectory — not the flat job path we were watching, so it looked idle rather than busy-doing-the-wrong-thing.

**Do this instead:** launch a small standalone `harbor run` (not `resume`) into a *new* job name, scoped to just the tasks you need via repeated `--include-task-name <name>`, pointed at the new endpoint. Merge results at final-tally time by scanning every relevant job directory and, per task name, keeping whichever trial has the latest `finished_at`.

---

# Run 2: gpt-oss-20b (terminal-bench 2.1)

## 1. Launch the model server

The model card's own recommended command (`uv pip install --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ ...`) pins a launch-day (Aug 2025) PyTorch nightly build that has since been pruned from the nightly index — it fails with `No solution found when resolving dependencies`. Used current stable vLLM instead:

```bash
hf jobs run \
  --flavor l40sx1 \
  --expose 8000 \
  --secrets HF_TOKEN \
  --timeout 12h \
  --label project=terminal-bench-harness \
  --label model=gpt-oss-20b \
  -d \
  vllm/vllm-openai:latest \
  vllm serve openai/gpt-oss-20b --host 0.0.0.0 --port 8000
```

Note: `hf jobs run <image> <command>` passes `<command>` as a literal argv, not through the image's shell/`ENTRYPOINT` — spell out the full command (`vllm serve openai/gpt-oss-20b ...`), don't rely on bare flags assuming an entrypoint wraps them (`--model ...` alone fails with `exec: "--model": executable file not found in $PATH`).

## 2. Run the benchmark

Same pattern as Run 1, orchestrator on its own HF Job with the bucket mounted — but written to land **flat** at the bucket root by passing `--jobs-dir .` after `cd /data` (the default `--jobs-dir jobs` would otherwise nest output at `<bucket>/jobs/<job-name>`, inconsistent with every other job dir here):

```bash
# inside the job container, after cd /data
OPENAI_API_KEY="$HF_TOKEN" harbor run \
  --agent terminus-2 \
  --model "openai/openai/gpt-oss-20b" \
  --ak api_base="https://<job_id>--8000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --agent-timeout-multiplier 4 \
  --jobs-dir . \
  --job-name gpt-oss-20b-terminus2-tb21-full \
  --n-concurrent 3 \
  -y
```

(`--model "openai/openai/gpt-oss-20b"` is correct, not a typo — the LiteLLM `openai/` provider prefix plus the HF repo id `openai/gpt-oss-20b` just happen to both start with "openai".)

Ran a 1-task smoke test first (`--n-tasks 1`, plain local Docker, no `--env hf-sandbox` needed for a single task) to confirm the endpoint/harness wiring before committing to the full run.

### Final result (89/89 tasks, single clean pass, no resume needed)

| Outcome | Count | % of 89 |
|---|---|---|
| Passed | 4 | 4.5% |
| Failed (genuine, incl. `AgentTimeoutError`) | 81 | 91.0% |
| Errored (infra, excluded) | 4 | 4.5% |

**Pass rate excluding infra errors: 4.71% (4/85)** — spot-checked the 4 passes for real multi-episode agent activity (not degenerate zero-effort completions): confirmed genuine. HF Sandbox reliability was strong this run (only 4/89 excluded: 2 `SandboxError`, 1 `RemoteProtocolError`, 1 `CancelledError` from the outer job hitting its 12h timeout mid-trial at the very end — harbor's own process had already finished cleanly and printed its `89/89 Mean: 0.045` summary, but a lingering unclosed aiohttp session kept the container alive past that point until the timeout force-killed it). Only 3 genuine `AgentTimeoutError`s — this smaller/faster model wasn't timeout-bound the way the 27B reasoning model was; the whole 89-task run finished in ~13h vs. days for Run 1.

For comparison: Qwen3.8-27B-FP8 scored 62.35% on the same dataset/harness (Run 1) — gpt-oss-20b genuinely struggled with terminal-bench 2.1's agentic coding tasks.

---

# Run 3: Qwen3.8-27B-FP8 with mini-swe-agent (terminal-bench 2.1)

Same model as Run 1, different harness agent, to compare terminus-2 vs mini-swe-agent on identical model/dataset. `claude-code` was considered first but ruled out: it requires an `ANTHROPIC_BASE_URL` speaking the Anthropic Messages API, incompatible with sglang's OpenAI-format endpoint without a LiteLLM proxy bridge — not worth building for this comparison, so mini-swe-agent was used instead (its CLI runs *inside* the sandbox, so wiring is via `--ae`, not `--ak`, unlike terminus-2 — see agent-wiring notes below).

## 1. Launch the model server

```bash
hf jobs run \
  --flavor h200 --expose 30000 --secrets HF_TOKEN --timeout 12h \
  --label project=terminal-bench-harness --label model=qwen3_8-27b-fp8 --label agent=mini-swe-agent \
  -d lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code --model-path Qwen/Qwen3.8-27B-FP8 \
    --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.85 \
    --attention-backend flashinfer --chunked-prefill-size 32768 --max-prefill-tokens 32768 \
    --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 3.67 --host 0.0.0.0 --port 30000 \
    --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype float32 \
    --default-chat-template-kwargs '{"enable_thinking": false}'
```

Note the added `--default-chat-template-kwargs '{"enable_thinking": false}'` vs. Run 1 — Qwen3's reasoning/thinking mode was on by default and caused runaway generation (`AgentTimeoutError`s with suspiciously low output-token counts, e.g. 650 tokens over 3600s — confirmed via the server's own decode-batch logs showing `#full token` climbing continuously, and via non-zero `reasoning_tokens` in the trajectory JSON). Disabling it server-side fixed this (verified: `reasoning_content: null`, `reasoning_tokens: 0` on a test completion).

A newer sglang cookbook variant adding speculative decoding (`--speculative-algorithm DFLASH` + draft model, `--mamba-full-memory-ratio 11.01`) was tried and OOM'd on H200 (confirmed CUDA OOM traceback) — reverted to the proven config above (no speculative decoding, ratio 3.67).

HF Inference Providers were tried as a self-hosting alternative for a bit (`Qwen/Qwen3.8-27B:deepinfra`) but produced repeated `504 Gateway Time-out`s mid-run under sustained agentic load (LiteLLM retry-looping) — reverted to self-hosting.

## 2. Run the benchmark

```bash
# orchestrator, as its own HF Job with the bucket mounted at /data (git-lfs required, jobs-dir . for flat bucket layout)
pip install -q --pre "harbor[hf-sandbox]==0.22.1.dev202609020403"   # see pipefail/dash fix note below
harbor run \
  --agent mini-swe-agent \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ae OPENAI_API_KEY="$HF_TOKEN" \
  --ae OPENAI_BASE_URL="https://<job_id>--30000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --ek job_timeout=2h \
  --agent-timeout-multiplier 4 \
  --jobs-dir . \
  --job-name qwen3_8-27b-fp8-mswea-tb21-full \
  --n-concurrent 3 \
  -y
```

Notes:
- **mini-swe-agent's CLI runs *inside* the sandbox** (unlike terminus-2's LiteLLM, which runs on the host) — so it's wired via `--ae OPENAI_API_KEY=... --ae OPENAI_BASE_URL=...` (env vars forwarded into the sandboxed exec), not `--ak api_base=...` + a host-exported key.
- **`set -o pipefail`/dash bug**: hit `NonZeroAgentExitCodeError: set: Illegal option -o pipefail` on all 89 trials instantly (6 min) — harbor's `ensure_system_dependencies` prepends bash-only `set -o pipefail;`, but `--env hf-sandbox` execs via POSIX `/bin/sh` (dash), not bash (local Docker wraps in bash explicitly, hf-sandbox didn't). Fixed upstream in [harbor-framework/harbor#3016](https://github.com/harbor-framework/harbor/pull/3016); pin `harbor[hf-sandbox]==0.22.1.dev202609020403` (a `--pre` PyPI dev build) until it lands in a stable release.
- **Sandbox idle-timeout bug**: `RemoteProtocolError`/`SandboxError` recurring on the same step even after retries — root cause found by checking the *underlying ephemeral sandbox job's own logs* (`hf jobs ps -a` to find it, then `hf jobs logs`), not harbor's wrapper: `sbx-server: idle for 601917ms, shutting down`. hf-sandbox's default 600s idle timeout was killing the sandbox mid-task because mini-swe-agent runs its whole agent loop as one long single exec call (no intermediate keep-alive activity harbor can see). Fixed with `--ek job_timeout=2h` (maps to the sandbox's own `idle_timeout`).

### Final result (89/89 tasks, single pass, no resume needed)

| Outcome | Count | % of 89 |
|---|---|---|
| Passed | 42 | 47.2% |
| Failed (genuine, incl. `AgentTimeoutError` + `ContextWindowExceededError` + 1 exec-timeout) | 39 | 43.8% |
| Errored (infra, excluded) | 8 | 9.0% |

**Pass rate excluding infra errors: 51.9% (42/81).** Full per-task breakdown: `qwen3_8-27b-fp8-mswea-tb21-full/README.md` in this bucket.

Of the 11 initial `NonZeroAgentExitCodeError` trials, only 3 were genuine infra (`qemu-startup`, `qemu-alpine-ssh`: apt-get failing on an expired Debian `bullseye-security` archive mirror; `make-doom-for-mips`: `/bin/sh: ps: not found` in a minimal task image) — the other 8 were real model/agent capability limits: 7 `ContextWindowExceededError` (mini-swe-agent resends full growing conversation history every turn; hit Qwen's 262144-token context limit on long tasks) and 1 exec-level timeout kill (`query-optimize`, agent launched a long-running background shell command and got cut off). None were a recurrence of the pipefail/dash bug above — that fix held across the full run.

### Comparison: terminus-2 vs mini-swe-agent, same model (Qwen3.8-27B-FP8, terminal-bench 2.1)

| Agent | Pass rate (excl. infra) | Passed / Scored |
|---|---|---|
| terminus-2 | 76.8% | 53/69 |
| mini-swe-agent | 51.9% | 42/81 |

terminus-2 substantially outperformed mini-swe-agent on the same model/dataset — plausibly related to mini-swe-agent's full-history-resend pattern (causing the context-window failures above) vs. terminus-2's context handling, though this wasn't isolated further.

Xet Storage Details

Size:: 16.6 kB
Xet hash:: 69eb646af25c2604500d4d89bf95db6c5f71090fb7eb4471e7399a9cabfd67e9

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Terminal-Bench Harness — Benchmark Runs

	## 1. Launch the model server (HF Jobs)

	```bash
	hf jobs run \
	--flavor h200 \
	--expose 30000 \
	--secrets HF_TOKEN \
	--timeout 4h \
	--label project=terminal-bench-harness \
	--label model=qwen3_8-27b-fp8 \
	-d \
	lmsysorg/sglang:latest \
	sglang serve \
	--trust-remote-code \
	--model-path Qwen/Qwen3.8-27B-FP8 \
	--kv-cache-dtype fp8_e4m3 \
	--mem-fraction-static 0.85 \
	--attention-backend flashinfer \
	--chunked-prefill-size 32768 \
	--max-prefill-tokens 32768 \
	--reasoning-parser qwen3 \
	--tool-call-parser qwen3_coder \
	--mamba-full-memory-ratio 3.67 \
	--host 0.0.0.0 \
	--port 30000 \
	--mamba-radix-cache-strategy extra_buffer_lazy \
	--mamba-ssm-dtype float32
	```

	This exposes an OpenAI-compatible endpoint at `https://<job_id>--30000.hf.jobs/v1`, reachable with an HF token that has read access to the job's namespace.

	## 2. Run the benchmark (Harbor + terminal-bench 3.0)

	Prerequisite: `git-lfs` must be installed (`brew install git-lfs && git lfs install`) or task binary fixtures fail to download correctly.

	```bash
	OPENAI_API_KEY="$(hf auth token)" harbor run \
	--agent terminus-2 \
	--model "openai/Qwen/Qwen3.8-27B-FP8" \
	--ak api_base="https://<job_id>--30000.hf.jobs/v1" \
	--repo https://huggingface.co/datasets/harborframework/terminal-bench-3.0 \
	--include-task-name atrx-vep-crispr \
	--n-tasks 1 \
	--job-name qwen3_8-27b-fp8-terminus2-smoketest-v3 \
	--n-concurrent 1 \
	-y
	```

	Notes:
	- `OPENAI_API_KEY` must be exported into the shell running `harbor`, not passed via `--ae` (that flag only injects env vars into the sandboxed task container, not the host process making the LLM call).
	- Drop `--include-task-name` and raise `--n-tasks` to run more of the dataset.

	## Result: single-task smoke test (terminal-bench 3.0)

	`atrx-vep-crispr__DvYyCpF`: completed with 0 exceptions, reward 0 (task itself unsolved — it's a 7h expert-time-estimate task). Runtime 3h6m, ~5.68M input / ~681K output tokens.

	## 3. Full run (terminal-bench 2.1, all 89 tasks, HF Sandbox environments)

	Running all 89 tasks locally via Docker is impractical (each task's environment container competes for local CPU/RAM). Instead use `--env hf-sandbox`, which runs each task's environment as its own HF Jobs sandbox instead of a local container — every task in terminal-bench 2.1 already references a prebuilt public `docker_image` in its `task.toml`, so no local builds are needed.

	```bash
	OPENAI_API_KEY="$(hf auth token)" harbor run \
	--agent terminus-2 \
	--model "openai/Qwen/Qwen3.8-27B-FP8" \
	--ak api_base="https://<job_id>--30000.hf.jobs/v1" \
	--ak reasoning_effort=low \
	--repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
	--env hf-sandbox \
	--ek flavor=cpu-upgrade \
	--agent-timeout-multiplier 4 \
	--job-name qwen3_8-27b-fp8-terminus2-tb21-full-v2 \
	--n-concurrent 3 \
	-y
	```

	Notes:
	- `--agent-timeout-multiplier 4`: this model is a heavy reasoner (burns lots of tokens/time per turn); the terminal-bench default per-task agent timeout (often 900s) is far too tight and causes widespread `AgentTimeoutError`. 4x (60 min/task) balances fairness against runtime.
	- `--ak reasoning_effort=low`: keeps the model's thinking budget in check so it doesn't blow through the timeout on easy tasks.
	- `--n-concurrent`: at `8`, ~70% of trials errored out with `SandboxError`/`InternalServerError`/`ReadTimeout` (HF Sandbox infra strain, not model failures). Dropping to `3` brought the error rate down substantially. Lower is more reliable but slower.
	- Mop up leftover infra errors without re-running everything:
	```bash
	# first, if you changed --n-concurrent, also edit it in the job's config.json (and lock.json to match) before resuming
	OPENAI_API_KEY="$(hf auth token)" harbor job resume \
	-p jobs/<job-name> \
	--filter-error-type SandboxError \
	--filter-error-type InternalServerError \
	--filter-error-type ReadTimeout \
	--filter-error-type ReadError \
	--filter-error-type ConnectTimeout \
	--filter-error-type RemoteProtocolError
	# deliberately NOT AgentTimeoutError — that's a legitimate result, not infra flakiness
	```
	Repeat as needed; each pass only touches the filtered trials, everything else is preserved.
	- Harbor's own printed end-of-run summary double-counts across resume passes (it doesn't dedupe eval stats between attempts) — don't trust the final `Trials`/`Exceptions`/`Mean` table it prints after a multi-pass run. Get the real per-task outcome by taking, for each task name, the trial directory with the latest `finished_at` in `result.json` (reward at `verifier_result.rewards.reward`, or `exception_info.exception_type` if it errored).

	### Final result (89/89 tasks, after 3 resume/mop-up passes)

	\| Outcome \| Count \| % of 89 \|
	\|---\|---\|---\|
	\| Passed \| 53 \| 59.6% \|
	\| Failed (genuine, reward 0) \| 16 \| 18.0% \|
	\| Errored (unscored) \| 20 \| 22.5% \|

	Pass rate among the 69 genuinely-scored tasks: 76.8%. Of the 20 errors, 16 are legitimate `AgentTimeoutError` (task too slow even at 4x budget); the remaining 4 (`circuit-fibsqrt`, `feal-linear-cryptanalysis`, `query-optimize`, `schemelike-metacircular-eval`) failed with `SandboxError`/`RemoteProtocolError` on every one of 3 separate retry attempts (across 2 different model-serving sessions) and were accepted as final errors — see the pitfall below before trying a 4th time.

	### Pitfall: resuming a job after the model server URL changes

	If the model-serving job dies (it will — HF Jobs `--timeout` is enforced eventually, just unpredictably; ours ran ~91h against a 24h setting before getting killed) and you relaunch it, do not `harbor job resume` the original job just because you updated `api_base` in its `config.json`/`lock.json`. `Job.create()`'s resume path matches each on-disk completed trial against a freshly-resolved `TrialConfig` via exact equality — which includes the agent's `kwargs`. Editing only the job-level files doesn't touch the ~85 individual completed trials' own per-trial `config.json`, so none of them match anymore and harbor concludes the entire job is still pending, silently re-running everything (including tasks you already solved) instead of just the ones you meant to retry. This wasted ~8 hours (a 2h + a 6h job) for us with zero output, because the runs actually landed under an unexpected `jobs/<job-name>/` bucket path — the default `harbor run`/`resume` subdirectory — not the flat job path we were watching, so it looked idle rather than busy-doing-the-wrong-thing.

	Do this instead: launch a small standalone `harbor run` (not `resume`) into a new job name, scoped to just the tasks you need via repeated `--include-task-name <name>`, pointed at the new endpoint. Merge results at final-tally time by scanning every relevant job directory and, per task name, keeping whichever trial has the latest `finished_at`.

	---

	# Run 2: gpt-oss-20b (terminal-bench 2.1)

	## 1. Launch the model server

	The model card's own recommended command (`uv pip install --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ ...`) pins a launch-day (Aug 2025) PyTorch nightly build that has since been pruned from the nightly index — it fails with `No solution found when resolving dependencies`. Used current stable vLLM instead:

	```bash
	hf jobs run \
	--flavor l40sx1 \
	--expose 8000 \
	--secrets HF_TOKEN \
	--timeout 12h \
	--label project=terminal-bench-harness \
	--label model=gpt-oss-20b \
	-d \
	vllm/vllm-openai:latest \
	vllm serve openai/gpt-oss-20b --host 0.0.0.0 --port 8000
	```

	Note: `hf jobs run <image> <command>` passes `<command>` as a literal argv, not through the image's shell/`ENTRYPOINT` — spell out the full command (`vllm serve openai/gpt-oss-20b ...`), don't rely on bare flags assuming an entrypoint wraps them (`--model ...` alone fails with `exec: "--model": executable file not found in $PATH`).

	## 2. Run the benchmark

	Same pattern as Run 1, orchestrator on its own HF Job with the bucket mounted — but written to land flat at the bucket root by passing `--jobs-dir .` after `cd /data` (the default `--jobs-dir jobs` would otherwise nest output at `<bucket>/jobs/<job-name>`, inconsistent with every other job dir here):

	```bash
	# inside the job container, after cd /data
	OPENAI_API_KEY="$HF_TOKEN" harbor run \
	--agent terminus-2 \
	--model "openai/openai/gpt-oss-20b" \
	--ak api_base="https://<job_id>--8000.hf.jobs/v1" \
	--repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
	--env hf-sandbox \
	--ek flavor=cpu-upgrade \
	--agent-timeout-multiplier 4 \
	--jobs-dir . \
	--job-name gpt-oss-20b-terminus2-tb21-full \
	--n-concurrent 3 \
	-y
	```

	(`--model "openai/openai/gpt-oss-20b"` is correct, not a typo — the LiteLLM `openai/` provider prefix plus the HF repo id `openai/gpt-oss-20b` just happen to both start with "openai".)

	Ran a 1-task smoke test first (`--n-tasks 1`, plain local Docker, no `--env hf-sandbox` needed for a single task) to confirm the endpoint/harness wiring before committing to the full run.

	### Final result (89/89 tasks, single clean pass, no resume needed)

	\| Outcome \| Count \| % of 89 \|
	\|---\|---\|---\|
	\| Passed \| 4 \| 4.5% \|
	\| Failed (genuine, incl. `AgentTimeoutError`) \| 81 \| 91.0% \|
	\| Errored (infra, excluded) \| 4 \| 4.5% \|

	Pass rate excluding infra errors: 4.71% (4/85) — spot-checked the 4 passes for real multi-episode agent activity (not degenerate zero-effort completions): confirmed genuine. HF Sandbox reliability was strong this run (only 4/89 excluded: 2 `SandboxError`, 1 `RemoteProtocolError`, 1 `CancelledError` from the outer job hitting its 12h timeout mid-trial at the very end — harbor's own process had already finished cleanly and printed its `89/89 Mean: 0.045` summary, but a lingering unclosed aiohttp session kept the container alive past that point until the timeout force-killed it). Only 3 genuine `AgentTimeoutError`s — this smaller/faster model wasn't timeout-bound the way the 27B reasoning model was; the whole 89-task run finished in ~13h vs. days for Run 1.

	For comparison: Qwen3.8-27B-FP8 scored 62.35% on the same dataset/harness (Run 1) — gpt-oss-20b genuinely struggled with terminal-bench 2.1's agentic coding tasks.

	---

	# Run 3: Qwen3.8-27B-FP8 with mini-swe-agent (terminal-bench 2.1)

	Same model as Run 1, different harness agent, to compare terminus-2 vs mini-swe-agent on identical model/dataset. `claude-code` was considered first but ruled out: it requires an `ANTHROPIC_BASE_URL` speaking the Anthropic Messages API, incompatible with sglang's OpenAI-format endpoint without a LiteLLM proxy bridge — not worth building for this comparison, so mini-swe-agent was used instead (its CLI runs inside the sandbox, so wiring is via `--ae`, not `--ak`, unlike terminus-2 — see agent-wiring notes below).

	## 1. Launch the model server

	```bash
	hf jobs run \
	--flavor h200 --expose 30000 --secrets HF_TOKEN --timeout 12h \
	--label project=terminal-bench-harness --label model=qwen3_8-27b-fp8 --label agent=mini-swe-agent \
	-d lmsysorg/sglang:latest \
	sglang serve \
	--trust-remote-code --model-path Qwen/Qwen3.8-27B-FP8 \
	--kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.85 \
	--attention-backend flashinfer --chunked-prefill-size 32768 --max-prefill-tokens 32768 \
	--reasoning-parser qwen3 --tool-call-parser qwen3_coder \
	--mamba-full-memory-ratio 3.67 --host 0.0.0.0 --port 30000 \
	--mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype float32 \
	--default-chat-template-kwargs '{"enable_thinking": false}'
	```

	Note the added `--default-chat-template-kwargs '{"enable_thinking": false}'` vs. Run 1 — Qwen3's reasoning/thinking mode was on by default and caused runaway generation (`AgentTimeoutError`s with suspiciously low output-token counts, e.g. 650 tokens over 3600s — confirmed via the server's own decode-batch logs showing `#full token` climbing continuously, and via non-zero `reasoning_tokens` in the trajectory JSON). Disabling it server-side fixed this (verified: `reasoning_content: null`, `reasoning_tokens: 0` on a test completion).

	A newer sglang cookbook variant adding speculative decoding (`--speculative-algorithm DFLASH` + draft model, `--mamba-full-memory-ratio 11.01`) was tried and OOM'd on H200 (confirmed CUDA OOM traceback) — reverted to the proven config above (no speculative decoding, ratio 3.67).

	HF Inference Providers were tried as a self-hosting alternative for a bit (`Qwen/Qwen3.8-27B:deepinfra`) but produced repeated `504 Gateway Time-out`s mid-run under sustained agentic load (LiteLLM retry-looping) — reverted to self-hosting.

	## 2. Run the benchmark

	```bash
	# orchestrator, as its own HF Job with the bucket mounted at /data (git-lfs required, jobs-dir . for flat bucket layout)
	pip install -q --pre "harbor[hf-sandbox]==0.22.1.dev202609020403" # see pipefail/dash fix note below
	harbor run \
	--agent mini-swe-agent \
	--model "openai/Qwen/Qwen3.8-27B-FP8" \
	--ae OPENAI_API_KEY="$HF_TOKEN" \
	--ae OPENAI_BASE_URL="https://<job_id>--30000.hf.jobs/v1" \
	--repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
	--env hf-sandbox \
	--ek flavor=cpu-upgrade \
	--ek job_timeout=2h \
	--agent-timeout-multiplier 4 \
	--jobs-dir . \
	--job-name qwen3_8-27b-fp8-mswea-tb21-full \
	--n-concurrent 3 \
	-y
	```

	Notes:
	- *mini-swe-agent's CLI runs inside* the sandbox** (unlike terminus-2's LiteLLM, which runs on the host) — so it's wired via `--ae OPENAI_API_KEY=... --ae OPENAI_BASE_URL=...` (env vars forwarded into the sandboxed exec), not `--ak api_base=...` + a host-exported key.
	- `set -o pipefail`/dash bug: hit `NonZeroAgentExitCodeError: set: Illegal option -o pipefail` on all 89 trials instantly (6 min) — harbor's `ensure_system_dependencies` prepends bash-only `set -o pipefail;`, but `--env hf-sandbox` execs via POSIX `/bin/sh` (dash), not bash (local Docker wraps in bash explicitly, hf-sandbox didn't). Fixed upstream in [harbor-framework/harbor#3016](https://github.com/harbor-framework/harbor/pull/3016); pin `harbor[hf-sandbox]==0.22.1.dev202609020403` (a `--pre` PyPI dev build) until it lands in a stable release.
	- Sandbox idle-timeout bug: `RemoteProtocolError`/`SandboxError` recurring on the same step even after retries — root cause found by checking the underlying ephemeral sandbox job's own logs (`hf jobs ps -a` to find it, then `hf jobs logs`), not harbor's wrapper: `sbx-server: idle for 601917ms, shutting down`. hf-sandbox's default 600s idle timeout was killing the sandbox mid-task because mini-swe-agent runs its whole agent loop as one long single exec call (no intermediate keep-alive activity harbor can see). Fixed with `--ek job_timeout=2h` (maps to the sandbox's own `idle_timeout`).

	### Final result (89/89 tasks, single pass, no resume needed)

	\| Outcome \| Count \| % of 89 \|
	\|---\|---\|---\|
	\| Passed \| 42 \| 47.2% \|
	\| Failed (genuine, incl. `AgentTimeoutError` + `ContextWindowExceededError` + 1 exec-timeout) \| 39 \| 43.8% \|
	\| Errored (infra, excluded) \| 8 \| 9.0% \|

	Pass rate excluding infra errors: 51.9% (42/81). Full per-task breakdown: `qwen3_8-27b-fp8-mswea-tb21-full/README.md` in this bucket.

	Of the 11 initial `NonZeroAgentExitCodeError` trials, only 3 were genuine infra (`qemu-startup`, `qemu-alpine-ssh`: apt-get failing on an expired Debian `bullseye-security` archive mirror; `make-doom-for-mips`: `/bin/sh: ps: not found` in a minimal task image) — the other 8 were real model/agent capability limits: 7 `ContextWindowExceededError` (mini-swe-agent resends full growing conversation history every turn; hit Qwen's 262144-token context limit on long tasks) and 1 exec-level timeout kill (`query-optimize`, agent launched a long-running background shell command and got cut off). None were a recurrence of the pipefail/dash bug above — that fix held across the full run.

	### Comparison: terminus-2 vs mini-swe-agent, same model (Qwen3.8-27B-FP8, terminal-bench 2.1)

	\| Agent \| Pass rate (excl. infra) \| Passed / Scored \|
	\|---\|---\|---\|
	\| terminus-2 \| 76.8% \| 53/69 \|
	\| mini-swe-agent \| 51.9% \| 42/81 \|

	terminus-2 substantially outperformed mini-swe-agent on the same model/dataset — plausibly related to mini-swe-agent's full-history-resend pattern (causing the context-window failures above) vs. terminus-2's context handling, though this wasn't isolated further.