Sync from GitHub c4e4dad

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+plots/qwen3_14b_training_reward_evidence_dashboard.png filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.venv/
+*.egg-info/
+.DS_Store

Dockerfile

@@ -0,0 +1,20 @@
+FROM python:3.12-slim
+
+ENV PYTHONDONTWRITEBYTECODE=1
+ENV PYTHONUNBUFFERED=1
+ENV PYTHONPATH=/app
+ENV ENABLE_WEB_INTERFACE=true
+
+WORKDIR /app
+
+COPY server/requirements.txt /tmp/server-requirements.txt
+RUN pip install --no-cache-dir -r /tmp/server-requirements.txt "openai>=1.0" "huggingface_hub>=1.0"
+
+COPY . ./legacy_cobol_env
+
+EXPOSE 8000
+
+HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
+    CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=2)"
+
+CMD ["python", "-m", "uvicorn", "legacy_cobol_env.server.app:app", "--host", "0.0.0.0", "--port", "8000"]

README.md

@@ -1,10 +1,166 @@
 ---
-title: Mainframe Modernization Openenv
-emoji: 💻
-colorFrom: purple
-colorTo: pink
+title: Legacy COBOL Migration Workbench
+emoji: 🧾
+colorFrom: blue
+colorTo: green
 sdk: docker
 pinned: false
+app_port: 8000
+base_path: /web
+tags:
+  - openenv
+  - cobol
+  - reinforcement-learning
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Legacy COBOL Migration Workbench
+
+Legacy COBOL Migration Workbench is an OpenEnv environment for training an LLM to act like a legacy modernization engineer. The agent receives a migration ticket, inspects COBOL programs and copybooks through tools, writes Python, runs visible tests, studies structured diffs, repairs its draft, and submits code scored on hidden and fresh tests.
+
+The target capability is not one-shot translation. Real modernization depends on fixed-width records, copybooks, implied decimals, `OCCURS` tables, level-88 condition names, branch precedence, and exact output formatting. This environment turns that workflow into a partially observable, tool-mediated training loop.
+
+## Submission Materials
+
+- Mini-blog: [`blog.md`](blog.md)
+- Hugging Face Space: <https://huggingface.co/spaces/Ishangtxl/mainframe-modernization-openenv>
+- Training evidence: [`outputs/training/sft_run_metadata.json`](outputs/training/sft_run_metadata.json)
+- Training log: [`outputs/training/hf_job_qwen3_14b_logs.txt`](outputs/training/hf_job_qwen3_14b_logs.txt)
+- Score summary: [`outputs/evals/score_summary.json`](outputs/evals/score_summary.json)
+
+## Environment
+
+Each episode starts with a partial ticket. The agent can discover details through MCP tools:
+
+- `read_cobol_file`
+- `read_copybook`
+- `parse_copybook_layout`
+- `inspect_business_rules`
+- `write_python_solution`
+- `run_visible_tests`
+- `inspect_diff`
+- `submit_final`
+
+The submitted Python code must define:
+
+```python
+def migrate(input_record: str) -> str:
+    ...
+```
+
+Episodes are capped at 24 tool steps. Visible tests are available for debugging, but hidden and fresh cases are withheld until final scoring.
+
+## Reward
+
+```text
+final_reward =
+  0.55 * hidden_correctness
++ 0.15 * fresh_correctness
++ 0.10 * interface_contract
++ 0.08 * type_and_layout_fidelity
++ 0.07 * anti_hardcoding
++ 0.05 * safety
+```
+
+The reward is correctness-heavy, but it also penalizes unsafe code, broken interfaces, layout drift, and visible-case hardcoding.
+
+## Task Families
+
+| Task | Difficulty | COBOL concepts | Main failure modes |
+| --- | --- | --- | --- |
+| `fixed_width_customer` | easy | `PIC X`, padding/truncation, status mapping | trimmed spaces, lost ZIP leading zeros, bad output width |
+| `decimal_copybook_payroll` | medium | copybook layout, implied decimals, level-88 bonus flag | float drift, wrong rounding, wrong fixed-width net pay |
+| `claims_eligibility_branching` | medium | `EVALUATE TRUE`, branch precedence | wrong first-match branch, boundary mistakes |
+| `account_status_level88` | medium | level-88 status conditions, signed amount | treating condition names as variables, wrong precedence |
+| `date_normalization` | medium | legacy YYMMDD windowing, validation | wrong century window, over-rejecting legacy dates |
+| `invoice_occurs_totals` | hard | multi-file `INVTOTAL.cbl`/`TAXRATE.cbl`, `OCCURS`, copybook tax-code metadata | wrong stride, ignoring tax-code lookup, overfitting visible invoice IDs |
+
+`inspect_business_rules` exposes agent-facing hints only. Exact reference behavior stays internal to hidden and fresh tests.
+
+## Results
+
+The current training run uses Hugging Face TRL LoRA SFT on `Qwen/Qwen3-14B` with 15 oracle and repair examples. Evaluation uses the same OpenEnv rollout harness before and after training.
+
+| Policy | Role | Mean public score | Accepted tasks |
+| --- | --- | ---: | ---: |
+| deterministic identity | deterministic baseline | 0.1500 | 0 / 6 |
+| deterministic blank width | deterministic baseline | 0.1767 | 0 / 6 |
+| base `Qwen/Qwen3-14B` | model before SFT | 0.5320 | 2 / 6 |
+| trained `Qwen/Qwen3-14B` LoRA SFT | model after SFT | 0.7971 | 4 / 6 |
+| `oracle-model` | sanity check with reference solutions | 1.0000 | 6 / 6 |
+
+![Qwen3-14B training reward evidence dashboard](plots/qwen3_14b_training_reward_evidence_dashboard.png)
+
+This dashboard summarizes the before/after reward evidence for the Qwen3-14B LoRA SFT run.
+
+![Qwen3-14B LoRA SFT loss](plots/qwen3_14b_loss_curve.png)
+
+The SFT run reduced loss from `1.1350` to `0.1924` and improved mean token accuracy from `0.7938` to `0.9483`.
+
+![Qwen3-14B reward before and after SFT](plots/qwen3_14b_reward_before_after.png)
+
+The trained checkpoint improved mean public reward from `0.5320` to `0.7971`.
+
+![Qwen3-14B accepted tasks before and after SFT](plots/qwen3_14b_accepted_before_after.png)
+
+Accepted tasks improved from `2 / 6` to `4 / 6`.
+
+## Training Artifacts
+
+Kept in the submission root:
+
+- `outputs/training/sft_run_metadata.json`: completed-run metadata for the Qwen3-14B LoRA SFT run
+- `outputs/training/sft_loss.csv`: real training loss and token-accuracy history
+- `outputs/training/qwen3_14b_training_evidence_summary.json`: compact extracted training/eval summary
+- `outputs/training/hf_job_qwen3_14b_logs.txt`: raw HF Jobs log for provenance
+- `outputs/evals/base_qwen3_14b_all_tasks.json`: before-training rollout
+- `outputs/evals/score_summary.json`: judge-facing summary with deterministic, base-model, trained-model, and oracle rows
+- `plots/qwen3_14b_training_reward_evidence_dashboard.png`
+- `plots/qwen3_14b_loss_curve.png`
+- `plots/qwen3_14b_reward_before_after.png`
+- `plots/qwen3_14b_accepted_before_after.png`
+
+The old dry-run `sft_loss.svg` scaffold is intentionally not included as judge-facing training evidence.
+
+## Historical Artifacts
+
+Older Azure `gpt-5.4-mini` rollouts are archived under `outputs/evals/historical/`. They were produced before the invoice task was hardened into a multi-file tax-code task, so they are preserved for context but excluded from the current score summary.
+
+## Run Locally
+
+```bash
+python -m venv .venv
+.venv/bin/pip install -e ".[dev]"
+PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=. .venv/bin/pytest tests -q -p no:cacheprovider
+```
+
+Run the server:
+
+```bash
+PYTHONPATH=. .venv/bin/python -m uvicorn legacy_cobol_env.server.app:app --host 127.0.0.1 --port 8000
+curl -sS http://127.0.0.1:8000/health
+curl -sS http://127.0.0.1:8000/schema
+```
+
+Validate the OpenEnv package:
+
+```bash
+.venv/bin/openenv validate --verbose
+```
+
+Regenerate the score summary:
+
+```bash
+.venv/bin/python -m legacy_cobol_env.eval.run_evidence_report
+```
+
+Run the root inference script:
+
+```bash
+API_BASE_URL="https://..." MODEL_NAME="..." HF_TOKEN="..." .venv/bin/python inference.py --max-repairs 1
+```
+
+The inference script emits `[START]`, one `[STEP]` per task, and `[END]`.
+
+## Safety Note
+
+Candidate code is checked for common unsafe imports and builtins, executed in a temporary subprocess with cleared environment variables, and bounded by a timeout. This is layered mitigation for a hackathon environment, not a complete secure sandbox.

__init__.py

@@ -0,0 +1,10 @@
+"""Legacy COBOL Migration Workbench."""
+
+try:
+    from .client import LegacyCobolEnv
+    from .models import LegacyCobolState
+except ImportError:
+    from client import LegacyCobolEnv
+    from models import LegacyCobolState
+
+__all__ = ["LegacyCobolEnv", "LegacyCobolState"]

blog.md

@@ -0,0 +1,78 @@
+# Replacing Ourselves at the Job
+
+Hi. We're QuietLab.
+
+We're Ishan and Niranjan, and on most Mondays we put on our IBM badges and go work on mainframe modernization at IBM. Specifically: on the tooling that helps engineers move COBOL code off the mainframe, with an AI assistant that suggests the modernized version.
+
+So when this hackathon was announced, picking the topic took about thirty seconds.
+
+## A two-paragraph mainframe primer
+
+Mainframes are big computers from an earlier era of computing. IBM mostly. They're physically huge, eye-wateringly expensive, and absurdly reliable. They don't go down, they don't really break, and they process transactions at a scale the cloud still finds embarrassing. About 95% of US ATM transactions touch one. Roughly $3 trillion in daily commerce flows across them. By IBM's count, ~70% of global business transactions still pass through a mainframe somewhere in the pipeline.
+
+These machines mostly run code written in COBOL, a language designed in 1959. Despite "COBOL is dying" being declared once a decade for the last fifty years, there are around **220 billion lines of it in active production today**. Banks, insurance companies, governments, airlines, payroll, tax systems. The plumbing of the world.
+
+## The not-so-fun part
+
+The engineers who wrote this stuff are retiring. The average COBOL developer is now 55. About 10% of that workforce hits retirement every year. Universities stopped teaching the language somewhere around 1995. The result is a quiet but compounding crisis: the code that runs the world is being maintained by a shrinking group of people who are increasingly tired and would frankly like to be golfing.
+
+So companies are paying enormous amounts of money to migrate off it. The mainframe modernization market is roughly **$8.4B in 2025**, projected to hit **$13.3B by 2030**, and that's the narrow slice. The broader application modernization market is on track for **$100B+ by 2035**. Replacing a single working COBOL system routinely costs upward of $50 million. AWS, IBM, Capgemini, Accenture, TCS: every major consulting firm has a dedicated practice for it. Most of the migrations target Java or Python.
+
+There's a temptation to think AI just solves this. In February 2026, Anthropic published a blog suggesting Claude could "rapidly refactor" COBOL codebases. IBM's stock dropped 13% in a single day, its worst crash since 2000, and Wall Street briefly believed the mainframe business was dead.
+
+Then everyone calmed down, because IBM SVP Rob Thomas pointed out the obvious: **translation isn't modernization.** Real legacy code lives inside fixed-width records, copybooks, implied decimals, level-88 condition names, branch precedence, weird date semantics, and small formatting details that have to be preserved exactly or your entire payroll system silently breaks at midnight. A model that one-shots COBOL into Python is a toy. A model that actually does the *work* of a modernization engineer is the unsolved problem.
+
+## The pitch
+
+We don't love the job.
+
+So, like any reasonable software engineers, we tried to build a thing that might one day replace us at it.
+
+## What we built
+
+**Legacy COBOL Migration Workbench**: an OpenEnv environment where a small LLM is trained, via reinforcement learning, to behave less like a translator and more like a careful modernization engineer.
+
+Each episode is one migration ticket. The agent receives a high-level request and a list of available files, and that's it. From there it has tools:
+
+- `read_cobol_file`: read a COBOL source file
+- `read_copybook`: read a referenced copybook
+- `parse_copybook_layout`: extract field offsets, lengths, scales, condition names
+- `inspect_business_rules`: surface hints about branch precedence and edge cases
+- `write_python_solution`: draft a Python implementation
+- `run_visible_tests`: try the draft against a few sample cases
+- `inspect_diff`: see exactly where output disagrees with expected
+- `submit_final`: end the episode
+
+The required output is a Python `migrate(input_record: str) -> str` that preserves the fixed-width record contract exactly.
+
+The catch, and the whole point, is that we grade on **hidden tests** the agent never sees, plus **freshly generated tests** at scoring time. So the model can't win by memorizing the visible cases. It actually has to understand the legacy behaviour.
+
+The reward is multi-component on purpose: hidden correctness, fresh correctness, interface contract, type and layout fidelity, an explicit anti-hardcoding signal, and sandbox safety. We learned the hard way during dry runs that any single reward gets gamed. Six independent signals are much harder to fool.
+
+The task bank covers six families pulled from problems we've actually seen at work: customer records with fixed-width string fields, payroll with copybooks and signed deductions, claims eligibility with `EVALUATE TRUE` branching, account status driven by level-88 condition names, multi-file invoices with `OCCURS` groups and an external tax-rate program, and date normalization with the two-digit-year windowing rules COBOL is famous for.
+
+The environment is hosted on Hugging Face Spaces. Training is GRPO via TRL, with Unsloth handling the efficient inference. The whole thing speaks the standard OpenEnv `reset` / `step` / `state` interface.
+
+## The point of the demo
+
+If the trained agent just produces more Python, we've failed. The point isn't volume. It's *behaviour*. What we want to show is a trained model doing the things a junior modernization engineer does:
+
+inspect before acting, read the copybook before guessing field offsets, run tests before submitting, read the diff when something fails, and then fix the actual mistake instead of writing more code on top of it.
+
+That's the kind of skill an environment can teach that a static dataset can't.
+
+## What's next
+
+We didn't get to **Java** in 48 hours, and that's the obvious next target. Most real enterprise mainframe modernization goes COBOL → Java, not COBOL → Python. Same environment shape, different code generator. The reward design transfers cleanly because it's all behavioural, not language-specific.
+
+After that: **JCL → shell/Python** for batch jobs, multi-program CICS-style transaction flows, and eventually wiring the workbench into an actual Zowe CLI loop so the agent operates against a live z/OS sandbox instead of pre-computed test fixtures. That last one is where it stops being a hackathon project and starts being something IBM customers might genuinely use.
+
+Until one of them does, we'll be at our desks.
+
+Modernizing COBOL.
+
+Manually.
+
+For now.
+
+Ishan & Niranjan, Team QuietLab

client.py

@@ -0,0 +1,9 @@
+"""Client for the Legacy COBOL Migration Workbench."""
+
+from openenv.core.mcp_client import MCPToolClient
+
+
+class LegacyCobolEnv(MCPToolClient):
+    """MCP tool client for the workbench."""
+
+    pass

cobol_oracles/invoice/Dockerfile

@@ -0,0 +1,13 @@
+FROM debian:bookworm-slim
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends gnucobol ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /oracle
+
+COPY INVOICE_ORACLE.cbl TAXRATE.cbl INVOICE_REC.cpy TAX_CODE.cpy ./
+RUN cobc -x -free -o invoice_oracle INVOICE_ORACLE.cbl TAXRATE.cbl
+
+WORKDIR /work
+ENTRYPOINT ["/oracle/invoice_oracle"]

cobol_oracles/invoice/INVOICE_ORACLE.cbl

@@ -0,0 +1,85 @@
+       >>SOURCE FORMAT FREE
+       IDENTIFICATION DIVISION.
+       PROGRAM-ID. INVOICE-ORACLE.
+
+       ENVIRONMENT DIVISION.
+       INPUT-OUTPUT SECTION.
+       FILE-CONTROL.
+           SELECT INPUT-FILE ASSIGN TO "input.txt"
+               ORGANIZATION IS LINE SEQUENTIAL.
+           SELECT OUTPUT-FILE ASSIGN TO "output.txt"
+               ORGANIZATION IS LINE SEQUENTIAL.
+
+       DATA DIVISION.
+       FILE SECTION.
+       FD INPUT-FILE.
+       01 INPUT-LINE                 PIC X(44).
+       FD OUTPUT-FILE.
+       01 OUTPUT-LINE                PIC X(18).
+
+       WORKING-STORAGE SECTION.
+       COPY INVOICE_REC.
+       COPY TAX_CODE.
+       01 EOF-FLAG                   PIC X VALUE "N".
+       01 WS-IDX                     PIC 9 VALUE 0.
+       01 WS-ITEM-START              PIC 99 VALUE 0.
+       01 WS-PRICE-START             PIC 99 VALUE 0.
+       01 WS-TAX-START               PIC 99 VALUE 0.
+       01 WS-QTY                     PIC 99 VALUE 0.
+       01 WS-PRICE-CENTS             PIC 9(6) VALUE 0.
+       01 WS-LINE-CENTS              PIC 9(9) VALUE 0.
+       01 WS-TAX-CENTS               PIC 9(9) VALUE 0.
+       01 WS-TAX-PERCENT             PIC 9V9999 VALUE 0.
+       01 WS-TOTAL-CENTS             PIC 9(9) VALUE 0.
+       01 OUTPUT-RECORD.
+          05 OUT-INVOICE-ID          PIC X(6).
+          05 OUT-TOTAL               PIC 9(9).
+          05 OUT-ITEM-COUNT          PIC 99.
+          05 OUT-FLAG                PIC X.
+
+       PROCEDURE DIVISION.
+           OPEN INPUT INPUT-FILE
+           OPEN OUTPUT OUTPUT-FILE
+           PERFORM UNTIL EOF-FLAG = "Y"
+              READ INPUT-FILE
+                 AT END
+                    MOVE "Y" TO EOF-FLAG
+                 NOT AT END
+                    PERFORM MIGRATE-INVOICE
+                    MOVE OUTPUT-RECORD TO OUTPUT-LINE
+                    WRITE OUTPUT-LINE
+              END-READ
+           END-PERFORM
+           CLOSE INPUT-FILE
+           CLOSE OUTPUT-FILE
+           STOP RUN.
+
+       MIGRATE-INVOICE.
+           MOVE INPUT-LINE TO INVOICE-RECORD
+           MOVE ZERO TO WS-TOTAL-CENTS
+           IF ITEM-COUNT > 4
+              MOVE 4 TO ITEM-COUNT
+           END-IF
+
+           PERFORM VARYING WS-IDX FROM 1 BY 1 UNTIL WS-IDX > ITEM-COUNT
+              COMPUTE WS-ITEM-START = 9 + ((WS-IDX - 1) * 9)
+              COMPUTE WS-PRICE-START = WS-ITEM-START + 2
+              COMPUTE WS-TAX-START = WS-ITEM-START + 8
+              MOVE FUNCTION NUMVAL(INPUT-LINE(WS-ITEM-START:2)) TO WS-QTY
+              MOVE FUNCTION NUMVAL(INPUT-LINE(WS-PRICE-START:6)) TO WS-PRICE-CENTS
+              MOVE INPUT-LINE(WS-TAX-START:1) TO TAX-CODE(WS-IDX)
+              COMPUTE WS-LINE-CENTS = WS-QTY * WS-PRICE-CENTS
+              CALL "TAXRATE" USING TAX-CODE(WS-IDX) WS-TAX-PERCENT
+              COMPUTE WS-TAX-CENTS ROUNDED = WS-LINE-CENTS * WS-TAX-PERCENT
+              ADD WS-TAX-CENTS TO WS-LINE-CENTS
+              ADD WS-LINE-CENTS TO WS-TOTAL-CENTS
+           END-PERFORM
+
+           MOVE INVOICE-ID TO OUT-INVOICE-ID
+           MOVE WS-TOTAL-CENTS TO OUT-TOTAL
+           MOVE ITEM-COUNT TO OUT-ITEM-COUNT
+           IF WS-TOTAL-CENTS >= 100000
+              MOVE "H" TO OUT-FLAG
+           ELSE
+              MOVE "L" TO OUT-FLAG
+           END-IF.

cobol_oracles/invoice/INVOICE_REC.cpy

@@ -0,0 +1,7 @@
+       01  INVOICE-RECORD.
+           05 INVOICE-ID             PIC X(6).
+           05 ITEM-COUNT             PIC 9(2).
+           05 LINE-ITEM OCCURS 4 TIMES.
+              10 ITEM-QTY            PIC 9(2).
+              10 ITEM-PRICE          PIC 9(4)V99.
+              10 TAX-CODE            PIC X.

cobol_oracles/invoice/TAXRATE.cbl

@@ -0,0 +1,17 @@
+       >>SOURCE FORMAT FREE
+       IDENTIFICATION DIVISION.
+       PROGRAM-ID. TAXRATE.
+
+       DATA DIVISION.
+       LINKAGE SECTION.
+       01 LK-TAX-CODE                PIC X.
+       01 LK-TAX-PERCENT             PIC 9V9999.
+
+       PROCEDURE DIVISION USING LK-TAX-CODE LK-TAX-PERCENT.
+           EVALUATE LK-TAX-CODE
+              WHEN "S" MOVE 0.0725 TO LK-TAX-PERCENT
+              WHEN "R" MOVE 0.0250 TO LK-TAX-PERCENT
+              WHEN "L" MOVE 0.1000 TO LK-TAX-PERCENT
+              WHEN OTHER MOVE 0.0000 TO LK-TAX-PERCENT
+           END-EVALUATE
+           GOBACK.

cobol_oracles/invoice/TAX_CODE.cpy

@@ -0,0 +1,4 @@
+       01  TAX-CODE-TABLE.
+           05 TAX-CODE-ENTRY OCCURS 4 TIMES.
+              10 TAX-CODE-KEY        PIC X.
+              10 TAX-RATE            PIC 9V9999.

eval/__init__.py

@@ -0,0 +1,2 @@
+"""Evaluation helpers for Legacy COBOL Migration Workbench."""
+

eval/cobol_oracle.py

@@ -0,0 +1,160 @@
+"""Dockerized COBOL oracle checks for the hard invoice task."""
+
+from __future__ import annotations
+
+import json
+import shutil
+import subprocess
+import tempfile
+from pathlib import Path
+from typing import Any
+
+from legacy_cobol_env.server import task_bank
+from legacy_cobol_env.server.task_bank import TestCase, generate_fresh_tests, load_task
+
+
+ENV_ROOT = Path(__file__).resolve().parents[1]
+INVOICE_ORACLE_DIR = ENV_ROOT / "cobol_oracles" / "invoice"
+DEFAULT_IMAGE_TAG = "legacy-cobol-invoice-oracle:local"
+
+
+def invoice_oracle_cases(include_fresh: bool = True) -> list[TestCase]:
+    task = load_task(task_id="invoice_occurs_001")
+    cases = [*task.visible_tests, *task.hidden_tests]
+    if include_fresh:
+        cases.extend(generate_fresh_tests(task))
+    return cases
+
+
+def build_invoice_oracle_image(image_tag: str = DEFAULT_IMAGE_TAG) -> None:
+    _require_docker()
+    subprocess.run(
+        ["docker", "build", "-t", image_tag, str(INVOICE_ORACLE_DIR)],
+        check=True,
+        text=True,
+    )
+
+
+def compile_invoice_task_sources(
+    image_tag: str = DEFAULT_IMAGE_TAG,
+    build: bool = True,
+) -> dict[str, Any]:
+    _require_docker()
+    if build:
+        build_invoice_oracle_image(image_tag=image_tag)
+
+    with tempfile.TemporaryDirectory(prefix="invoice-task-compile-") as tmp:
+        workdir = Path(tmp)
+        (workdir / "INVTOTAL.cbl").write_text(task_bank.INVOICE_COBOL, encoding="utf-8")
+        (workdir / "TAXRATE.cbl").write_text(task_bank.TAXRATE_COBOL, encoding="utf-8")
+        (workdir / "INVOICE_REC.cpy").write_text(task_bank.INVOICE_COPYBOOK, encoding="utf-8")
+        (workdir / "TAX_CODE.cpy").write_text(task_bank.TAX_CODE_COPYBOOK, encoding="utf-8")
+        completed = subprocess.run(
+            [
+                "docker",
+                "run",
+                "--rm",
+                "--entrypoint",
+                "sh",
+                "-v",
+                f"{workdir}:/work",
+                "-w",
+                "/work",
+                image_tag,
+                "-lc",
+                "cobc -m -free INVTOTAL.cbl TAXRATE.cbl",
+            ],
+            check=False,
+            text=True,
+            capture_output=True,
+        )
+
+    return {
+        "ok": completed.returncode == 0,
+        "compiled_sources": ["INVTOTAL.cbl", "TAXRATE.cbl"],
+        "returncode": completed.returncode,
+        "stdout": completed.stdout,
+        "stderr": completed.stderr,
+    }
+
+
+def run_invoice_oracle(
+    records: list[str],
+    image_tag: str = DEFAULT_IMAGE_TAG,
+    build: bool = False,
+) -> list[str]:
+    _require_docker()
+    if build:
+        build_invoice_oracle_image(image_tag=image_tag)
+
+    with tempfile.TemporaryDirectory(prefix="invoice-cobol-oracle-") as tmp:
+        workdir = Path(tmp)
+        input_path = workdir / "input.txt"
+        output_path = workdir / "output.txt"
+        input_path.write_text("".join(f"{record}\n" for record in records), encoding="utf-8")
+
+        subprocess.run(
+            [
+                "docker",
+                "run",
+                "--rm",
+                "-v",
+                f"{workdir}:/work",
+                image_tag,
+            ],
+            check=True,
+            text=True,
+            capture_output=True,
+        )
+        outputs = output_path.read_text(encoding="utf-8").splitlines()
+
+    if len(outputs) != len(records):
+        raise RuntimeError(f"COBOL oracle returned {len(outputs)} outputs for {len(records)} records")
+    return outputs
+
+
+def compare_invoice_oracle(
+    image_tag: str = DEFAULT_IMAGE_TAG,
+    build: bool = False,
+    include_fresh: bool = True,
+) -> dict[str, Any]:
+    cases = invoice_oracle_cases(include_fresh=include_fresh)
+    actual_outputs = run_invoice_oracle(
+        [case.input_record for case in cases],
+        image_tag=image_tag,
+        build=build,
+    )
+    rows = []
+    mismatches = []
+    for case, actual in zip(cases, actual_outputs, strict=True):
+        passed = actual == case.expected_output
+        row = {
+            "case_id": case.case_id,
+            "summary": case.summary,
+            "input_record": case.input_record,
+            "expected_output": case.expected_output,
+            "actual_output": actual,
+            "passed": passed,
+        }
+        rows.append(row)
+        if not passed:
+            mismatches.append(row)
+
+    return {
+        "ok": not mismatches,
+        "image_tag": image_tag,
+        "case_count": len(cases),
+        "passed_count": len(cases) - len(mismatches),
+        "mismatches": mismatches,
+        "cases": rows,
+    }
+
+
+def write_comparison_report(result: dict[str, Any], output_path: Path) -> None:
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(json.dumps(result, indent=2), encoding="utf-8")
+
+
+def _require_docker() -> None:
+    if shutil.which("docker") is None:
+        raise RuntimeError("docker is required for COBOL oracle checks")

eval/evidence_report.py

@@ -0,0 +1,197 @@
+"""Build judge-facing score summaries from evaluation artifacts."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+def build_score_summary(
+    baseline: dict[str, Any],
+    zeroshot: dict[str, Any] | None = None,
+    repair: dict[str, Any] | None = None,
+    oracle_model: dict[str, Any] | None = None,
+    base_model: dict[str, Any] | None = None,
+    trained_summary: dict[str, Any] | None = None,
+    evidence_notes: list[str] | None = None,
+) -> dict[str, Any]:
+    baseline_task_count = baseline.get("task_count") or len({row["task_id"] for row in baseline["results"]})
+    policies = {
+        name: {
+            "mean_public_score": score,
+            "accepted_count": sum(
+                1 for row in baseline["results"] if row["policy"] == name and row.get("accepted")
+            ),
+            "task_count": baseline_task_count,
+        }
+        for name, score in baseline["mean_public_score"].items()
+    }
+    if oracle_model is not None:
+        policies["oracle-model"] = {
+            "mean_public_score": oracle_model["mean_public_score"],
+            "accepted_count": oracle_model["accepted_count"],
+            "task_count": oracle_model["task_count"],
+            "role": "oracle_sanity",
+        }
+    if base_model is not None:
+        policies["base Qwen3-14B"] = {
+            "mean_public_score": base_model["mean_public_score"],
+            "accepted_count": base_model["accepted_count"],
+            "task_count": base_model["task_count"],
+            "role": "base_model_before_sft",
+        }
+    if zeroshot is not None:
+        policies["gpt-5.4-mini zero-shot"] = {
+            "mean_public_score": zeroshot["mean_public_score"],
+            "accepted_count": zeroshot["accepted_count"],
+            "task_count": zeroshot["task_count"],
+            "role": "historical_model_baseline",
+        }
+    if repair is not None:
+        policies["gpt-5.4-mini + repair1"] = {
+            "mean_public_score": repair["mean_public_score"],
+            "accepted_count": repair["accepted_count"],
+            "task_count": repair["task_count"],
+            "role": "historical_model_repair",
+        }
+    if trained_summary is not None:
+        policies["trained Qwen3-14B LoRA SFT"] = {
+            "mean_public_score": trained_summary["trained_mean_public_score"],
+            "accepted_count": trained_summary["trained_accepted_count"],
+            "task_count": trained_summary["task_count"],
+            "role": "trained_model_after_sft",
+            "model": trained_summary.get("model"),
+            "method": trained_summary.get("method"),
+        }
+
+    task_scores: dict[str, dict[str, Any]] = {}
+    for row in baseline["results"]:
+        task_scores.setdefault(row["task_id"], {})[row["policy"]] = row.get("public_score")
+    if oracle_model is not None:
+        for trajectory in oracle_model["trajectories"]:
+            task_scores.setdefault(trajectory["task_id"], {})["oracle_model"] = trajectory["final"]["public_score"]
+    if base_model is not None:
+        for trajectory in base_model["trajectories"]:
+            task_scores.setdefault(trajectory["task_id"], {})["base_qwen3_14b"] = trajectory["final"]["public_score"]
+    if zeroshot is not None:
+        for trajectory in zeroshot["trajectories"]:
+            task_scores.setdefault(trajectory["task_id"], {})["zeroshot"] = trajectory["final"]["public_score"]
+    if repair is not None:
+        for trajectory in repair["trajectories"]:
+            task_scores.setdefault(trajectory["task_id"], {})["repair1"] = trajectory["final"]["public_score"]
+            task_scores[trajectory["task_id"]]["repair1_accepted"] = trajectory["final"]["accepted"]
+
+    training_targets = []
+    if repair is not None:
+        for trajectory in repair["trajectories"]:
+            final = trajectory["final"]
+            visible = trajectory["visible"]
+            if visible["pass_rate"] == 1.0 and not final["accepted"]:
+                training_targets.append(
+                    {
+                        "task_id": trajectory["task_id"],
+                        "family_id": trajectory.get("family_id"),
+                        "reason": "visible-pass-hidden-fresh-gap",
+                        "public_score": final["public_score"],
+                        "weak_components": {
+                            key: value
+                            for key, value in final["components"].items()
+                            if isinstance(value, int | float) and value < 1.0
+                        },
+                    }
+                )
+
+    return {
+        "policies": policies,
+        "task_scores": task_scores,
+        "judge_table": _judge_table(policies),
+        "training_evidence": _training_evidence(trained_summary),
+        "training_targets": training_targets,
+        "evidence_notes": evidence_notes or [],
+    }
+
+
+def _judge_table(policies: dict[str, dict[str, Any]]) -> list[dict[str, Any]]:
+    return [
+        {
+            "policy": policy,
+            "role": data.get("role"),
+            "mean_public_score": data["mean_public_score"],
+            "accepted_count": data["accepted_count"],
+            "task_count": data["task_count"],
+        }
+        for policy, data in policies.items()
+    ]
+
+
+def _training_evidence(trained_summary: dict[str, Any] | None) -> dict[str, Any] | None:
+    if trained_summary is None:
+        return None
+    return {
+        "model": trained_summary.get("model"),
+        "method": trained_summary.get("method"),
+        "dataset_examples": trained_summary.get("dataset_examples"),
+        "loss_first": trained_summary.get("loss_first"),
+        "loss_last": trained_summary.get("loss_last"),
+        "mean_token_accuracy_first": trained_summary.get("mean_token_accuracy_first"),
+        "mean_token_accuracy_last": trained_summary.get("mean_token_accuracy_last"),
+        "base_mean_public_score": trained_summary.get("base_mean_public_score"),
+        "trained_mean_public_score": trained_summary.get("trained_mean_public_score"),
+        "base_accepted_count": trained_summary.get("base_accepted_count"),
+        "trained_accepted_count": trained_summary.get("trained_accepted_count"),
+        "task_count": trained_summary.get("task_count"),
+    }
+
+
+def write_score_plot(summary: dict[str, Any], path: Path) -> None:
+    policies = list(summary["policies"].items())
+    width = 760
+    height = 320
+    margin = 48
+    plot_height = height - margin * 2
+    bar_width = 86
+    gap = 32
+    colors = ["#64748b", "#22c55e", "#2563eb", "#f97316"]
+    parts = [
+        f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" viewBox="0 0 {width} {height}">',
+        '<rect width="100%" height="100%" fill="#ffffff"/>',
+        '<text x="40" y="30" font-family="Arial" font-size="18" font-weight="700">Legacy COBOL OpenEnv evaluation</text>',
+        f'<line x1="{margin}" y1="{height - margin}" x2="{width - margin}" y2="{height - margin}" stroke="#1f2937"/>',
+        f'<line x1="{margin}" y1="{height - margin}" x2="{margin}" y2="{margin}" stroke="#1f2937"/>',
+    ]
+    x = margin + 20
+    for index, (name, data) in enumerate(policies):
+        score = float(data["mean_public_score"])
+        bar_height = score * plot_height
+        y = height - margin - bar_height
+        color = colors[index % len(colors)]
+        parts.append(f'<rect x="{x}" y="{y:.1f}" width="{bar_width}" height="{bar_height:.1f}" rx="4" fill="{color}"/>')
+        parts.append(f'<text x="{x + 8}" y="{y - 8:.1f}" font-family="Arial" font-size="12">{score:.3f}</text>')
+        accepted = f'{data["accepted_count"]}/{data["task_count"]} accepted'
+        parts.append(f'<text x="{x}" y="{height - 28}" font-family="Arial" font-size="10">{accepted}</text>')
+        for line_index, line in enumerate(_wrap_label(name)):
+            parts.append(f'<text x="{x}" y="{height - 14 + line_index * 11}" font-family="Arial" font-size="10">{line}</text>')
+        x += bar_width + gap
+    parts.append("</svg>")
+    path.write_text("\n".join(parts), encoding="utf-8")
+
+
+def _wrap_label(label: str) -> list[str]:
+    words = label.split()
+    lines: list[str] = []
+    current = ""
+    for word in words:
+        candidate = f"{current} {word}".strip()
+        if len(candidate) > 18 and current:
+            lines.append(current)
+            current = word
+        else:
+            current = candidate
+    if current:
+        lines.append(current)
+    return lines[:2]
+
+
+def load_json(path: Path) -> dict[str, Any]:
+    return json.loads(path.read_text(encoding="utf-8"))

eval/inspect_rollout.py

@@ -0,0 +1,114 @@
+"""Inspect saved model rollout artifacts for debugging."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from legacy_cobol_env.eval.model_rollout import extract_code_from_response
+
+
+STEP_NAMES_TO_SHOW = {"write_python_solution", "run_visible_tests", "inspect_diff", "submit_final"}
+
+
+def inspect_rollout_file(path: Path, max_chars: int = 4000) -> str:
+    summary = json.loads(path.read_text(encoding="utf-8"))
+    return inspect_rollout_summary(summary, max_chars=max_chars)
+
+
+def inspect_rollout_summary(summary: dict[str, Any], max_chars: int = 4000) -> str:
+    lines: list[str] = []
+    lines.append("SUMMARY")
+    lines.append(
+        _json_preview(
+            {
+                key: summary.get(key)
+                for key in ["provider", "max_repairs", "task_count", "mean_public_score", "accepted_count"]
+                if key in summary
+            },
+            max_chars,
+        )
+    )
+
+    trajectories = summary.get("trajectories", [])
+    if not trajectories:
+        lines.append("\nNO TRAJECTORIES")
+        return "\n".join(lines)
+
+    for index, trajectory in enumerate(trajectories, start=1):
+        lines.append(f"\nTRAJECTORY {index}")
+        lines.append(
+            _json_preview(
+                {
+                    key: trajectory.get(key)
+                    for key in ["task_id", "family_id", "policy"]
+                    if key in trajectory
+                },
+                max_chars,
+            )
+        )
+        lines.append("\nFINAL")
+        lines.append(_json_preview(trajectory.get("final", {}), max_chars))
+        lines.append("\nVISIBLE")
+        lines.append(_json_preview(trajectory.get("visible", {}), max_chars))
+        lines.append("\nMODEL TURNS")
+        lines.extend(_format_model_turns(trajectory.get("model_turns", []), max_chars))
+        lines.append("\nWRITE / TEST STEPS")
+        lines.extend(_format_steps(trajectory.get("steps", []), max_chars))
+
+    return "\n".join(lines)
+
+
+def _format_model_turns(model_turns: list[dict[str, Any]], max_chars: int) -> list[str]:
+    lines: list[str] = []
+    for index, turn in enumerate(model_turns, start=1):
+        response = str(turn.get("response", ""))
+        lines.append(f"\n--- turn {index} raw response ---")
+        lines.append(_text_preview(response, max_chars))
+        lines.append(f"\n--- turn {index} extracted code ---")
+        try:
+            lines.append(_text_preview(extract_code_from_response(response), max_chars))
+        except Exception as exc:  # pragma: no cover - exact parser errors vary.
+            lines.append(f"extract failed: {exc!r}")
+    if not lines:
+        lines.append("(none)")
+    return lines
+
+
+def _format_steps(steps: list[dict[str, Any]], max_chars: int) -> list[str]:
+    lines: list[str] = []
+    for step in steps:
+        tool_name = step.get("tool_name")
+        if tool_name not in STEP_NAMES_TO_SHOW:
+            continue
+        lines.append(f"\n{tool_name}")
+        lines.append(_json_preview(step.get("result", {}), max_chars))
+    if not lines:
+        lines.append("(none)")
+    return lines
+
+
+def _json_preview(value: Any, max_chars: int) -> str:
+    return _text_preview(json.dumps(value, indent=2, sort_keys=True), max_chars)
+
+
+def _text_preview(value: str, max_chars: int) -> str:
+    if len(value) <= max_chars:
+        return value
+    omitted = len(value) - max_chars
+    return f"{value[:max_chars]}\n... <truncated {omitted} chars>"
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Print the useful parts of a saved rollout JSON artifact.")
+    parser.add_argument("path", type=Path, help="Path to a run_model_rollouts JSON output.")
+    parser.add_argument("--max-chars", type=int, default=4000, help="Maximum characters per response/result block.")
+    args = parser.parse_args()
+
+    print(inspect_rollout_file(args.path, max_chars=args.max_chars))
+
+
+if __name__ == "__main__":
+    main()

eval/model_rollout.py

@@ -0,0 +1,460 @@
+"""Run provider-backed code-generation rollouts against the workbench."""
+
+from __future__ import annotations
+
+import json
+import ast
+import re
+from typing import Any
+
+from legacy_cobol_env.eval.providers import TextProvider
+from legacy_cobol_env.eval.trajectory import call_tool
+from legacy_cobol_env.server.sandbox import ALLOWED_IMPORTS, FORBIDDEN_IMPORTS
+from legacy_cobol_env.server.legacy_cobol_env_environment import LegacyCobolEnvironment
+from legacy_cobol_env.server.task_bank import TaskInstance
+
+
+ALLOWED_IMPORT_TEXT = ", ".join(name for name in ["decimal", "datetime", "math", "re", "typing"] if name in ALLOWED_IMPORTS)
+COBOL_NUMERIC_RULE = (
+    "PIC fields with V use an implied decimal point: parse them by integer digits and scale; "
+    "fixed-width numeric outputs are digits only, zero-padded to their field length, with no decimal point or spaces."
+)
+REPAIR_DISCIPLINE = (
+    "Repair discipline: visible failures are actionable. Do not return the previous code unchanged when pass_rate < 1.0. "
+    "If an expected numeric output is all digits but the actual output contains a decimal point, spaces, or a shifted value, "
+    "replace Decimal-string formatting with scaled integer digit formatting and re-check PIC V input parsing."
+)
+
+
+def extract_code_from_response(response: str) -> str:
+    candidates = [response.strip(), _strip_fence(response.strip())]
+    start = response.find("{")
+    end = response.rfind("}")
+    if 0 <= start < end:
+        candidates.append(response[start : end + 1])
+
+    for candidate in candidates:
+        data = _load_response_object(candidate)
+        if data is None:
+            continue
+        if isinstance(data, dict) and isinstance(data.get("code"), str):
+            return _remove_unused_disallowed_imports(data["code"])
+
+    unfenced = _strip_fence(response.strip())
+    if "def migrate" in unfenced:
+        return _remove_unused_disallowed_imports(unfenced)
+    raise ValueError("model response did not contain JSON with a code field")
+
+
+def _load_response_object(candidate: str) -> object | None:
+    try:
+        return json.loads(candidate)
+    except json.JSONDecodeError:
+        pass
+
+    try:
+        return ast.literal_eval(candidate)
+    except (SyntaxError, ValueError):
+        return None
+
+
+def run_model_rollout(
+    task: TaskInstance,
+    provider: TextProvider,
+) -> dict[str, Any]:
+    env = LegacyCobolEnvironment()
+    reset_observation = env.reset(task_id=task.task_id)
+    ticket = reset_observation.result["ticket"]
+    steps: list[dict[str, Any]] = []
+    context: dict[str, Any] = {"cobol_files": {}, "copybooks": {}, "layouts": {}, "business_rules": []}
+
+    def record(tool_name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+        result, reward, done = call_tool(env, tool_name, **arguments)
+        saved_arguments = dict(arguments)
+        if "code" in saved_arguments:
+            saved_arguments["code_chars"] = len(saved_arguments.pop("code"))
+        steps.append(
+            {
+                "tool_name": tool_name,
+                "arguments": saved_arguments,
+                "reward": reward,
+                "done": done,
+                "result": result,
+            }
+        )
+        return result
+
+    for filename in ticket["available_files"]:
+        result = record("read_cobol_file", {"filename": filename})
+        context["cobol_files"][filename] = result["content"]
+    for filename in ticket["available_copybooks"]:
+        copybook = record("read_copybook", {"filename": filename})
+        layout = record("parse_copybook_layout", {"filename": filename})
+        context["copybooks"][filename] = copybook["content"]
+        context["layouts"][filename] = layout
+    rules = record("inspect_business_rules", {})
+    context["business_rules"] = rules["rules"]
+
+    prompt = build_migration_prompt(ticket, context)
+    response = provider.generate(prompt)
+    code = extract_code_from_response(response)
+    written = record("write_python_solution", {"code": code})
+    visible = record("run_visible_tests", {"draft_id": written["draft_id"]})
+    final = record("submit_final", {"draft_id": written["draft_id"]})
+
+    return {
+        "policy": provider.name,
+        "task_id": task.task_id,
+        "family_id": task.family_id,
+        "ticket": ticket,
+        "model_turns": [
+            {
+                "provider": provider.name,
+                "prompt": prompt,
+                "response": response,
+                "code_chars": len(code),
+            }
+        ],
+        "visible": {
+            "passed": visible["passed"],
+            "total": visible["total"],
+            "pass_rate": visible["pass_rate"],
+            "failures": visible["failures"],
+        },
+        "final": {
+            "public_score": final["public_score"],
+            "accepted": final["accepted"],
+            "components": final["components"],
+            "hidden_passed": final["hidden_passed"],
+            "hidden_total": final["hidden_total"],
+            "fresh_passed": final["fresh_passed"],
+            "fresh_total": final["fresh_total"],
+        },
+        "steps": steps,
+    }
+
+
+def run_model_repair_rollout(
+    task: TaskInstance,
+    provider: TextProvider,
+    max_repairs: int = 1,
+) -> dict[str, Any]:
+    env = LegacyCobolEnvironment()
+    reset_observation = env.reset(task_id=task.task_id)
+    ticket = reset_observation.result["ticket"]
+    steps: list[dict[str, Any]] = []
+    model_turns: list[dict[str, Any]] = []
+    context: dict[str, Any] = {"cobol_files": {}, "copybooks": {}, "layouts": {}, "business_rules": []}
+
+    def record(tool_name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+        result, reward, done = call_tool(env, tool_name, **arguments)
+        saved_arguments = dict(arguments)
+        if "code" in saved_arguments:
+            saved_arguments["code_chars"] = len(saved_arguments.pop("code"))
+        steps.append(
+            {
+                "tool_name": tool_name,
+                "arguments": saved_arguments,
+                "reward": reward,
+                "done": done,
+                "result": result,
+            }
+        )
+        return result
+
+    for filename in ticket["available_files"]:
+        result = record("read_cobol_file", {"filename": filename})
+        context["cobol_files"][filename] = result["content"]
+    for filename in ticket["available_copybooks"]:
+        copybook = record("read_copybook", {"filename": filename})
+        layout = record("parse_copybook_layout", {"filename": filename})
+        context["copybooks"][filename] = copybook["content"]
+        context["layouts"][filename] = layout
+    rules = record("inspect_business_rules", {})
+    context["business_rules"] = rules["rules"]
+
+    prompt = build_migration_prompt(ticket, context)
+    response = provider.generate(prompt)
+    code = extract_code_from_response(response)
+    model_turns.append({"provider": provider.name, "prompt": prompt, "response": response, "code_chars": len(code)})
+    written = record("write_python_solution", {"code": code})
+    visible = record("run_visible_tests", {"draft_id": written["draft_id"]})
+
+    for _ in range(max_repairs):
+        if visible["pass_rate"] == 1.0:
+            break
+        diffs = [record("inspect_diff", {"case_id": failure["case_id"]}) for failure in visible["failures"][:2]]
+        repair_prompt = build_repair_prompt(ticket, context, code, visible, diffs)
+        response = provider.generate(repair_prompt)
+        code = extract_code_from_response(response)
+        model_turns.append({"provider": provider.name, "prompt": repair_prompt, "response": response, "code_chars": len(code)})
+        written = record("write_python_solution", {"code": code})
+        visible = record("run_visible_tests", {"draft_id": written["draft_id"]})
+
+    final = record("submit_final", {"draft_id": written["draft_id"]})
+    return {
+        "policy": provider.name,
+        "task_id": task.task_id,
+        "family_id": task.family_id,
+        "ticket": ticket,
+        "model_turns": model_turns,
+        "visible": {
+            "passed": visible["passed"],
+            "total": visible["total"],
+            "pass_rate": visible["pass_rate"],
+            "failures": visible["failures"],
+        },
+        "final": {
+            "public_score": final["public_score"],
+            "accepted": final["accepted"],
+            "components": final["components"],
+            "hidden_passed": final["hidden_passed"],
+            "hidden_total": final["hidden_total"],
+            "fresh_passed": final["fresh_passed"],
+            "fresh_total": final["fresh_total"],
+        },
+        "steps": steps,
+    }
+
+
+def build_migration_prompt(ticket: dict[str, Any], context: dict[str, Any]) -> str:
+    return "\n\n".join(
+        [
+            "You are migrating a legacy COBOL routine into Python.",
+            "Return only JSON in this shape: {\"code\": \"...python source...\"}.",
+            "The Python source must define migrate(input_record: str) -> str.",
+            f"Allowed imports: {ALLOWED_IMPORT_TEXT}. Do not import any other modules.",
+            COBOL_NUMERIC_RULE,
+            build_numeric_layout_reminders(ticket, context),
+            "Returned records must match output_width exactly; never append newline characters.",
+            f"Ticket:\n{json.dumps(ticket, indent=2)}",
+            f"COBOL files:\n{json.dumps(context['cobol_files'], indent=2)}",
+            f"Copybooks:\n{json.dumps(context['copybooks'], indent=2)}",
+            f"Parsed layouts:\n{json.dumps(context['layouts'], indent=2)}",
+            f"Business rules:\n{json.dumps(context['business_rules'], indent=2)}",
+        ]
+    )
+
+
+def build_repair_prompt(
+    ticket: dict[str, Any],
+    context: dict[str, Any],
+    previous_code: str,
+    visible: dict[str, Any],
+    diffs: list[dict[str, Any]],
+) -> str:
+    return "\n\n".join(
+        [
+            "Repair the Python migration after visible test feedback.",
+            "Return only JSON in this shape: {\"code\": \"...python source...\"}.",
+            "The Python source must define migrate(input_record: str) -> str.",
+            f"Allowed imports: {ALLOWED_IMPORT_TEXT}. Do not import any other modules.",
+            COBOL_NUMERIC_RULE,
+            REPAIR_DISCIPLINE,
+            build_runtime_error_repair_checklist(visible),
+            build_numeric_layout_reminders(ticket, context),
+            build_diff_repair_checklist(diffs),
+            "Returned records must match output_width exactly; never append newline characters.",
+            f"Ticket:\n{json.dumps(ticket, indent=2)}",
+            f"COBOL files:\n{json.dumps(context['cobol_files'], indent=2)}",
+            f"Copybooks:\n{json.dumps(context['copybooks'], indent=2)}",
+            f"Parsed layouts:\n{json.dumps(context['layouts'], indent=2)}",
+            f"Business rules:\n{json.dumps(context['business_rules'], indent=2)}",
+            f"Previous code:\n{previous_code}",
+            f"Visible test status:\n{json.dumps(_visible_status_for_prompt(visible), indent=2)}",
+            f"Visible failures:\n{json.dumps(visible['failures'], indent=2)}",
+            f"Inspected field_diffs:\n{json.dumps(diffs, indent=2)}",
+        ]
+    )
+
+
+def build_runtime_error_repair_checklist(visible: dict[str, Any]) -> str:
+    errors = "\n".join(str(failure.get("error") or "") for failure in visible.get("failures", []))
+    lines = ["Runtime error repair checklist:"]
+    if "slice indices must be integers" in errors:
+        lines.append(
+            "- TypeError 'slice indices must be integers': convert ITEM-COUNT with int(input_record[6:8]) before using it; "
+            "do not slice range(...) with input_record[6:8]; for OCCURS use range(count) with start = 8 + idx * 9."
+        )
+    if "'slice' object is not callable" in errors:
+        lines.append(
+            "- TypeError 'slice' object is not callable: do not call slice objects like field(input_record); "
+            "read fields with direct string slicing such as input_record[start:end]. Avoid generic layout parsers unless every slice is applied with data[slice_obj]."
+        )
+    if "invalid format string" in errors or "unsupported format string passed to decimal.Decimal" in errors:
+        lines.append(
+            "- Decimal formatting error: do not format Decimal values with :d or include decimal points in fixed-width PIC V output. "
+            "Convert monetary totals to integer cents first, for example total_cents = int((total * Decimal(\"100\")).quantize(Decimal(\"1\"))), "
+            "then emit format(total_cents, \"09d\")."
+        )
+    if "object has no attribute 'to_string'" in errors:
+        lines.append(
+            "- Decimal to_string error: do not use Decimal.to_string or integer-like .to_string methods; "
+            "format fixed-width fields with format(int(cents), \"09d\") and format(count, \"02d\")."
+        )
+    if "NameError: name 'Decimal' is not defined" in errors:
+        lines.append("- Decimal NameError: include from decimal import Decimal at module scope if Decimal is used.")
+    if "cannot access local variable 'record'" in errors:
+        lines.append(
+            "- UnboundLocalError for record: do not reference record while constructing it. "
+            "Assign count = int(input_record[6:8]) before building lists, then iterate with for idx in range(count) and start = 8 + idx * 9."
+        )
+    if "KeyError" in errors and ("TAX-CODE" in errors or re.search(r"KeyError: ['\"][A-Z0-9]['\"]", errors)):
+        lines.append("- Tax lookup KeyError: use tax_rates.get(tax_code, Decimal(\"0.0000\")) for unknown/non-taxable codes.")
+    if len(lines) == 1:
+        lines.append("- No recognized runtime pattern; fix the exception shown in Visible test status/failures before changing formatting.")
+    return "\n".join(lines)
+
+
+def build_numeric_layout_reminders(ticket: dict[str, Any], context: dict[str, Any]) -> str:
+    lines = ["Numeric layout reminders:"]
+    for filename, layout in sorted(context["layouts"].items()):
+        for field_path, field in _walk_layout_fields(layout.get("fields", [])):
+            scale = _field_scale(field)
+            if scale <= 0:
+                continue
+            divisor = 10**scale
+            field_name = field_path[-1]
+            lines.append(
+                f"- {field_name}: PIC {field['pic']} has implied scale {scale}; "
+                f"parse raw digits with Decimal(int(slice)) / Decimal(\"{divisor}\")."
+            )
+
+    for field in ticket.get("output_layout", []):
+        pic = str(field.get("pic", ""))
+        if not _is_numeric_pic(pic):
+            continue
+        scale = _pic_implied_scale(pic)
+        if scale > 0:
+            lines.append(
+                f"- {field['name']}: PIC {pic} is {field['length']} fixed-width digits with implied scale {scale}; "
+                "format scaled integer digits, not a decimal string."
+            )
+        else:
+            lines.append(
+                f"- {field['name']}: PIC {pic} is {field['length']} fixed-width digits; "
+                "format exactly that many digits."
+            )
+    return "\n".join(lines)
+
+
+def build_diff_repair_checklist(diffs: list[dict[str, Any]]) -> str:
+    lines = ["Field diff repair checklist:"]
+    for diff_result in diffs:
+        for field_diff in diff_result.get("field_diffs", []):
+            field_name = field_diff.get("field")
+            expected = field_diff.get("expected")
+            actual = field_diff.get("actual")
+            if isinstance(expected, str) and expected.isdigit() and isinstance(actual, str) and not actual.isdigit():
+                lines.append(
+                    f"- {field_name}: expected all digits {expected!r}, but actual was {actual!r}; "
+                    "emit scaled integer digits with no decimal point or spaces."
+                )
+            elif field_name:
+                lines.append(f"- {field_name}: expected {expected!r}, actual {actual!r}; update the code path that produces this field.")
+    if len(lines) == 1:
+        lines.append("- No structured field mismatch was available; repair the runtime or interface error first.")
+    return "\n".join(lines)
+
+
+def _visible_status_for_prompt(visible: dict[str, Any]) -> dict[str, Any]:
+    keys = [
+        "passed",
+        "total",
+        "pass_rate",
+        "syntax_ok",
+        "safety_ok",
+        "interface_ok",
+        "timed_out",
+        "error",
+    ]
+    return {key: visible.get(key) for key in keys if key in visible}
+
+
+def _walk_layout_fields(fields: list[dict[str, Any]], prefix: tuple[str, ...] = ()) -> list[tuple[tuple[str, ...], dict[str, Any]]]:
+    walked = []
+    for field in fields:
+        path = (*prefix, field["name"])
+        walked.append((path, field))
+        children = field.get("children")
+        if children:
+            walked.extend(_walk_layout_fields(children, path))
+    return walked
+
+
+def _field_scale(field: dict[str, Any]) -> int:
+    if isinstance(field.get("scale"), int):
+        return field["scale"]
+    return _pic_implied_scale(str(field.get("pic", "")))
+
+
+def _pic_implied_scale(pic: str) -> int:
+    upper = pic.upper()
+    if "V" not in upper:
+        return 0
+    after_decimal = upper.split("V", 1)[1]
+    return sum(int(match.group(1) or 1) for match in re.finditer(r"9(?:\((\d+)\))?", after_decimal))
+
+
+def _is_numeric_pic(pic: str) -> bool:
+    return pic.strip().upper().lstrip("S").startswith("9")
+
+
+def _remove_unused_disallowed_imports(code: str) -> str:
+    try:
+        tree = ast.parse(code)
+    except SyntaxError:
+        return code
+
+    import_lines: set[int] = set()
+    imported_names: dict[str, str] = {}
+    for node in tree.body:
+        if isinstance(node, ast.Import):
+            for alias in node.names:
+                root = alias.name.split(".", 1)[0]
+                if root in FORBIDDEN_IMPORTS or root not in ALLOWED_IMPORTS:
+                    imported_names[alias.asname or root] = root
+            if node.lineno:
+                import_lines.add(node.lineno)
+        elif isinstance(node, ast.ImportFrom):
+            root = (node.module or "").split(".", 1)[0]
+            if root in FORBIDDEN_IMPORTS or root not in ALLOWED_IMPORTS:
+                for alias in node.names:
+                    imported_names[alias.asname or alias.name] = root
+                if node.lineno:
+                    import_lines.add(node.lineno)
+
+    if not imported_names:
+        return code
+
+    used_names = {
+        node.id
+        for node in ast.walk(tree)
+        if isinstance(node, ast.Name) and node.id in imported_names and node.lineno not in import_lines
+    }
+    removable_lines = {
+        line
+        for line in import_lines
+        if not any(
+            isinstance(node, (ast.Import, ast.ImportFrom))
+            and node.lineno == line
+            and any((alias.asname or alias.name.split(".", 1)[0]) in used_names for alias in node.names)
+            for node in tree.body
+        )
+    }
+    if not removable_lines:
+        return code
+
+    lines = code.splitlines(keepends=True)
+    return "".join(line for index, line in enumerate(lines, start=1) if index not in removable_lines)
+
+
+def _strip_fence(text: str) -> str:
+    if not text.startswith("```"):
+        return text
+    lines = text.splitlines()
+    if lines:
+        lines = lines[1:]
+    if lines and lines[-1].startswith("```"):
+        lines = lines[:-1]
+    return "\n".join(lines)

eval/oracle_solutions.py

@@ -0,0 +1,161 @@
+"""Deterministic reference candidate solutions for sanity and demo traces."""
+
+from __future__ import annotations
+
+from legacy_cobol_env.server.task_bank import TaskInstance
+
+
+PAYROLL_SOLUTION = r'''
+from decimal import Decimal, ROUND_HALF_UP
+
+
+def migrate(input_record: str) -> str:
+    emp_id = input_record[0:6]
+    emp_name = input_record[6:18]
+    gross = Decimal(int(input_record[18:27])) / Decimal("100")
+    tax_rate = Decimal(int(input_record[27:31])) / Decimal("1000")
+    raw_deductions = input_record[31:39]
+    sign = -1 if raw_deductions[0] == "-" else 1
+    deductions = Decimal(sign * int(raw_deductions[1:])) / Decimal("100")
+    tax = (gross * tax_rate).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
+    net = gross - tax - deductions
+    if input_record[39:40] == "Y":
+        net += Decimal("50.00")
+    if net < 0:
+        net = Decimal("0.00")
+    if net >= Decimal("5000.00"):
+        category = "H"
+    elif net >= Decimal("2500.00"):
+        category = "M"
+    else:
+        category = "L"
+    cents = int((net * 100).quantize(Decimal("1"), rounding=ROUND_HALF_UP))
+    return f"{emp_id}{emp_name[:12].ljust(12)}{cents:09d}{category}"
+'''
+
+
+CUSTOMER_SOLUTION = r'''
+def migrate(input_record: str) -> str:
+    cust_id = input_record[0:5]
+    first = input_record[5:15].rstrip()
+    last = input_record[15:27].rstrip()
+    zip_code = input_record[27:32]
+    status = {"A": "O", "S": "S"}.get(input_record[32:33], "C")
+    balance = int(input_record[33:40])
+    full_name = f"{last}, {first}"[:22].ljust(22)
+    return f"{cust_id}{full_name}{zip_code}{status}{balance:08d}"
+'''
+
+
+CLAIMS_SOLUTION = r'''
+def migrate(input_record: str) -> str:
+    claim_id = input_record[0:6]
+    age = int(input_record[6:9])
+    plan = input_record[9:10]
+    days = int(input_record[10:13])
+    preauth = input_record[13:14]
+    amount_cents = int(input_record[14:21])
+    if age < 18:
+        decision, reason = "D", "A1"
+    elif plan == "B" and amount_cents > 150000:
+        decision, reason = "R", "B2"
+    elif preauth == "N" and amount_cents > 100000:
+        decision, reason = "D", "P1"
+    elif days > 30:
+        decision, reason = "R", "L1"
+    else:
+        decision, reason = "A", "OK"
+    return f"{claim_id}{decision}{reason}"
+'''
+
+
+ACCOUNT_SOLUTION = r'''
+def migrate(input_record: str) -> str:
+    account_id = input_record[0:6]
+    status = input_record[6:7]
+    raw_balance = input_record[7:16]
+    sign = -1 if raw_balance[0] == "-" else 1
+    balance_cents = sign * int(raw_balance[1:])
+    days = int(input_record[16:19])
+    if status == "C":
+        category, action = "CL", "N"
+    elif status == "F":
+        category, action = "FR", "H"
+    elif days >= 90:
+        category, action = "DL", "C"
+    elif balance_cents < 0:
+        category, action = "OD", "R"
+    else:
+        category, action = "OK", "N"
+    return f"{account_id}{category}{action}"
+'''
+
+
+INVOICE_SOLUTION = r'''
+from decimal import Decimal, ROUND_HALF_UP
+
+
+def migrate(input_record: str) -> str:
+    invoice_id = input_record[0:6]
+    count = min(int(input_record[6:8]), 4)
+    tax_rates = {
+        "S": Decimal("0.0725"),
+        "R": Decimal("0.0250"),
+        "L": Decimal("0.1000"),
+    }
+    total = Decimal("0.00")
+    for idx in range(count):
+        start = 8 + idx * 9
+        qty = int(input_record[start:start + 2])
+        price = Decimal(int(input_record[start + 2:start + 8])) / Decimal("100")
+        tax_code = input_record[start + 8:start + 9]
+        line = (Decimal(qty) * price).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
+        tax = (line * tax_rates.get(tax_code, Decimal("0.0000"))).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
+        line += tax
+        total += line
+    cents = int((total * 100).quantize(Decimal("1"), rounding=ROUND_HALF_UP))
+    flag = "H" if total >= Decimal("1000.00") else "L"
+    return f"{invoice_id}{cents:09d}{count:02d}{flag}"
+'''
+
+
+DATE_SOLUTION = r'''
+def is_leap(year: int) -> bool:
+    return year % 4 == 0 and (year % 100 != 0 or year % 400 == 0)
+
+
+def valid_date(year: int, month: int, day: int) -> bool:
+    month_lengths = [31, 29 if is_leap(year) else 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
+    return 1 <= month <= 12 and 1 <= day <= month_lengths[month - 1]
+
+
+def migrate(input_record: str) -> str:
+    policy_id = input_record[0:6]
+    raw = input_record[6:12]
+    window = int(input_record[12:14])
+    amount = input_record[14:21]
+    yy = int(raw[0:2])
+    mm = int(raw[2:4])
+    dd = int(raw[4:6])
+    year = 1900 + yy if yy >= window else 2000 + yy
+    normalized = f"{year:04d}{mm:02d}{dd:02d}" if valid_date(year, mm, dd) else "00000000"
+    valid = "Y" if normalized != "00000000" else "N"
+    return f"{policy_id}{normalized}{valid}{amount}"
+'''
+
+
+SOLUTIONS_BY_FAMILY = {
+    "decimal_copybook_payroll": PAYROLL_SOLUTION,
+    "fixed_width_customer": CUSTOMER_SOLUTION,
+    "claims_eligibility_branching": CLAIMS_SOLUTION,
+    "account_status_level88": ACCOUNT_SOLUTION,
+    "invoice_occurs_totals": INVOICE_SOLUTION,
+    "date_normalization": DATE_SOLUTION,
+}
+
+
+def solution_for_task(task: TaskInstance) -> str:
+    try:
+        return SOLUTIONS_BY_FAMILY[task.family_id].strip() + "\n"
+    except KeyError as exc:
+        raise ValueError(f"no oracle solution for family: {task.family_id}") from exc

eval/providers.py

@@ -0,0 +1,332 @@
+"""Provider adapters for model-driven evaluation rollouts."""
+
+from __future__ import annotations
+
+import json
+import urllib.error
+import urllib.request
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Mapping, Protocol
+
+
+class TextProvider(Protocol):
+    name: str
+
+    def generate(self, prompt: str) -> str:
+        """Return provider text for one migration prompt."""
+
+
+@dataclass(frozen=True)
+class StaticResponseProvider:
+    name: str
+    response: str
+
+    def generate(self, prompt: str) -> str:
+        return self.response
+
+
+@dataclass
+class SequenceResponseProvider:
+    name: str
+    responses: list[str]
+    index: int = 0
+
+    def generate(self, prompt: str) -> str:
+        if self.index >= len(self.responses):
+            return self.responses[-1]
+        response = self.responses[self.index]
+        self.index += 1
+        return response
+
+
+@dataclass(frozen=True)
+class AzureOpenAIProvider:
+    endpoint: str
+    api_key: str
+    deployment: str
+    api_version: str = "2024-02-15-preview"
+    name: str = "azure-openai"
+    timeout_s: float = 60.0
+
+    def generate(self, prompt: str) -> str:
+        url = (
+            f"{self.endpoint.rstrip('/')}/openai/deployments/{self.deployment}"
+            f"/chat/completions?api-version={self.api_version}"
+        )
+        payload = {
+            "messages": [
+                {
+                    "role": "system",
+                    "content": "Return only JSON with a single code field containing a Python migrate function.",
+                },
+                {"role": "user", "content": prompt},
+            ],
+            "temperature": 0,
+        }
+        data = _post_json(
+            url,
+            payload,
+            headers={"api-key": self.api_key, "Content-Type": "application/json"},
+            timeout_s=self.timeout_s,
+        )
+        return data["choices"][0]["message"]["content"]
+
+
+@dataclass(frozen=True)
+class HuggingFaceEndpointProvider:
+    endpoint: str
+    token: str
+    name: str = "hf-endpoint"
+    timeout_s: float = 60.0
+
+    def generate(self, prompt: str) -> str:
+        data = _post_json(
+            self.endpoint,
+            {"inputs": prompt, "parameters": {"temperature": 0.0, "max_new_tokens": 1800}},
+            headers={"Authorization": f"Bearer {self.token}", "Content-Type": "application/json"},
+            timeout_s=self.timeout_s,
+        )
+        if isinstance(data, list) and data and "generated_text" in data[0]:
+            return data[0]["generated_text"]
+        if isinstance(data, dict) and "generated_text" in data:
+            return data["generated_text"]
+        if isinstance(data, dict) and "choices" in data:
+            return data["choices"][0]["message"]["content"]
+        raise ValueError("HF endpoint response did not contain generated text")
+
+
+@dataclass(frozen=True)
+class HuggingFaceChatProvider:
+    model: str
+    token: str
+    provider: str | None = None
+    name: str = "hf-chat"
+    timeout_s: float = 120.0
+    max_tokens: int = 2200
+    temperature: float = 0.1
+    top_p: float = 0.9
+
+    def generate(self, prompt: str) -> str:
+        try:
+            from huggingface_hub import InferenceClient
+        except ImportError as exc:
+            raise RuntimeError("install huggingface_hub to use hf-chat provider") from exc
+
+        client = InferenceClient(
+            model=self.model,
+            provider=self.provider or "auto",
+            token=self.token,
+            timeout=self.timeout_s,
+        )
+        response = client.chat_completion(
+            messages=[
+                {
+                    "role": "system",
+                    "content": "Return only JSON with a single code field containing a Python migrate function.",
+                },
+                {"role": "user", "content": prompt},
+            ],
+            max_tokens=self.max_tokens,
+            temperature=self.temperature,
+            top_p=self.top_p,
+        )
+        return _chat_completion_content(response)
+
+
+@dataclass
+class LocalTransformersProvider:
+    model_path: str
+    base_model_path: str | None = None
+    name: str = "local-transformers"
+    max_new_tokens: int = 1800
+    load_in_4bit: bool = True
+    do_sample: bool = False
+    temperature: float = 0.2
+    top_p: float = 0.95
+    _model: object | None = None
+    _tokenizer: object | None = None
+
+    def generate(self, prompt: str) -> str:
+        tokenizer, model = self._load()
+        prompt_text = self._format_prompt(tokenizer, prompt)
+        inputs = tokenizer(prompt_text, return_tensors="pt")
+        inputs = inputs.to(model.device) if hasattr(inputs, "to") else inputs
+        generation_kwargs = {
+            "max_new_tokens": self.max_new_tokens,
+            "pad_token_id": tokenizer.eos_token_id,
+            "do_sample": self.do_sample,
+        }
+        if self.do_sample:
+            generation_kwargs["temperature"] = self.temperature
+            generation_kwargs["top_p"] = self.top_p
+        output_ids = model.generate(
+            **inputs,
+            **generation_kwargs,
+        )
+        prompt_length = inputs["input_ids"].shape[-1]
+        generated = output_ids[0][prompt_length:]
+        return tokenizer.decode(generated, skip_special_tokens=True)
+
+    def _format_prompt(self, tokenizer: object, prompt: str) -> str:
+        messages = [
+            {
+                "role": "system",
+                "content": "Return only JSON with a single code field containing a Python migrate function.",
+            },
+            {"role": "user", "content": prompt},
+        ]
+        if hasattr(tokenizer, "apply_chat_template"):
+            return tokenizer.apply_chat_template(
+                messages,
+                tokenize=False,
+                add_generation_prompt=True,
+            )
+        return prompt
+
+    def _load(self) -> tuple[object, object]:
+        if self._model is not None and self._tokenizer is not None:
+            return self._tokenizer, self._model
+        try:
+            from transformers import AutoModelForCausalLM, AutoTokenizer
+        except ImportError as exc:
+            raise RuntimeError("install training/requirements-gpu.txt to use local-transformers") from exc
+        quantization_config = self._quantization_config()
+        adapter_base = self._adapter_base_model_path()
+        if adapter_base:
+            try:
+                from peft import PeftModel
+            except ImportError as exc:
+                raise RuntimeError("install peft from training/requirements-gpu.txt to load LoRA adapters") from exc
+            tokenizer = AutoTokenizer.from_pretrained(self.model_path, trust_remote_code=True)
+            model = AutoModelForCausalLM.from_pretrained(
+                adapter_base,
+                device_map="auto",
+                quantization_config=quantization_config,
+                trust_remote_code=True,
+            )
+            model = PeftModel.from_pretrained(model, self.model_path)
+        else:
+            tokenizer = AutoTokenizer.from_pretrained(self.model_path, trust_remote_code=True)
+            model = AutoModelForCausalLM.from_pretrained(
+                self.model_path,
+                device_map="auto",
+                quantization_config=quantization_config,
+                trust_remote_code=True,
+            )
+        self._tokenizer = tokenizer
+        self._model = model
+        return tokenizer, model
+
+    def _adapter_base_model_path(self) -> str | None:
+        if self.base_model_path:
+            return self.base_model_path
+        config_path = Path(self.model_path) / "adapter_config.json"
+        if not config_path.exists():
+            return None
+        data = json.loads(config_path.read_text(encoding="utf-8"))
+        return data.get("base_model_name_or_path")
+
+    def _quantization_config(self) -> object | None:
+        if not self.load_in_4bit:
+            return None
+        try:
+            from transformers import BitsAndBytesConfig
+        except ImportError as exc:
+            raise RuntimeError("install bitsandbytes and transformers from training/requirements-gpu.txt") from exc
+        return BitsAndBytesConfig(
+            load_in_4bit=True,
+            bnb_4bit_quant_type="nf4",
+            bnb_4bit_compute_dtype="bfloat16",
+        )
+
+
+def create_provider(kind: str, env: Mapping[str, str]) -> TextProvider:
+    if kind == "azure-openai":
+        required = ["AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_API_KEY", "AZURE_OPENAI_DEPLOYMENT"]
+        missing = [key for key in required if not env.get(key)]
+        if missing:
+            raise ValueError(f"missing Azure OpenAI configuration: {', '.join(missing)}")
+        return AzureOpenAIProvider(
+            endpoint=env["AZURE_OPENAI_ENDPOINT"],
+            api_key=env["AZURE_OPENAI_API_KEY"],
+            deployment=env["AZURE_OPENAI_DEPLOYMENT"],
+            api_version=env.get("AZURE_OPENAI_API_VERSION", "2024-02-15-preview"),
+        )
+
+    if kind == "hf-endpoint":
+        required = ["HF_INFERENCE_ENDPOINT", "HF_TOKEN"]
+        missing = [key for key in required if not env.get(key)]
+        if missing:
+            raise ValueError(f"missing Hugging Face endpoint configuration: {', '.join(missing)}")
+        return HuggingFaceEndpointProvider(
+            endpoint=env["HF_INFERENCE_ENDPOINT"],
+            token=env["HF_TOKEN"],
+        )
+
+    if kind == "hf-chat":
+        required = ["HF_MODEL", "HF_TOKEN"]
+        missing = [key for key in required if not env.get(key)]
+        if missing:
+            raise ValueError(f"missing Hugging Face chat configuration: {', '.join(missing)}")
+        return HuggingFaceChatProvider(
+            model=env["HF_MODEL"],
+            token=env["HF_TOKEN"],
+            provider=env.get("HF_PROVIDER"),
+            timeout_s=float(env.get("HF_TIMEOUT_S", "120")),
+            max_tokens=int(env.get("HF_MAX_TOKENS", "2200")),
+            temperature=float(env.get("HF_TEMPERATURE", "0.1")),
+            top_p=float(env.get("HF_TOP_P", "0.9")),
+        )
+
+    if kind == "static":
+        return StaticResponseProvider("static", env.get("STATIC_RESPONSE", '{"code": "def migrate(input_record: str) -> str:\\n    return input_record\\n"}'))
+
+    if kind == "local-transformers":
+        if not env.get("LOCAL_MODEL_PATH"):
+            raise ValueError("missing local Transformers configuration: LOCAL_MODEL_PATH")
+        return LocalTransformersProvider(
+            model_path=env["LOCAL_MODEL_PATH"],
+            base_model_path=env.get("LOCAL_BASE_MODEL_PATH"),
+            max_new_tokens=int(env.get("LOCAL_MAX_NEW_TOKENS", "1800")),
+            load_in_4bit=env.get("LOCAL_LOAD_IN_4BIT", "1").lower() not in {"0", "false", "no"},
+            do_sample=env.get("LOCAL_DO_SAMPLE", "0").lower() in {"1", "true", "yes"},
+            temperature=float(env.get("LOCAL_TEMPERATURE", "0.2")),
+            top_p=float(env.get("LOCAL_TOP_P", "0.95")),
+        )
+
+    raise ValueError(f"unknown provider: {kind}")
+
+
+def _chat_completion_content(response: object) -> str:
+    choices = getattr(response, "choices", None)
+    if choices is None and isinstance(response, dict):
+        choices = response.get("choices")
+    if not choices:
+        raise ValueError("HF chat response did not contain choices")
+
+    first = choices[0]
+    message = getattr(first, "message", None)
+    if message is None and isinstance(first, dict):
+        message = first.get("message")
+    content = getattr(message, "content", None)
+    if content is None and isinstance(message, dict):
+        content = message.get("content")
+    if not isinstance(content, str):
+        raise ValueError("HF chat response did not contain message content")
+    return content
+
+
+def _post_json(url: str, payload: object, headers: dict[str, str], timeout_s: float) -> object:
+    request = urllib.request.Request(
+        url,
+        data=json.dumps(payload).encode("utf-8"),
+        headers=headers,
+        method="POST",
+    )
+    try:
+        with urllib.request.urlopen(request, timeout=timeout_s) as response:
+            return json.loads(response.read().decode("utf-8"))
+    except urllib.error.HTTPError as exc:
+        body = exc.read().decode("utf-8", errors="replace")
+        raise RuntimeError(f"provider HTTP {exc.code}: {body}") from exc

eval/run_baselines.py

@@ -0,0 +1,127 @@
+"""Run deterministic non-model baselines across all task families.
+
+This is not the final training evidence. It gives the project a repeatable
+evaluation harness so baseline, SFT, and RL runs can share one result shape.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from statistics import mean
+
+from openenv.core.env_server.mcp_types import CallToolAction
+
+from legacy_cobol_env.server.legacy_cobol_env_environment import LegacyCobolEnvironment
+from legacy_cobol_env.server.task_bank import all_tasks
+
+
+ENV_ROOT = Path(__file__).resolve().parents[1]
+OUTPUT_DIR = ENV_ROOT / "outputs" / "evals"
+PLOT_DIR = ENV_ROOT / "plots"
+
+
+def call(env: LegacyCobolEnvironment, tool_name: str, **arguments):
+    obs = env.step(CallToolAction(tool_name=tool_name, arguments=arguments))
+    return obs.result.data
+
+
+def identity_solution() -> str:
+    return """def migrate(input_record: str) -> str:
+    return input_record
+"""
+
+
+def blank_width_solution(width: int) -> str:
+    return f"""def migrate(input_record: str) -> str:
+    return " " * {width}
+"""
+
+
+def run_policy(policy_name: str, task, code: str) -> dict:
+    env = LegacyCobolEnvironment()
+    env.reset(task_id=task.task_id)
+    written = call(env, "write_python_solution", code=code)
+    visible = call(env, "run_visible_tests", draft_id=written["draft_id"])
+    final = call(env, "submit_final", draft_id=written["draft_id"])
+    return {
+        "policy": policy_name,
+        "task_id": task.task_id,
+        "family_id": task.family_id,
+        "visible_pass_rate": visible["pass_rate"],
+        "public_score": final["public_score"],
+        "components": final["components"],
+        "accepted": final["accepted"],
+    }
+
+
+def write_svg(results: list[dict], path: Path) -> None:
+    families = []
+    scores_by_policy: dict[str, list[float]] = {}
+    for row in results:
+        if row["task_id"] not in families:
+            families.append(row["task_id"])
+        scores_by_policy.setdefault(row["policy"], []).append(row["public_score"])
+
+    width = 960
+    height = 360
+    margin = 48
+    bar_w = 26
+    gap = 22
+    colors = {"identity": "#3b82f6", "blank_width": "#10b981"}
+    parts = [
+        f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" viewBox="0 0 {width} {height}">',
+        '<rect width="100%" height="100%" fill="#ffffff"/>',
+        '<text x="48" y="30" font-family="Arial" font-size="18" font-weight="700">Baseline public score by task family</text>',
+        f'<line x1="{margin}" y1="{height - margin}" x2="{width - margin}" y2="{height - margin}" stroke="#222"/>',
+        f'<line x1="{margin}" y1="{height - margin}" x2="{margin}" y2="{margin}" stroke="#222"/>',
+    ]
+    x = margin + 20
+    by_task = {(row["task_id"], row["policy"]): row["public_score"] for row in results}
+    for task_id in families:
+        for offset, policy in enumerate(scores_by_policy):
+            score = by_task[(task_id, policy)]
+            bar_h = score * (height - 2 * margin)
+            bx = x + offset * (bar_w + 4)
+            by = height - margin - bar_h
+            parts.append(f'<rect x="{bx}" y="{by:.1f}" width="{bar_w}" height="{bar_h:.1f}" fill="{colors.get(policy, "#64748b")}"/>')
+            parts.append(f'<text x="{bx}" y="{by - 4:.1f}" font-family="Arial" font-size="10">{score:.2f}</text>')
+        label = task_id.replace("_001", "").replace("_", " ")
+        parts.append(f'<text x="{x - 6}" y="{height - 16}" font-family="Arial" font-size="10" transform="rotate(25 {x - 6},{height - 16})">{label}</text>')
+        x += len(scores_by_policy) * (bar_w + 4) + gap
+
+    legend_x = width - 210
+    for idx, policy in enumerate(scores_by_policy):
+        y = 52 + idx * 20
+        parts.append(f'<rect x="{legend_x}" y="{y - 10}" width="12" height="12" fill="{colors.get(policy, "#64748b")}"/>')
+        parts.append(f'<text x="{legend_x + 18}" y="{y}" font-family="Arial" font-size="12">{policy}</text>')
+    parts.append("</svg>")
+    path.write_text("\n".join(parts), encoding="utf-8")
+
+
+def main() -> None:
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    PLOT_DIR.mkdir(parents=True, exist_ok=True)
+
+    results = []
+    for task in all_tasks():
+        results.append(run_policy("identity", task, identity_solution()))
+        results.append(run_policy("blank_width", task, blank_width_solution(task.metadata["output_width"])))
+
+    summary = {
+        "policies": sorted({row["policy"] for row in results}),
+        "task_count": len(all_tasks()),
+        "mean_public_score": {
+            policy: mean(row["public_score"] for row in results if row["policy"] == policy)
+            for policy in sorted({row["policy"] for row in results})
+        },
+        "results": results,
+    }
+
+    (OUTPUT_DIR / "baseline_results.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
+    write_svg(results, PLOT_DIR / "baseline_scores.svg")
+    print(json.dumps(summary["mean_public_score"], indent=2))
+
+
+if __name__ == "__main__":
+    main()

eval/run_cobol_oracle_checks.py

@@ -0,0 +1,56 @@
+"""Run compiler-backed COBOL oracle checks for the invoice task."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from legacy_cobol_env.eval.cobol_oracle import (
+    DEFAULT_IMAGE_TAG,
+    compile_invoice_task_sources,
+    compare_invoice_oracle,
+    write_comparison_report,
+)
+
+
+ENV_ROOT = Path(__file__).resolve().parents[1]
+OUTPUT_DIR = ENV_ROOT / "outputs" / "evals"
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--build", action="store_true", help="Build the GnuCOBOL oracle image before running")
+    parser.add_argument("--image-tag", default=DEFAULT_IMAGE_TAG)
+    parser.add_argument("--no-fresh", action="store_true", help="Check only visible and hidden invoice cases")
+    parser.add_argument("--output", default=str(OUTPUT_DIR / "cobol_invoice_oracle_check.json"))
+    args = parser.parse_args()
+
+    result = compare_invoice_oracle(
+        image_tag=args.image_tag,
+        build=args.build,
+        include_fresh=not args.no_fresh,
+    )
+    result["source_compile"] = compile_invoice_task_sources(
+        image_tag=args.image_tag,
+        build=False,
+    )
+    result["ok"] = result["ok"] and result["source_compile"]["ok"]
+    write_comparison_report(result, Path(args.output))
+    print(
+        json.dumps(
+            {
+                "ok": result["ok"],
+                "case_count": result["case_count"],
+                "passed_count": result["passed_count"],
+                "source_compile_ok": result["source_compile"]["ok"],
+            },
+            indent=2,
+        )
+    )
+    if not result["ok"]:
+        raise SystemExit(1)
+
+
+if __name__ == "__main__":
+    main()

eval/run_evidence_report.py

@@ -0,0 +1,106 @@
+"""Generate score summary and plot artifacts for the submission."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from legacy_cobol_env.eval.evidence_report import build_score_summary, load_json, write_score_plot
+from legacy_cobol_env.server.task_bank import all_tasks
+
+
+ENV_ROOT = Path(__file__).resolve().parents[1]
+OUTPUT_DIR = ENV_ROOT / "outputs" / "evals"
+TRAINING_DIR = ENV_ROOT / "outputs" / "training"
+HISTORICAL_OUTPUT_DIR = OUTPUT_DIR / "historical"
+PLOT_DIR = ENV_ROOT / "plots"
+
+
+def main() -> None:
+    baseline = load_json(OUTPUT_DIR / "baseline_results.json")
+    evidence_notes = []
+    oracle_model = _load_current_rollout(OUTPUT_DIR / "oracle_model_rollouts.json", evidence_notes)
+    base_qwen3 = _load_current_rollout(OUTPUT_DIR / "base_qwen3_14b_all_tasks.json", evidence_notes)
+    zeroshot = _load_current_rollout(OUTPUT_DIR / "azure_gpt54mini_zeroshot_rollouts.json", evidence_notes)
+    repair = _load_current_rollout(OUTPUT_DIR / "azure_gpt54mini_repair1_rollouts.json", evidence_notes)
+    trained_summary = _load_training_summary(TRAINING_DIR / "sft_run_metadata.json", evidence_notes)
+    _record_historical_artifacts(evidence_notes)
+    summary = build_score_summary(
+        baseline,
+        zeroshot=zeroshot,
+        repair=repair,
+        oracle_model=oracle_model,
+        base_model=base_qwen3,
+        trained_summary=trained_summary,
+        evidence_notes=evidence_notes,
+    )
+
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    PLOT_DIR.mkdir(parents=True, exist_ok=True)
+    (OUTPUT_DIR / "score_summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
+    write_score_plot(summary, PLOT_DIR / "model_scores.svg")
+    print(json.dumps(summary["policies"], indent=2))
+
+
+def _load_current_rollout(path: Path, evidence_notes: list[str]) -> dict | None:
+    if not path.exists():
+        evidence_notes.append(f"missing rollout artifact: {_display_path(path)}")
+        return None
+    artifact = load_json(path)
+    if _matches_current_task_artifacts(artifact):
+        return artifact
+    evidence_notes.append(f"stale rollout artifact skipped after task hardening: {_display_path(path)}")
+    return None
+
+
+def _load_training_summary(path: Path, evidence_notes: list[str]) -> dict | None:
+    if not path.exists():
+        evidence_notes.append(f"missing training metadata: {_display_path(path)}")
+        return None
+    metadata = load_json(path)
+    if metadata.get("status") != "completed":
+        evidence_notes.append(f"non-completed training metadata skipped: {_display_path(path)}")
+        return None
+    return metadata
+
+
+def _record_historical_artifacts(evidence_notes: list[str]) -> None:
+    if not HISTORICAL_OUTPUT_DIR.exists():
+        return
+    for path in sorted(HISTORICAL_OUTPUT_DIR.glob("azure_gpt54mini_*.json")):
+        evidence_notes.append(f"historical artifact retained outside current score summary: {_display_path(path)}")
+
+
+def _display_path(path: Path) -> str:
+    try:
+        return path.relative_to(ENV_ROOT).as_posix()
+    except ValueError:
+        return path.name
+
+
+def _matches_current_task_artifacts(artifact: dict) -> bool:
+    current = {
+        task.task_id: {
+            "files": sorted(task.cobol_files),
+            "copybooks": sorted(task.copybooks),
+        }
+        for task in all_tasks()
+    }
+    trajectories = artifact.get("trajectories", [])
+    if len(trajectories) != len(current):
+        return False
+    for trajectory in trajectories:
+        task_id = trajectory.get("task_id")
+        ticket = trajectory.get("ticket") or {}
+        expected = current.get(task_id)
+        if expected is None:
+            return False
+        if sorted(ticket.get("available_files", [])) != expected["files"]:
+            return False
+        if sorted(ticket.get("available_copybooks", [])) != expected["copybooks"]:
+            return False
+    return True
+
+
+if __name__ == "__main__":
+    main()

eval/run_model_rollouts.py

@@ -0,0 +1,68 @@
+"""Run provider-backed model rollouts and write evaluation artifacts."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+from pathlib import Path
+from statistics import mean
+
+from legacy_cobol_env.eval.model_rollout import run_model_repair_rollout, run_model_rollout
+from legacy_cobol_env.eval.oracle_solutions import solution_for_task
+from legacy_cobol_env.eval.providers import StaticResponseProvider, create_provider
+from legacy_cobol_env.server.task_bank import all_tasks, load_task
+
+
+ENV_ROOT = Path(__file__).resolve().parents[1]
+OUTPUT_DIR = ENV_ROOT / "outputs" / "evals"
+
+
+def run_rollouts(provider_name: str, task_id: str | None = None, max_repairs: int = 0) -> dict:
+    tasks = [load_task(task_id=task_id)] if task_id else all_tasks()
+    trajectories = []
+    for task in tasks:
+        provider = (
+            StaticResponseProvider("oracle-model", json.dumps({"code": solution_for_task(task)}))
+            if provider_name == "oracle-model"
+            else create_provider(provider_name, os.environ)
+        )
+        trajectory = (
+            run_model_repair_rollout(task=task, provider=provider, max_repairs=max_repairs)
+            if max_repairs > 0
+            else run_model_rollout(task=task, provider=provider)
+        )
+        trajectories.append(trajectory)
+
+    return {
+        "provider": provider_name,
+        "max_repairs": max_repairs,
+        "task_count": len(trajectories),
+        "mean_public_score": mean(item["final"]["public_score"] for item in trajectories),
+        "accepted_count": sum(1 for item in trajectories if item["final"]["accepted"]),
+        "trajectories": trajectories,
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--provider",
+        default="oracle-model",
+        choices=["oracle-model", "static", "azure-openai", "hf-endpoint", "hf-chat", "local-transformers"],
+    )
+    parser.add_argument("--task-id")
+    parser.add_argument("--max-repairs", type=int, default=0)
+    parser.add_argument("--output", default=str(OUTPUT_DIR / "model_rollouts.json"))
+    args = parser.parse_args()
+
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    summary = run_rollouts(provider_name=args.provider, task_id=args.task_id, max_repairs=args.max_repairs)
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(json.dumps(summary, indent=2), encoding="utf-8")
+    print(json.dumps({key: summary[key] for key in ["provider", "task_count", "mean_public_score", "accepted_count"]}, indent=2))
+
+
+if __name__ == "__main__":
+    main()

eval/run_oracles.py

@@ -0,0 +1,46 @@
+"""Generate oracle trajectories for all task families."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from statistics import mean
+
+from legacy_cobol_env.eval.oracle_solutions import solution_for_task
+from legacy_cobol_env.eval.trajectory import run_solution_trajectory
+from legacy_cobol_env.server.task_bank import all_tasks
+
+
+ENV_ROOT = Path(__file__).resolve().parents[1]
+OUTPUT_DIR = ENV_ROOT / "outputs" / "evals"
+
+
+def run_oracle_evaluation() -> dict:
+    trajectories = [
+        run_solution_trajectory(
+            policy_name="oracle",
+            task=task,
+            code=solution_for_task(task),
+        )
+        for task in all_tasks()
+    ]
+    return {
+        "policy": "oracle",
+        "task_count": len(trajectories),
+        "mean_public_score": mean(item["final"]["public_score"] for item in trajectories),
+        "accepted_count": sum(1 for item in trajectories if item["final"]["accepted"]),
+        "trajectories": trajectories,
+    }
+
+
+def main() -> None:
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    summary = run_oracle_evaluation()
+    output_path = OUTPUT_DIR / "oracle_trajectories.json"
+    output_path.write_text(json.dumps(summary, indent=2), encoding="utf-8")
+    print(json.dumps({key: summary[key] for key in ["task_count", "mean_public_score", "accepted_count"]}, indent=2))
+
+
+if __name__ == "__main__":
+    main()
+

eval/trajectory.py

@@ -0,0 +1,76 @@
+"""Reusable tool-call trajectory runner for local and model-driven policies."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from openenv.core.env_server.mcp_types import CallToolAction
+
+from legacy_cobol_env.server.legacy_cobol_env_environment import LegacyCobolEnvironment
+from legacy_cobol_env.server.task_bank import TaskInstance
+
+
+def call_tool(env: LegacyCobolEnvironment, tool_name: str, **arguments: Any) -> tuple[dict[str, Any], float, bool]:
+    observation = env.step(CallToolAction(tool_name=tool_name, arguments=arguments))
+    return observation.result.data, observation.reward, observation.done
+
+
+def run_solution_trajectory(
+    policy_name: str,
+    task: TaskInstance,
+    code: str,
+) -> dict[str, Any]:
+    env = LegacyCobolEnvironment()
+    reset_observation = env.reset(task_id=task.task_id)
+    ticket = reset_observation.result["ticket"]
+    steps: list[dict[str, Any]] = []
+
+    def record(tool_name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+        result, reward, done = call_tool(env, tool_name, **arguments)
+        saved_arguments = dict(arguments)
+        if "code" in saved_arguments:
+            saved_arguments["code_chars"] = len(saved_arguments.pop("code"))
+        steps.append(
+            {
+                "tool_name": tool_name,
+                "arguments": saved_arguments,
+                "reward": reward,
+                "done": done,
+                "result": result,
+            }
+        )
+        return result
+
+    for filename in ticket["available_files"]:
+        record("read_cobol_file", {"filename": filename})
+    for filename in ticket["available_copybooks"]:
+        record("read_copybook", {"filename": filename})
+        record("parse_copybook_layout", {"filename": filename})
+    record("inspect_business_rules", {})
+    written = record("write_python_solution", {"code": code})
+    visible = record("run_visible_tests", {"draft_id": written["draft_id"]})
+    final = record("submit_final", {"draft_id": written["draft_id"]})
+
+    return {
+        "policy": policy_name,
+        "task_id": task.task_id,
+        "family_id": task.family_id,
+        "ticket": ticket,
+        "visible": {
+            "passed": visible["passed"],
+            "total": visible["total"],
+            "pass_rate": visible["pass_rate"],
+            "failures": visible["failures"],
+        },
+        "final": {
+            "public_score": final["public_score"],
+            "accepted": final["accepted"],
+            "components": final["components"],
+            "hidden_passed": final["hidden_passed"],
+            "hidden_total": final["hidden_total"],
+            "fresh_passed": final["fresh_passed"],
+            "fresh_total": final["fresh_total"],
+        },
+        "steps": steps,
+    }
+

inference.py

@@ -0,0 +1,252 @@
+"""Submission-root inference entrypoint for OpenEnv gates."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from statistics import mean
+from typing import Any, Mapping, Sequence, TextIO
+from urllib.parse import urlparse
+
+
+ENV_ROOT = Path(__file__).resolve().parent
+PARENT = ENV_ROOT.parent
+if str(PARENT) not in sys.path:
+    sys.path.insert(0, str(PARENT))
+
+from openai import OpenAI
+
+from legacy_cobol_env.eval.model_rollout import run_model_repair_rollout, run_model_rollout
+from legacy_cobol_env.eval.providers import StaticResponseProvider, TextProvider
+from legacy_cobol_env.server.task_bank import TaskInstance, all_tasks, load_task
+
+
+VALID_MARKERS = {"START", "STEP", "END"}
+STATIC_RESPONSE = '{"code": "def migrate(input_record: str) -> str:\\n    return input_record\\n"}'
+
+
+@dataclass(frozen=True)
+class RuntimeConfig:
+    api_base_url: str
+    model_name: str
+    hf_token: str
+    mode: str
+    api_version: str = "2024-12-01-preview"
+
+
+def load_runtime_config(env: Mapping[str, str] | None = None, mode: str | None = None) -> RuntimeConfig:
+    values = os.environ if env is None else env
+    selected_mode = mode or values.get("INFERENCE_MODE") or values.get("MODE") or "live"
+
+    if selected_mode in {"static", "mock"}:
+        return RuntimeConfig(
+            api_base_url=values.get("API_BASE_URL", ""),
+            model_name=values.get("MODEL_NAME", "static"),
+            hf_token=values.get("HF_TOKEN", ""),
+            mode=selected_mode,
+            api_version=values.get(
+                "API_VERSION",
+                values.get(
+                    "OPENAI_API_VERSION",
+                    values.get("AZURE_OPENAI_API_VERSION", "2024-12-01-preview"),
+                ),
+            ),
+        )
+
+    required = ["API_BASE_URL", "MODEL_NAME", "HF_TOKEN"]
+    missing = [key for key in required if not values.get(key)]
+    if missing:
+        raise ValueError(f"missing inference configuration: {', '.join(missing)}")
+
+    return RuntimeConfig(
+        api_base_url=values["API_BASE_URL"],
+        model_name=values["MODEL_NAME"],
+        hf_token=values["HF_TOKEN"],
+        mode=selected_mode,
+        api_version=values.get(
+            "API_VERSION",
+            values.get("OPENAI_API_VERSION", values.get("AZURE_OPENAI_API_VERSION", "2024-12-01-preview")),
+        ),
+    )
+
+
+def format_event(marker: str, payload: Mapping[str, object]) -> str:
+    if marker not in VALID_MARKERS:
+        raise ValueError(f"invalid log marker: {marker}")
+    data = json.dumps(dict(payload), sort_keys=True, separators=(",", ":"))
+    return f"[{marker}] {data}"
+
+
+def build_openai_client(config: RuntimeConfig) -> OpenAI:
+    base_url = config.api_base_url.rstrip("/")
+    if _is_azure_endpoint(base_url):
+        deployment_base = (
+            base_url
+            if "/openai/deployments/" in base_url
+            else f"{base_url}/openai/deployments/{config.model_name}"
+        )
+        return OpenAI(
+            base_url=deployment_base,
+            api_key=config.hf_token,
+            default_headers={"api-key": config.hf_token},
+            default_query={"api-version": config.api_version},
+            timeout=60.0,
+        )
+    return OpenAI(base_url=base_url, api_key=config.hf_token, timeout=60.0)
+
+
+class OpenAITextProvider:
+    name = "openai-client"
+
+    def __init__(self, client: OpenAI, model_name: str) -> None:
+        self._client = client
+        self._model_name = model_name
+
+    def generate(self, prompt: str) -> str:
+        response = self._client.chat.completions.create(
+            model=self._model_name,
+            messages=[
+                {
+                    "role": "system",
+                    "content": "Return only JSON with a single code field containing a Python migrate function.",
+                },
+                {"role": "user", "content": prompt},
+            ],
+            temperature=0,
+        )
+        return response.choices[0].message.content or ""
+
+
+def build_provider(config: RuntimeConfig) -> TextProvider:
+    if config.mode in {"static", "mock"}:
+        return StaticResponseProvider(config.mode, STATIC_RESPONSE)
+    return OpenAITextProvider(build_openai_client(config), config.model_name)
+
+
+def run_inference(task_id: str | None, max_repairs: int, config: RuntimeConfig) -> dict[str, object]:
+    tasks = [load_task(task_id=task_id)] if task_id else all_tasks()
+    provider = build_provider(config)
+    results = [_run_task(task, provider, max_repairs) for task in tasks]
+
+    return {
+        "mode": config.mode,
+        "model_name": config.model_name,
+        "max_repairs": max_repairs,
+        "task_count": len(results),
+        "mean_public_score": mean(item["score"] for item in results) if results else 0.0,
+        "accepted_count": sum(1 for item in results if item["accepted"]),
+        "results": results,
+    }
+
+
+def _run_task(task: TaskInstance, provider: TextProvider, max_repairs: int) -> dict[str, Any]:
+    started = time.perf_counter()
+    try:
+        trajectory = (
+            run_model_repair_rollout(task=task, provider=provider, max_repairs=max_repairs)
+            if max_repairs > 0
+            else run_model_rollout(task=task, provider=provider)
+        )
+    except Exception as exc:
+        return {
+            "task_id": task.task_id,
+            "family_id": task.family_id,
+            "difficulty": task.metadata["difficulty"],
+            "score": 0.0,
+            "accepted": False,
+            "duration_s": round(time.perf_counter() - started, 3),
+            "error": f"{type(exc).__name__}: {exc}",
+        }
+
+    final = trajectory["final"]
+    return {
+        "task_id": task.task_id,
+        "family_id": task.family_id,
+        "difficulty": task.metadata["difficulty"],
+        "score": final["public_score"],
+        "accepted": final["accepted"],
+        "visible_pass_rate": trajectory["visible"]["pass_rate"],
+        "duration_s": round(time.perf_counter() - started, 3),
+        "trajectory": trajectory,
+    }
+
+
+def write_output(path: str | None, payload: Mapping[str, object]) -> None:
+    if not path:
+        return
+    output_path = Path(path)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(json.dumps(dict(payload), sort_keys=True, indent=2) + "\n", encoding="utf-8")
+
+
+def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="Run OpenEnv submission inference gate.")
+    parser.add_argument("--task-id")
+    parser.add_argument("--max-repairs", type=int, default=0)
+    parser.add_argument("--output")
+    parser.add_argument("--mode", choices=["live", "static", "mock"])
+    return parser.parse_args(argv)
+
+
+def main(
+    argv: Sequence[str] | None = None,
+    env: Mapping[str, str] | None = None,
+    stdout: TextIO | None = None,
+) -> int:
+    args = parse_args(argv)
+    stream = sys.stdout if stdout is None else stdout
+    config = load_runtime_config(env, mode=args.mode)
+
+    print(
+        format_event(
+            "START",
+            {
+                "mode": config.mode,
+                "model_name": config.model_name,
+                "task_id": args.task_id or "all",
+                "max_repairs": args.max_repairs,
+            },
+        ),
+        file=stream,
+    )
+    result = run_inference(args.task_id, args.max_repairs, config)
+    for index, task_result in enumerate(result["results"], start=1):
+        step_payload = {
+            "index": index,
+            "task_id": task_result["task_id"],
+            "family_id": task_result["family_id"],
+            "difficulty": task_result["difficulty"],
+            "score": task_result["score"],
+            "accepted": task_result["accepted"],
+        }
+        if "error" in task_result:
+            step_payload["error"] = task_result["error"]
+        print(format_event("STEP", step_payload), file=stream)
+    write_output(args.output, result)
+    print(
+        format_event(
+            "END",
+            {
+                "mode": config.mode,
+                "task_count": result["task_count"],
+                "mean_public_score": result["mean_public_score"],
+                "accepted_count": result["accepted_count"],
+            },
+        ),
+        file=stream,
+    )
+    return 0
+
+
+def _is_azure_endpoint(base_url: str) -> bool:
+    host = urlparse(base_url).netloc.lower()
+    return "openai.azure.com" in host or "cognitiveservices.azure.com" in host
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

models.py

@@ -0,0 +1,77 @@
+"""Shared state models for the Legacy COBOL Migration Workbench."""
+
+from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
+from openenv.core.env_server.types import State
+from pydantic import BaseModel, Field
+
+
+Score = float
+
+
+class RewardComponents(BaseModel):
+    """Public final reward component contract."""
+
+    hidden_correctness: Score = Field(ge=0.0, le=1.0)
+    fresh_correctness: Score = Field(ge=0.0, le=1.0)
+    interface_contract: Score = Field(ge=0.0, le=1.0)
+    type_and_layout_fidelity: Score = Field(ge=0.0, le=1.0)
+    anti_hardcoding: Score = Field(ge=0.0, le=1.0)
+    safety: Score = Field(ge=0.0, le=1.0)
+
+
+class RewardResponse(BaseModel):
+    """Reward-ish response shared by terminal and scoring payloads."""
+
+    ok: bool
+    reward: Score = Field(default=0.0, ge=0.0, le=1.0)
+    error: str | None = None
+
+
+class FinalSubmissionResult(BaseModel):
+    """Structured result returned by submit_final."""
+
+    ok: bool
+    accepted: bool
+    episode_done: bool
+    public_score: Score = Field(ge=0.0, le=1.0)
+    components: RewardComponents
+    hidden_passed: int | None = None
+    hidden_total: int | None = None
+    fresh_passed: int | None = None
+    fresh_total: int | None = None
+    notes: str | None = None
+
+
+class TerminalStepResult(BaseModel):
+    """Structured no-op result for blocked episode steps."""
+
+    ok: bool = False
+    error: str
+    terminal: bool = True
+
+
+class ToolActionWrapper(CallToolAction):
+    """Project-level wrapper for MCP tool actions."""
+
+
+class ToolObservationWrapper(CallToolObservation):
+    """Project-level wrapper for MCP tool observations."""
+
+
+class LegacyCobolState(State):
+    """Inspectable episode state for the migration workbench."""
+
+    task_id: str | None = Field(default=None, description="Selected task ID")
+    done: bool = Field(default=False, description="Whether the episode is complete")
+    files_read: list[str] = Field(default_factory=list)
+    copybooks_read: list[str] = Field(default_factory=list)
+    layouts_parsed: list[str] = Field(default_factory=list)
+    business_rules_inspected: bool = Field(default=False)
+    diffs_inspected: list[str] = Field(default_factory=list)
+    draft_count: int = Field(default=0)
+    visible_runs: int = Field(default=0)
+    best_visible_pass_rate: float = Field(default=0.0)
+    final_score: float | None = Field(default=None)
+    last_tool: str | None = Field(default=None)
+    last_result_summary: str | None = Field(default=None)
+    reward_components: dict[str, float] = Field(default_factory=dict)

openenv.yaml

@@ -0,0 +1,7 @@
+spec_version: 1
+name: legacy_cobol_env
+type: space
+runtime: fastapi
+app: server.app:app
+port: 8000
+

outputs/evals/baseline_results.json

@@ -0,0 +1,205 @@
+{
+  "policies": [
+    "blank_width",
+    "identity"
+  ],
+  "task_count": 6,
+  "mean_public_score": {
+    "blank_width": 0.17666666666666667,
+    "identity": 0.15
+  },
+  "results": [
+    {
+      "policy": "identity",
+      "task_id": "payroll_net_pay_001",
+      "family_id": "decimal_copybook_payroll",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "blank_width",
+      "task_id": "payroll_net_pay_001",
+      "family_id": "decimal_copybook_payroll",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "identity",
+      "task_id": "customer_format_001",
+      "family_id": "fixed_width_customer",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "blank_width",
+      "task_id": "customer_format_001",
+      "family_id": "fixed_width_customer",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "identity",
+      "task_id": "claims_eligibility_001",
+      "family_id": "claims_eligibility_branching",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "blank_width",
+      "task_id": "claims_eligibility_001",
+      "family_id": "claims_eligibility_branching",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.23,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 1.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "identity",
+      "task_id": "account_status_001",
+      "family_id": "account_status_level88",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "blank_width",
+      "task_id": "account_status_001",
+      "family_id": "account_status_level88",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.23,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 1.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "identity",
+      "task_id": "invoice_occurs_001",
+      "family_id": "invoice_occurs_totals",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "blank_width",
+      "task_id": "invoice_occurs_001",
+      "family_id": "invoice_occurs_totals",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "identity",
+      "task_id": "date_normalization_001",
+      "family_id": "date_normalization",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    },
+    {
+      "policy": "blank_width",
+      "task_id": "date_normalization_001",
+      "family_id": "date_normalization",
+      "visible_pass_rate": 0.0,
+      "public_score": 0.15,
+      "components": {
+        "hidden_correctness": 0.0,
+        "fresh_correctness": 0.0,
+        "interface_contract": 1.0,
+        "type_and_layout_fidelity": 0.0,
+        "anti_hardcoding": 0.0,
+        "safety": 1.0
+      },
+      "accepted": false
+    }
+  ]
+}

outputs/evals/cobol_invoice_oracle_check.json

@@ -0,0 +1,123 @@
+{
+  "ok": true,
+  "image_tag": "legacy-cobol-invoice-oracle:local",
+  "case_count": 13,
+  "passed_count": 13,
+  "mismatches": [],
+  "cases": [
+    {
+      "case_id": "visible_1",
+      "summary": "two standard and non-taxable lines",
+      "input_record": "I000010202012345S01010000N00000000N00000000N",
+      "expected_output": "I0000100003648002L",
+      "actual_output": "I0000100003648002L",
+      "passed": true
+    },
+    {
+      "case_id": "visible_2",
+      "summary": "zero item invoice",
+      "input_record": "I000020000000000N00000000N00000000N00000000N",
+      "expected_output": "I0000200000000000L",
+      "actual_output": "I0000200000000000L",
+      "passed": true
+    },
+    {
+      "case_id": "visible_3",
+      "summary": "four item high invoice",
+      "input_record": "I000030404025000S01099000N03012345R02050000L",
+      "expected_output": "I0000300035421104H",
+      "actual_output": "I0000300035421104H",
+      "passed": true
+    },
+    {
+      "case_id": "hidden_1",
+      "summary": "standard tax rounding probe",
+      "input_record": "I000040103033333S00000000N00000000N00000000N",
+      "expected_output": "I0000400010724901H",
+      "actual_output": "I0000400010724901H",
+      "passed": true
+    },
+    {
+      "case_id": "hidden_2",
+      "summary": "count ignores padded groups",
+      "input_record": "I000050101000100N00000000N00000000N00000000N",
+      "expected_output": "I0000500000010001L",
+      "actual_output": "I0000500000010001L",
+      "passed": true
+    },
+    {
+      "case_id": "hidden_3",
+      "summary": "threshold flag high with luxury code",
+      "input_record": "I000060104030000L00000000N00000000N00000000N",
+      "expected_output": "I0000600013200001H",
+      "actual_output": "I0000600013200001H",
+      "passed": true
+    },
+    {
+      "case_id": "hidden_4",
+      "summary": "reduced code generalization",
+      "input_record": "I000070207022222R02010101S00000000N00000000N",
+      "expected_output": "I0000700018111002H",
+      "actual_output": "I0000700018111002H",
+      "passed": true
+    },
+    {
+      "case_id": "fresh_20260429_0",
+      "summary": "fresh invoice occurs case",
+      "input_record": "F000000000000000N00000000N00000000N00000000N",
+      "expected_output": "F0000000000000000L",
+      "actual_output": "F0000000000000000L",
+      "passed": true
+    },
+    {
+      "case_id": "fresh_20260429_1",
+      "summary": "fresh invoice occurs case",
+      "input_record": "F000010307027373N08012881S02050992L00000000N",
+      "expected_output": "F0000100041431203H",
+      "actual_output": "F0000100041431203H",
+      "passed": true
+    },
+    {
+      "case_id": "fresh_20260429_2",
+      "summary": "fresh invoice occurs case",
+      "input_record": "F000020101009336S00000000N00000000N00000000N",
+      "expected_output": "F0000200001001301L",
+      "actual_output": "F0000200001001301L",
+      "passed": true
+    },
+    {
+      "case_id": "fresh_20260429_3",
+      "summary": "fresh invoice occurs case",
+      "input_record": "F000030104016782N00000000N00000000N00000000N",
+      "expected_output": "F0000300006712801L",
+      "actual_output": "F0000300006712801L",
+      "passed": true
+    },
+    {
+      "case_id": "fresh_20260429_4",
+      "summary": "fresh invoice occurs case",
+      "input_record": "F000040303084629N08033585N08035187R00000000N",
+      "expected_output": "F0000400081110003H",
+      "actual_output": "F0000400081110003H",
+      "passed": true
+    },
+    {
+      "case_id": "fresh_20260429_5",
+      "summary": "fresh invoice occurs case",
+      "input_record": "F000050107087851R00000000N00000000N00000000N",
+      "expected_output": "F0000500063033101H",
+      "actual_output": "F0000500063033101H",
+      "passed": true
+    }
+  ],
+  "source_compile": {
+    "ok": true,
+    "compiled_sources": [
+      "INVTOTAL.cbl",
+      "TAXRATE.cbl"
+    ],
+    "returncode": 0,
+    "stdout": "",
+    "stderr": ""
+  }
+}

outputs/evals/historical/azure_gpt54mini_payroll_probe.json

@@ -0,0 +1,301 @@
+{
+  "provider": "azure-openai",
+  "task_count": 1,
+  "mean_public_score": 0.15,
+  "accepted_count": 0,
+  "trajectories": [
+    {
+      "policy": "azure-openai",
+      "task_id": "payroll_net_pay_001",
+      "family_id": "decimal_copybook_payroll",
+      "ticket": {
+        "task_id": "payroll_net_pay_001",
+        "family_id": "decimal_copybook_payroll",
+        "domain": "payroll",
+        "ticket": "Migrate PAYROLL.cbl while preserving copybook offsets, implied decimals, signed deductions, bonus condition names, and fixed-width ACH output.",
+        "available_files": [
+          "PAYROLL.cbl"
+        ],
+        "available_copybooks": [
+          "EMPLOYEE_PAY.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 5,
+        "input_width": 42,
+        "output_width": 28,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "azure-openai",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"payroll_net_pay_001\",\n  \"family_id\": \"decimal_copybook_payroll\",\n  \"domain\": \"payroll\",\n  \"ticket\": \"Migrate PAYROLL.cbl while preserving copybook offsets, implied decimals, signed deductions, bonus condition names, and fixed-width ACH output.\",\n  \"available_files\": [\n    \"PAYROLL.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"EMPLOYEE_PAY.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 5,\n  \"input_width\": 42,\n  \"output_width\": 28,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"PAYROLL.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. PAYROLLNET.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY EMPLOYEE_PAY.\\n       01 TAX-AMOUNT                 PIC 9(7)V99.\\n       01 NET-AMOUNT                 PIC 9(7)V99.\\n       01 PAY-CATEGORY               PIC X.\\n       PROCEDURE DIVISION.\\n           COMPUTE TAX-AMOUNT ROUNDED = GROSS-PAY * TAX-RATE\\n           COMPUTE NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS\\n           IF BONUS-ELIGIBLE ADD 50.00 TO NET-AMOUNT END-IF\\n           IF NET-AMOUNT < 0 MOVE 0 TO NET-AMOUNT END-IF\\n           EVALUATE TRUE\\n             WHEN NET-AMOUNT >= 5000.00 MOVE 'H' TO PAY-CATEGORY\\n             WHEN NET-AMOUNT >= 2500.00 MOVE 'M' TO PAY-CATEGORY\\n             WHEN OTHER MOVE 'L' TO PAY-CATEGORY\\n           END-EVALUATE\\n           GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"EMPLOYEE_PAY.cpy\": \"       01  EMPLOYEE-PAY-RECORD.\\n           05 EMP-ID                 PIC X(6).\\n           05 EMP-NAME               PIC X(12).\\n           05 GROSS-PAY              PIC 9(7)V99.\\n           05 TAX-RATE               PIC 9V999.\\n           05 DEDUCTIONS             PIC S9(5)V99 SIGN LEADING SEPARATE.\\n           05 BONUS-FLAG             PIC X(1).\\n              88 BONUS-ELIGIBLE      VALUE 'Y'.\\n              88 NO-BONUS            VALUE 'N'.\\n           05 FILLER                 PIC X(2).\\n\"\n}\n\nParsed layouts:\n{\n  \"EMPLOYEE_PAY.cpy\": {\n    \"ok\": true,\n    \"filename\": \"EMPLOYEE_PAY.cpy\",\n    \"record_name\": \"EMPLOYEE-PAY-RECORD\",\n    \"total_width\": 42,\n    \"fields\": [\n      {\n        \"name\": \"EMP-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"EMP-NAME\",\n        \"start\": 6,\n        \"end\": 18,\n        \"length\": 12,\n        \"pic\": \"X(12)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"GROSS-PAY\",\n        \"start\": 18,\n        \"end\": 27,\n        \"length\": 9,\n        \"pic\": \"9(7)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"TAX-RATE\",\n        \"start\": 27,\n        \"end\": 31,\n        \"length\": 4,\n        \"pic\": \"9V999\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 3\n      },\n      {\n        \"name\": \"DEDUCTIONS\",\n        \"start\": 31,\n        \"end\": 39,\n        \"length\": 8,\n        \"pic\": \"S9(5)V99 SIGN LEADING SEPARATE\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"BONUS-FLAG\",\n        \"start\": 39,\n        \"end\": 40,\n        \"length\": 1,\n        \"pic\": \"X(1)\",\n        \"python_type\": \"str\",\n        \"level_88\": {\n          \"BONUS-ELIGIBLE\": \"Y\",\n          \"NO-BONUS\": \"N\"\n        }\n      },\n      {\n        \"name\": \"FILLER\",\n        \"start\": 40,\n        \"end\": 42,\n        \"length\": 2,\n        \"pic\": \"X(2)\",\n        \"python_type\": \"str\"\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"Tax is rounded half up to cents after GROSS-PAY * TAX-RATE.\",\n  \"NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS.\",\n  \"BONUS-ELIGIBLE adds 50.00; negative net pay floors to zero.\",\n  \"Output net pay is a 9-digit zero-padded cents field.\"\n]",
+          "response": "{\"code\":\"from decimal import Decimal, ROUND_HALF_UP\\n\\ndef migrate(input_record: str) -> str:\\n    # Preserve fixed-width input layout from copybook (42 chars)\\n    rec = (input_record or \\\"\\\")[:42].ljust(42)\\n\\n    def dec_slice(start: int, end: int, scale: int) -> Decimal:\\n        raw = rec[start:end]\\n        raw = raw.strip()\\n        if not raw:\\n            return Decimal('0')\\n        sign = 1\\n        if raw[0] == '+':\\n            raw = raw[1:]\\n        elif raw[0] == '-':\\n            sign = -1\\n            raw = raw[1:]\\n        # COBOL SIGN LEADING SEPARATE may place a leading sign in the field.\\n        # For this layout, deductions is 8 chars and may include a sign.\\n        if raw and raw[0] in '+-':\\n            sign = -1 if raw[0] == '-' else 1\\n            raw = raw[1:]\\n        if not raw:\\n            return Decimal('0')\\n        if not raw.isdigit():\\n            # Be tolerant of spaces/garbage by extracting digits only.\\n            digits = ''.join(ch for ch in raw if ch.isdigit())\\n            if not digits:\\n                return Decimal('0')\\n            raw = digits\\n        return Decimal(sign) * (Decimal(int(raw)) / (Decimal(10) ** scale))\\n\\n    gross = dec_slice(18, 27, 2)\\n    tax_rate = dec_slice(27, 31, 3)\\n    deductions = dec_slice(31, 39, 2)\\n    bonus_flag = rec[39:40]\\n\\n    tax_amount = (gross * tax_rate).quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)\\n    net_amount = gross - tax_amount - deductions\\n    if bonus_flag == 'Y':\\n        net_amount += Decimal('50.00')\\n    if net_amount < 0:\\n        net_amount = Decimal('0.00')\\n\\n    # Output: 9-digit zero-padded cents field (fixed-width ACH output)\\n    cents = int((net_amount * 100).quantize(Decimal('1'), rounding=ROUND_HALF_UP))\\n    return f\\\"{cents:09d}\\\"\\n\"}",
+          "code_chars": 1726
+        }
+      ],
+      "visible": {
+        "passed": 0,
+        "total": 3,
+        "pass_rate": 0.0,
+        "failures": [
+          {
+            "case_id": "visible_1",
+            "input_summary": "ordinary bonus-eligible employee",
+            "expected_summary": "OUT-EMP-ID=E00001, OUT-EMP-NAME=ALICE       , OUT-NET-PAY=000093272, OUT-PAY-CATEGORY=L",
+            "actual_summary": "length 9 output: '000093272'",
+            "error": null
+          },
+          {
+            "case_id": "visible_2",
+            "input_summary": "negative adjustment and no bonus",
+            "expected_summary": "OUT-EMP-ID=E00002, OUT-EMP-NAME=BOB         , OUT-NET-PAY=000076750, OUT-PAY-CATEGORY=L",
+            "actual_summary": "length 9 output: '000076750'",
+            "error": null
+          },
+          {
+            "case_id": "visible_3",
+            "input_summary": "low net pay floors at zero",
+            "expected_summary": "OUT-EMP-ID=E00003, OUT-EMP-NAME=CHANDRA     , OUT-NET-PAY=000000000, OUT-PAY-CATEGORY=L",
+            "actual_summary": "length 9 output: '000000000'",
+            "error": null
+          }
+        ]
+      },
+      "final": {
+        "public_score": 0.15,
+        "accepted": false,
+        "components": {
+          "hidden_correctness": 0.0,
+          "fresh_correctness": 0.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 0.0,
+          "anti_hardcoding": 0.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 0,
+        "hidden_total": 5,
+        "fresh_passed": 0,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "PAYROLL.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "PAYROLL.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. PAYROLLNET.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY EMPLOYEE_PAY.\n       01 TAX-AMOUNT                 PIC 9(7)V99.\n       01 NET-AMOUNT                 PIC 9(7)V99.\n       01 PAY-CATEGORY               PIC X.\n       PROCEDURE DIVISION.\n           COMPUTE TAX-AMOUNT ROUNDED = GROSS-PAY * TAX-RATE\n           COMPUTE NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS\n           IF BONUS-ELIGIBLE ADD 50.00 TO NET-AMOUNT END-IF\n           IF NET-AMOUNT < 0 MOVE 0 TO NET-AMOUNT END-IF\n           EVALUATE TRUE\n             WHEN NET-AMOUNT >= 5000.00 MOVE 'H' TO PAY-CATEGORY\n             WHEN NET-AMOUNT >= 2500.00 MOVE 'M' TO PAY-CATEGORY\n             WHEN OTHER MOVE 'L' TO PAY-CATEGORY\n           END-EVALUATE\n           GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "content": "       01  EMPLOYEE-PAY-RECORD.\n           05 EMP-ID                 PIC X(6).\n           05 EMP-NAME               PIC X(12).\n           05 GROSS-PAY              PIC 9(7)V99.\n           05 TAX-RATE               PIC 9V999.\n           05 DEDUCTIONS             PIC S9(5)V99 SIGN LEADING SEPARATE.\n           05 BONUS-FLAG             PIC X(1).\n              88 BONUS-ELIGIBLE      VALUE 'Y'.\n              88 NO-BONUS            VALUE 'N'.\n           05 FILLER                 PIC X(2).\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "record_name": "EMPLOYEE-PAY-RECORD",
+            "total_width": 42,
+            "fields": [
+              {
+                "name": "EMP-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "EMP-NAME",
+                "start": 6,
+                "end": 18,
+                "length": 12,
+                "pic": "X(12)",
+                "python_type": "str"
+              },
+              {
+                "name": "GROSS-PAY",
+                "start": 18,
+                "end": 27,
+                "length": 9,
+                "pic": "9(7)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "TAX-RATE",
+                "start": 27,
+                "end": 31,
+                "length": 4,
+                "pic": "9V999",
+                "python_type": "Decimal",
+                "scale": 3
+              },
+              {
+                "name": "DEDUCTIONS",
+                "start": 31,
+                "end": 39,
+                "length": 8,
+                "pic": "S9(5)V99 SIGN LEADING SEPARATE",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "BONUS-FLAG",
+                "start": 39,
+                "end": 40,
+                "length": 1,
+                "pic": "X(1)",
+                "python_type": "str",
+                "level_88": {
+                  "BONUS-ELIGIBLE": "Y",
+                  "NO-BONUS": "N"
+                }
+              },
+              {
+                "name": "FILLER",
+                "start": 40,
+                "end": 42,
+                "length": 2,
+                "pic": "X(2)",
+                "python_type": "str"
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "Tax is rounded half up to cents after GROSS-PAY * TAX-RATE.",
+              "NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS.",
+              "BONUS-ELIGIBLE adds 50.00; negative net pay floors to zero.",
+              "Output net pay is a 9-digit zero-padded cents field."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 1726
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.0,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 0,
+            "total": 3,
+            "pass_rate": 0.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": [
+              {
+                "case_id": "visible_1",
+                "input_summary": "ordinary bonus-eligible employee",
+                "expected_summary": "OUT-EMP-ID=E00001, OUT-EMP-NAME=ALICE       , OUT-NET-PAY=000093272, OUT-PAY-CATEGORY=L",
+                "actual_summary": "length 9 output: '000093272'",
+                "error": null
+              },
+              {
+                "case_id": "visible_2",
+                "input_summary": "negative adjustment and no bonus",
+                "expected_summary": "OUT-EMP-ID=E00002, OUT-EMP-NAME=BOB         , OUT-NET-PAY=000076750, OUT-PAY-CATEGORY=L",
+                "actual_summary": "length 9 output: '000076750'",
+                "error": null
+              },
+              {
+                "case_id": "visible_3",
+                "input_summary": "low net pay floors at zero",
+                "expected_summary": "OUT-EMP-ID=E00003, OUT-EMP-NAME=CHANDRA     , OUT-NET-PAY=000000000, OUT-PAY-CATEGORY=L",
+                "actual_summary": "length 9 output: '000000000'",
+                "error": null
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.15,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": false,
+            "episode_done": true,
+            "public_score": 0.15,
+            "components": {
+              "hidden_correctness": 0.0,
+              "fresh_correctness": 0.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 0.0,
+              "anti_hardcoding": 0.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 0,
+            "hidden_total": 5,
+            "fresh_passed": 0,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    }
+  ]
+}

outputs/evals/historical/azure_gpt54mini_zeroshot_rollouts.json

@@ -0,0 +1,1595 @@
+{
+  "provider": "azure-openai",
+  "task_count": 6,
+  "mean_public_score": 0.305,
+  "accepted_count": 1,
+  "trajectories": [
+    {
+      "policy": "azure-openai",
+      "task_id": "payroll_net_pay_001",
+      "family_id": "decimal_copybook_payroll",
+      "ticket": {
+        "task_id": "payroll_net_pay_001",
+        "family_id": "decimal_copybook_payroll",
+        "domain": "payroll",
+        "ticket": "Migrate PAYROLL.cbl while preserving copybook offsets, implied decimals, signed deductions, bonus condition names, and fixed-width ACH output.",
+        "available_files": [
+          "PAYROLL.cbl"
+        ],
+        "available_copybooks": [
+          "EMPLOYEE_PAY.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 5,
+        "input_width": 42,
+        "output_width": 28,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "azure-openai",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"payroll_net_pay_001\",\n  \"family_id\": \"decimal_copybook_payroll\",\n  \"domain\": \"payroll\",\n  \"ticket\": \"Migrate PAYROLL.cbl while preserving copybook offsets, implied decimals, signed deductions, bonus condition names, and fixed-width ACH output.\",\n  \"available_files\": [\n    \"PAYROLL.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"EMPLOYEE_PAY.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 5,\n  \"input_width\": 42,\n  \"output_width\": 28,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"PAYROLL.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. PAYROLLNET.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY EMPLOYEE_PAY.\\n       01 TAX-AMOUNT                 PIC 9(7)V99.\\n       01 NET-AMOUNT                 PIC 9(7)V99.\\n       01 PAY-CATEGORY               PIC X.\\n       PROCEDURE DIVISION.\\n           COMPUTE TAX-AMOUNT ROUNDED = GROSS-PAY * TAX-RATE\\n           COMPUTE NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS\\n           IF BONUS-ELIGIBLE ADD 50.00 TO NET-AMOUNT END-IF\\n           IF NET-AMOUNT < 0 MOVE 0 TO NET-AMOUNT END-IF\\n           EVALUATE TRUE\\n             WHEN NET-AMOUNT >= 5000.00 MOVE 'H' TO PAY-CATEGORY\\n             WHEN NET-AMOUNT >= 2500.00 MOVE 'M' TO PAY-CATEGORY\\n             WHEN OTHER MOVE 'L' TO PAY-CATEGORY\\n           END-EVALUATE\\n           GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"EMPLOYEE_PAY.cpy\": \"       01  EMPLOYEE-PAY-RECORD.\\n           05 EMP-ID                 PIC X(6).\\n           05 EMP-NAME               PIC X(12).\\n           05 GROSS-PAY              PIC 9(7)V99.\\n           05 TAX-RATE               PIC 9V999.\\n           05 DEDUCTIONS             PIC S9(5)V99 SIGN LEADING SEPARATE.\\n           05 BONUS-FLAG             PIC X(1).\\n              88 BONUS-ELIGIBLE      VALUE 'Y'.\\n              88 NO-BONUS            VALUE 'N'.\\n           05 FILLER                 PIC X(2).\\n\"\n}\n\nParsed layouts:\n{\n  \"EMPLOYEE_PAY.cpy\": {\n    \"ok\": true,\n    \"filename\": \"EMPLOYEE_PAY.cpy\",\n    \"record_name\": \"EMPLOYEE-PAY-RECORD\",\n    \"total_width\": 42,\n    \"fields\": [\n      {\n        \"name\": \"EMP-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"EMP-NAME\",\n        \"start\": 6,\n        \"end\": 18,\n        \"length\": 12,\n        \"pic\": \"X(12)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"GROSS-PAY\",\n        \"start\": 18,\n        \"end\": 27,\n        \"length\": 9,\n        \"pic\": \"9(7)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"TAX-RATE\",\n        \"start\": 27,\n        \"end\": 31,\n        \"length\": 4,\n        \"pic\": \"9V999\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 3\n      },\n      {\n        \"name\": \"DEDUCTIONS\",\n        \"start\": 31,\n        \"end\": 39,\n        \"length\": 8,\n        \"pic\": \"S9(5)V99 SIGN LEADING SEPARATE\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"BONUS-FLAG\",\n        \"start\": 39,\n        \"end\": 40,\n        \"length\": 1,\n        \"pic\": \"X(1)\",\n        \"python_type\": \"str\",\n        \"level_88\": {\n          \"BONUS-ELIGIBLE\": \"Y\",\n          \"NO-BONUS\": \"N\"\n        }\n      },\n      {\n        \"name\": \"FILLER\",\n        \"start\": 40,\n        \"end\": 42,\n        \"length\": 2,\n        \"pic\": \"X(2)\",\n        \"python_type\": \"str\"\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"Tax is rounded half up to cents after GROSS-PAY * TAX-RATE.\",\n  \"NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS.\",\n  \"BONUS-ELIGIBLE adds 50.00; negative net pay floors to zero.\",\n  \"Output net pay is a 9-digit zero-padded cents field.\"\n]",
+          "response": "{\"code\":\"from decimal import Decimal, ROUND_HALF_UP\\n\\ndef migrate(input_record: str) -> str:\\n    rec = (input_record or \\\"\\\")\\n    if len(rec) < 42:\\n        rec = rec.ljust(42)\\n    else:\\n        rec = rec[:42]\\n\\n    gross_str = rec[18:27]\\n    tax_rate_str = rec[27:31]\\n    deductions_str = rec[31:39]\\n    bonus_flag = rec[39:40]\\n\\n    gross = Decimal(gross_str) / Decimal('100')\\n    tax_rate = Decimal(tax_rate_str) / Decimal('1000')\\n    deductions = Decimal(deductions_str) / Decimal('100')\\n\\n    tax_amount = (gross * tax_rate).quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)\\n    net_amount = gross - tax_amount - deductions\\n\\n    if bonus_flag == 'Y':\\n        net_amount += Decimal('50.00')\\n\\n    if net_amount < 0:\\n        net_amount = Decimal('0.00')\\n\\n    cents = int((net_amount * 100).quantize(Decimal('1'), rounding=ROUND_HALF_UP))\\n    return f\\\"{cents:09d}\\\"\\n\"}",
+          "code_chars": 850
+        }
+      ],
+      "visible": {
+        "passed": 0,
+        "total": 3,
+        "pass_rate": 0.0,
+        "failures": [
+          {
+            "case_id": "visible_1",
+            "input_summary": "ordinary bonus-eligible employee",
+            "expected_summary": "OUT-EMP-ID=E00001, OUT-EMP-NAME=ALICE       , OUT-NET-PAY=000093272, OUT-PAY-CATEGORY=L",
+            "actual_summary": "length 9 output: '000093272'",
+            "error": null
+          },
+          {
+            "case_id": "visible_2",
+            "input_summary": "negative adjustment and no bonus",
+            "expected_summary": "OUT-EMP-ID=E00002, OUT-EMP-NAME=BOB         , OUT-NET-PAY=000076750, OUT-PAY-CATEGORY=L",
+            "actual_summary": "length 9 output: '000076750'",
+            "error": null
+          },
+          {
+            "case_id": "visible_3",
+            "input_summary": "low net pay floors at zero",
+            "expected_summary": "OUT-EMP-ID=E00003, OUT-EMP-NAME=CHANDRA     , OUT-NET-PAY=000000000, OUT-PAY-CATEGORY=L",
+            "actual_summary": "length 9 output: '000000000'",
+            "error": null
+          }
+        ]
+      },
+      "final": {
+        "public_score": 0.15,
+        "accepted": false,
+        "components": {
+          "hidden_correctness": 0.0,
+          "fresh_correctness": 0.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 0.0,
+          "anti_hardcoding": 0.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 0,
+        "hidden_total": 5,
+        "fresh_passed": 0,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "PAYROLL.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "PAYROLL.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. PAYROLLNET.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY EMPLOYEE_PAY.\n       01 TAX-AMOUNT                 PIC 9(7)V99.\n       01 NET-AMOUNT                 PIC 9(7)V99.\n       01 PAY-CATEGORY               PIC X.\n       PROCEDURE DIVISION.\n           COMPUTE TAX-AMOUNT ROUNDED = GROSS-PAY * TAX-RATE\n           COMPUTE NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS\n           IF BONUS-ELIGIBLE ADD 50.00 TO NET-AMOUNT END-IF\n           IF NET-AMOUNT < 0 MOVE 0 TO NET-AMOUNT END-IF\n           EVALUATE TRUE\n             WHEN NET-AMOUNT >= 5000.00 MOVE 'H' TO PAY-CATEGORY\n             WHEN NET-AMOUNT >= 2500.00 MOVE 'M' TO PAY-CATEGORY\n             WHEN OTHER MOVE 'L' TO PAY-CATEGORY\n           END-EVALUATE\n           GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "content": "       01  EMPLOYEE-PAY-RECORD.\n           05 EMP-ID                 PIC X(6).\n           05 EMP-NAME               PIC X(12).\n           05 GROSS-PAY              PIC 9(7)V99.\n           05 TAX-RATE               PIC 9V999.\n           05 DEDUCTIONS             PIC S9(5)V99 SIGN LEADING SEPARATE.\n           05 BONUS-FLAG             PIC X(1).\n              88 BONUS-ELIGIBLE      VALUE 'Y'.\n              88 NO-BONUS            VALUE 'N'.\n           05 FILLER                 PIC X(2).\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "record_name": "EMPLOYEE-PAY-RECORD",
+            "total_width": 42,
+            "fields": [
+              {
+                "name": "EMP-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "EMP-NAME",
+                "start": 6,
+                "end": 18,
+                "length": 12,
+                "pic": "X(12)",
+                "python_type": "str"
+              },
+              {
+                "name": "GROSS-PAY",
+                "start": 18,
+                "end": 27,
+                "length": 9,
+                "pic": "9(7)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "TAX-RATE",
+                "start": 27,
+                "end": 31,
+                "length": 4,
+                "pic": "9V999",
+                "python_type": "Decimal",
+                "scale": 3
+              },
+              {
+                "name": "DEDUCTIONS",
+                "start": 31,
+                "end": 39,
+                "length": 8,
+                "pic": "S9(5)V99 SIGN LEADING SEPARATE",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "BONUS-FLAG",
+                "start": 39,
+                "end": 40,
+                "length": 1,
+                "pic": "X(1)",
+                "python_type": "str",
+                "level_88": {
+                  "BONUS-ELIGIBLE": "Y",
+                  "NO-BONUS": "N"
+                }
+              },
+              {
+                "name": "FILLER",
+                "start": 40,
+                "end": 42,
+                "length": 2,
+                "pic": "X(2)",
+                "python_type": "str"
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "Tax is rounded half up to cents after GROSS-PAY * TAX-RATE.",
+              "NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS.",
+              "BONUS-ELIGIBLE adds 50.00; negative net pay floors to zero.",
+              "Output net pay is a 9-digit zero-padded cents field."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 850
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.0,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 0,
+            "total": 3,
+            "pass_rate": 0.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": [
+              {
+                "case_id": "visible_1",
+                "input_summary": "ordinary bonus-eligible employee",
+                "expected_summary": "OUT-EMP-ID=E00001, OUT-EMP-NAME=ALICE       , OUT-NET-PAY=000093272, OUT-PAY-CATEGORY=L",
+                "actual_summary": "length 9 output: '000093272'",
+                "error": null
+              },
+              {
+                "case_id": "visible_2",
+                "input_summary": "negative adjustment and no bonus",
+                "expected_summary": "OUT-EMP-ID=E00002, OUT-EMP-NAME=BOB         , OUT-NET-PAY=000076750, OUT-PAY-CATEGORY=L",
+                "actual_summary": "length 9 output: '000076750'",
+                "error": null
+              },
+              {
+                "case_id": "visible_3",
+                "input_summary": "low net pay floors at zero",
+                "expected_summary": "OUT-EMP-ID=E00003, OUT-EMP-NAME=CHANDRA     , OUT-NET-PAY=000000000, OUT-PAY-CATEGORY=L",
+                "actual_summary": "length 9 output: '000000000'",
+                "error": null
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.15,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": false,
+            "episode_done": true,
+            "public_score": 0.15,
+            "components": {
+              "hidden_correctness": 0.0,
+              "fresh_correctness": 0.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 0.0,
+              "anti_hardcoding": 0.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 0,
+            "hidden_total": 5,
+            "fresh_passed": 0,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "azure-openai",
+      "task_id": "customer_format_001",
+      "family_id": "fixed_width_customer",
+      "ticket": {
+        "task_id": "customer_format_001",
+        "family_id": "fixed_width_customer",
+        "domain": "customer master data",
+        "ticket": "Migrate CUSTFMT.cbl. Preserve fixed-width name formatting, status mapping, ZIP leading zeros, and zero-padded balance output.",
+        "available_files": [
+          "CUSTFMT.cbl"
+        ],
+        "available_copybooks": [
+          "CUSTOMER_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 40,
+        "output_width": 41,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "azure-openai",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"customer_format_001\",\n  \"family_id\": \"fixed_width_customer\",\n  \"domain\": \"customer master data\",\n  \"ticket\": \"Migrate CUSTFMT.cbl. Preserve fixed-width name formatting, status mapping, ZIP leading zeros, and zero-padded balance output.\",\n  \"available_files\": [\n    \"CUSTFMT.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"CUSTOMER_REC.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 40,\n  \"output_width\": 41,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"CUSTFMT.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. CUSTFMT.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY CUSTOMER_REC.\\n       01 OUTPUT-RECORD.\\n          05 OUT-CUST-ID             PIC X(5).\\n          05 OUT-FULL-NAME           PIC X(22).\\n          05 OUT-ZIP-CODE            PIC 9(5).\\n          05 OUT-STATUS              PIC X.\\n          05 OUT-BALANCE             PIC 9(6)V99.\\n       PROCEDURE DIVISION.\\n          STRING LAST-NAME DELIMITED BY SIZE ', ' FIRST-NAME DELIMITED BY SIZE\\n             INTO OUT-FULL-NAME\\n          EVALUATE STATUS-CODE\\n             WHEN 'A' MOVE 'O' TO OUT-STATUS\\n             WHEN 'S' MOVE 'S' TO OUT-STATUS\\n             WHEN OTHER MOVE 'C' TO OUT-STATUS\\n          END-EVALUATE\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"CUSTOMER_REC.cpy\": \"       01  CUSTOMER-RECORD.\\n           05 CUST-ID                PIC X(5).\\n           05 FIRST-NAME             PIC X(10).\\n           05 LAST-NAME              PIC X(12).\\n           05 ZIP-CODE               PIC 9(5).\\n           05 STATUS-CODE            PIC X.\\n              88 ACTIVE-CUSTOMER     VALUE 'A'.\\n              88 SUSPENDED-CUSTOMER  VALUE 'S'.\\n              88 CLOSED-CUSTOMER     VALUE 'C'.\\n           05 BALANCE-DUE            PIC 9(5)V99.\\n\"\n}\n\nParsed layouts:\n{\n  \"CUSTOMER_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"CUSTOMER_REC.cpy\",\n    \"record_name\": \"CUSTOMER-RECORD\",\n    \"total_width\": 40,\n    \"fields\": [\n      {\n        \"name\": \"CUST-ID\",\n        \"start\": 0,\n        \"end\": 5,\n        \"length\": 5,\n        \"pic\": \"X(5)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"FIRST-NAME\",\n        \"start\": 5,\n        \"end\": 15,\n        \"length\": 10,\n        \"pic\": \"X(10)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"LAST-NAME\",\n        \"start\": 15,\n        \"end\": 27,\n        \"length\": 12,\n        \"pic\": \"X(12)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"ZIP-CODE\",\n        \"start\": 27,\n        \"end\": 32,\n        \"length\": 5,\n        \"pic\": \"9(5)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"STATUS-CODE\",\n        \"start\": 32,\n        \"end\": 33,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"BALANCE-DUE\",\n        \"start\": 33,\n        \"end\": 40,\n        \"length\": 7,\n        \"pic\": \"9(5)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"OUT-FULL-NAME is LAST-NAME, comma-space, FIRST-NAME, truncated/padded to 22.\",\n  \"A maps to O, S maps to S, anything else maps to C.\",\n  \"ZIP stays five digits and OUT-BALANCE is eight zero-padded digits.\"\n]",
+          "response": "{\"code\":\"def migrate(input_record: str) -> str:\\n    \\\"\\\"\\\"Migrate a fixed-width customer record from legacy COBOL format.\\n\\n    Input width: 40\\n    Output width: 41\\n    \\\"\\\"\\\"\\n    rec = (input_record or \\\"\\\")\\n    if len(rec) < 40:\\n        rec = rec.ljust(40)\\n    else:\\n        rec = rec[:40]\\n\\n    cust_id = rec[0:5]\\n    first_name = rec[5:15].rstrip()\\n    last_name = rec[15:27].rstrip()\\n    zip_code = rec[27:32]\\n    status_code = rec[32:33]\\n    balance_raw = rec[33:40]\\n\\n    full_name = f\\\"{last_name}, {first_name}\\\"\\n    full_name = full_name[:22].ljust(22)\\n\\n    if status_code == 'A':\\n        out_status = 'O'\\n    elif status_code == 'S':\\n        out_status = 'S'\\n    else:\\n        out_status = 'C'\\n\\n    # COBOL PIC 9(5)V99 -> 7 input digits, output requires 8 zero-padded digits.\\n    # Preserve numeric value and emit 8 digits.\\n    digits = ''.join(ch for ch in balance_raw if ch.isdigit())\\n    digits = digits.rjust(7, '0')[-7:]\\n    out_balance = digits.rjust(8, '0')\\n\\n    return f\\\"{cust_id}{full_name}{zip_code}{out_status}{out_balance}\\\"\\n\"}",
+          "code_chars": 1025
+        }
+      ],
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "CUSTFMT.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTFMT.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. CUSTFMT.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY CUSTOMER_REC.\n       01 OUTPUT-RECORD.\n          05 OUT-CUST-ID             PIC X(5).\n          05 OUT-FULL-NAME           PIC X(22).\n          05 OUT-ZIP-CODE            PIC 9(5).\n          05 OUT-STATUS              PIC X.\n          05 OUT-BALANCE             PIC 9(6)V99.\n       PROCEDURE DIVISION.\n          STRING LAST-NAME DELIMITED BY SIZE ', ' FIRST-NAME DELIMITED BY SIZE\n             INTO OUT-FULL-NAME\n          EVALUATE STATUS-CODE\n             WHEN 'A' MOVE 'O' TO OUT-STATUS\n             WHEN 'S' MOVE 'S' TO OUT-STATUS\n             WHEN OTHER MOVE 'C' TO OUT-STATUS\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "CUSTOMER_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTOMER_REC.cpy",
+            "content": "       01  CUSTOMER-RECORD.\n           05 CUST-ID                PIC X(5).\n           05 FIRST-NAME             PIC X(10).\n           05 LAST-NAME              PIC X(12).\n           05 ZIP-CODE               PIC 9(5).\n           05 STATUS-CODE            PIC X.\n              88 ACTIVE-CUSTOMER     VALUE 'A'.\n              88 SUSPENDED-CUSTOMER  VALUE 'S'.\n              88 CLOSED-CUSTOMER     VALUE 'C'.\n           05 BALANCE-DUE            PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "CUSTOMER_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTOMER_REC.cpy",
+            "record_name": "CUSTOMER-RECORD",
+            "total_width": 40,
+            "fields": [
+              {
+                "name": "CUST-ID",
+                "start": 0,
+                "end": 5,
+                "length": 5,
+                "pic": "X(5)",
+                "python_type": "str"
+              },
+              {
+                "name": "FIRST-NAME",
+                "start": 5,
+                "end": 15,
+                "length": 10,
+                "pic": "X(10)",
+                "python_type": "str"
+              },
+              {
+                "name": "LAST-NAME",
+                "start": 15,
+                "end": 27,
+                "length": 12,
+                "pic": "X(12)",
+                "python_type": "str"
+              },
+              {
+                "name": "ZIP-CODE",
+                "start": 27,
+                "end": 32,
+                "length": 5,
+                "pic": "9(5)",
+                "python_type": "str"
+              },
+              {
+                "name": "STATUS-CODE",
+                "start": 32,
+                "end": 33,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "BALANCE-DUE",
+                "start": 33,
+                "end": 40,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "OUT-FULL-NAME is LAST-NAME, comma-space, FIRST-NAME, truncated/padded to 22.",
+              "A maps to O, S maps to S, anything else maps to C.",
+              "ZIP stays five digits and OUT-BALANCE is eight zero-padded digits."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 1025
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "azure-openai",
+      "task_id": "claims_eligibility_001",
+      "family_id": "claims_eligibility_branching",
+      "ticket": {
+        "task_id": "claims_eligibility_001",
+        "family_id": "claims_eligibility_branching",
+        "domain": "insurance claims",
+        "ticket": "Migrate CLMELIG.cbl. Preserve EVALUATE TRUE branch precedence for claim approvals, denials, and manual review reasons.",
+        "available_files": [
+          "CLMELIG.cbl"
+        ],
+        "available_copybooks": [
+          "CLAIM_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 21,
+        "output_width": 9,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "azure-openai",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"claims_eligibility_001\",\n  \"family_id\": \"claims_eligibility_branching\",\n  \"domain\": \"insurance claims\",\n  \"ticket\": \"Migrate CLMELIG.cbl. Preserve EVALUATE TRUE branch precedence for claim approvals, denials, and manual review reasons.\",\n  \"available_files\": [\n    \"CLMELIG.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"CLAIM_REC.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 21,\n  \"output_width\": 9,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"CLMELIG.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. CLMELIG.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY CLAIM_REC.\\n       PROCEDURE DIVISION.\\n          EVALUATE TRUE\\n             WHEN AGE < 18 MOVE 'D' TO DECISION MOVE 'A1' TO REASON\\n             WHEN PLAN-CODE = 'B' AND CLAIM-AMOUNT > 1500.00 MOVE 'R' TO DECISION MOVE 'B2' TO REASON\\n             WHEN PREAUTH-FLAG = 'N' AND CLAIM-AMOUNT > 1000.00 MOVE 'D' TO DECISION MOVE 'P1' TO REASON\\n             WHEN SERVICE-DAYS > 030 MOVE 'R' TO DECISION MOVE 'L1' TO REASON\\n             WHEN OTHER MOVE 'A' TO DECISION MOVE 'OK' TO REASON\\n          END-EVALUATE\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"CLAIM_REC.cpy\": \"       01  CLAIM-RECORD.\\n           05 CLAIM-ID               PIC X(6).\\n           05 AGE                    PIC 9(3).\\n           05 PLAN-CODE              PIC X.\\n           05 SERVICE-DAYS           PIC 9(3).\\n           05 PREAUTH-FLAG           PIC X.\\n           05 CLAIM-AMOUNT           PIC 9(5)V99.\\n\"\n}\n\nParsed layouts:\n{\n  \"CLAIM_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"CLAIM_REC.cpy\",\n    \"record_name\": \"CLAIM-RECORD\",\n    \"total_width\": 21,\n    \"fields\": [\n      {\n        \"name\": \"CLAIM-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"AGE\",\n        \"start\": 6,\n        \"end\": 9,\n        \"length\": 3,\n        \"pic\": \"9(3)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"PLAN-CODE\",\n        \"start\": 9,\n        \"end\": 10,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"SERVICE-DAYS\",\n        \"start\": 10,\n        \"end\": 13,\n        \"length\": 3,\n        \"pic\": \"9(3)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"PREAUTH-FLAG\",\n        \"start\": 13,\n        \"end\": 14,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"CLAIM-AMOUNT\",\n        \"start\": 14,\n        \"end\": 21,\n        \"length\": 7,\n        \"pic\": \"9(5)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"Branch order matters: age, plan-B large claim, missing preauth, long service, default approve.\",\n  \"Amounts are implied cents; 100000 means 1000.00.\"\n]",
+          "response": "{\"code\":\"from decimal import Decimal\\n\\ndef migrate(input_record: str) -> str:\\n    rec = (input_record or \\\"\\\").ljust(21)[:21]\\n\\n    def to_int(s: str) -> int:\\n        s = s.strip()\\n        return int(s) if s else 0\\n\\n    def to_amount(s: str) -> Decimal:\\n        s = s.strip()\\n        if not s:\\n            return Decimal('0.00')\\n        return Decimal(int(s)) / Decimal('100')\\n\\n    age = to_int(rec[6:9])\\n    plan_code = rec[9:10]\\n    service_days = to_int(rec[10:13])\\n    preauth_flag = rec[13:14]\\n    claim_amount = to_amount(rec[14:21])\\n\\n    decision = 'A'\\n    reason = 'OK'\\n\\n    if age < 18:\\n        decision = 'D'\\n        reason = 'A1'\\n    elif plan_code == 'B' and claim_amount > Decimal('1500.00'):\\n        decision = 'R'\\n        reason = 'B2'\\n    elif preauth_flag == 'N' and claim_amount > Decimal('1000.00'):\\n        decision = 'D'\\n        reason = 'P1'\\n    elif service_days > 30:\\n        decision = 'R'\\n        reason = 'L1'\\n\\n    return f\\\"{decision}{reason}\\\"\\n\"}",
+          "code_chars": 958
+        }
+      ],
+      "visible": {
+        "passed": 0,
+        "total": 3,
+        "pass_rate": 0.0,
+        "failures": [
+          {
+            "case_id": "visible_1",
+            "input_summary": "ordinary approved claim",
+            "expected_summary": "OUT-CLAIM-ID=CLM001, OUT-DECISION=A, OUT-REASON=OK",
+            "actual_summary": "length 3 output: 'AOK'",
+            "error": null
+          },
+          {
+            "case_id": "visible_2",
+            "input_summary": "missing preauth denial",
+            "expected_summary": "OUT-CLAIM-ID=CLM002, OUT-DECISION=D, OUT-REASON=P1",
+            "actual_summary": "length 3 output: 'DP1'",
+            "error": null
+          },
+          {
+            "case_id": "visible_3",
+            "input_summary": "long service review",
+            "expected_summary": "OUT-CLAIM-ID=CLM003, OUT-DECISION=R, OUT-REASON=L1",
+            "actual_summary": "length 3 output: 'RL1'",
+            "error": null
+          }
+        ]
+      },
+      "final": {
+        "public_score": 0.15,
+        "accepted": false,
+        "components": {
+          "hidden_correctness": 0.0,
+          "fresh_correctness": 0.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 0.0,
+          "anti_hardcoding": 0.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 0,
+        "hidden_total": 3,
+        "fresh_passed": 0,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "CLMELIG.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLMELIG.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. CLMELIG.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY CLAIM_REC.\n       PROCEDURE DIVISION.\n          EVALUATE TRUE\n             WHEN AGE < 18 MOVE 'D' TO DECISION MOVE 'A1' TO REASON\n             WHEN PLAN-CODE = 'B' AND CLAIM-AMOUNT > 1500.00 MOVE 'R' TO DECISION MOVE 'B2' TO REASON\n             WHEN PREAUTH-FLAG = 'N' AND CLAIM-AMOUNT > 1000.00 MOVE 'D' TO DECISION MOVE 'P1' TO REASON\n             WHEN SERVICE-DAYS > 030 MOVE 'R' TO DECISION MOVE 'L1' TO REASON\n             WHEN OTHER MOVE 'A' TO DECISION MOVE 'OK' TO REASON\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "CLAIM_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLAIM_REC.cpy",
+            "content": "       01  CLAIM-RECORD.\n           05 CLAIM-ID               PIC X(6).\n           05 AGE                    PIC 9(3).\n           05 PLAN-CODE              PIC X.\n           05 SERVICE-DAYS           PIC 9(3).\n           05 PREAUTH-FLAG           PIC X.\n           05 CLAIM-AMOUNT           PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "CLAIM_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLAIM_REC.cpy",
+            "record_name": "CLAIM-RECORD",
+            "total_width": 21,
+            "fields": [
+              {
+                "name": "CLAIM-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "AGE",
+                "start": 6,
+                "end": 9,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              },
+              {
+                "name": "PLAN-CODE",
+                "start": 9,
+                "end": 10,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "SERVICE-DAYS",
+                "start": 10,
+                "end": 13,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              },
+              {
+                "name": "PREAUTH-FLAG",
+                "start": 13,
+                "end": 14,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "CLAIM-AMOUNT",
+                "start": 14,
+                "end": 21,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "Branch order matters: age, plan-B large claim, missing preauth, long service, default approve.",
+              "Amounts are implied cents; 100000 means 1000.00."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 958
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.0,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 0,
+            "total": 3,
+            "pass_rate": 0.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": [
+              {
+                "case_id": "visible_1",
+                "input_summary": "ordinary approved claim",
+                "expected_summary": "OUT-CLAIM-ID=CLM001, OUT-DECISION=A, OUT-REASON=OK",
+                "actual_summary": "length 3 output: 'AOK'",
+                "error": null
+              },
+              {
+                "case_id": "visible_2",
+                "input_summary": "missing preauth denial",
+                "expected_summary": "OUT-CLAIM-ID=CLM002, OUT-DECISION=D, OUT-REASON=P1",
+                "actual_summary": "length 3 output: 'DP1'",
+                "error": null
+              },
+              {
+                "case_id": "visible_3",
+                "input_summary": "long service review",
+                "expected_summary": "OUT-CLAIM-ID=CLM003, OUT-DECISION=R, OUT-REASON=L1",
+                "actual_summary": "length 3 output: 'RL1'",
+                "error": null
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.15,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": false,
+            "episode_done": true,
+            "public_score": 0.15,
+            "components": {
+              "hidden_correctness": 0.0,
+              "fresh_correctness": 0.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 0.0,
+              "anti_hardcoding": 0.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 0,
+            "hidden_total": 3,
+            "fresh_passed": 0,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "azure-openai",
+      "task_id": "account_status_001",
+      "family_id": "account_status_level88",
+      "ticket": {
+        "task_id": "account_status_001",
+        "family_id": "account_status_level88",
+        "domain": "banking",
+        "ticket": "Migrate ACCTSTAT.cbl. Correctly interpret level-88 status names and preserve branch precedence for account servicing actions.",
+        "available_files": [
+          "ACCTSTAT.cbl"
+        ],
+        "available_copybooks": [
+          "ACCOUNT_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 19,
+        "output_width": 9,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "azure-openai",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"account_status_001\",\n  \"family_id\": \"account_status_level88\",\n  \"domain\": \"banking\",\n  \"ticket\": \"Migrate ACCTSTAT.cbl. Correctly interpret level-88 status names and preserve branch precedence for account servicing actions.\",\n  \"available_files\": [\n    \"ACCTSTAT.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"ACCOUNT_REC.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 19,\n  \"output_width\": 9,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"ACCTSTAT.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. ACCTSTAT.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY ACCOUNT_REC.\\n       PROCEDURE DIVISION.\\n          EVALUATE TRUE\\n             WHEN CLOSED-ACCT MOVE 'CL' TO CATEGORY MOVE 'N' TO ACTION\\n             WHEN FROZEN-ACCT MOVE 'FR' TO CATEGORY MOVE 'H' TO ACTION\\n             WHEN DAYS-PAST-DUE >= 090 MOVE 'DL' TO CATEGORY MOVE 'C' TO ACTION\\n             WHEN BALANCE < 0 MOVE 'OD' TO CATEGORY MOVE 'R' TO ACTION\\n             WHEN OTHER MOVE 'OK' TO CATEGORY MOVE 'N' TO ACTION\\n          END-EVALUATE\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"ACCOUNT_REC.cpy\": \"       01  ACCOUNT-RECORD.\\n           05 ACCOUNT-ID             PIC X(6).\\n           05 STATUS-CODE            PIC X.\\n              88 ACTIVE-ACCT         VALUE 'A'.\\n              88 FROZEN-ACCT         VALUE 'F'.\\n              88 CLOSED-ACCT         VALUE 'C'.\\n           05 BALANCE                PIC S9(6)V99 SIGN LEADING SEPARATE.\\n           05 DAYS-PAST-DUE          PIC 9(3).\\n\"\n}\n\nParsed layouts:\n{\n  \"ACCOUNT_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"ACCOUNT_REC.cpy\",\n    \"record_name\": \"ACCOUNT-RECORD\",\n    \"total_width\": 19,\n    \"fields\": [\n      {\n        \"name\": \"ACCOUNT-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"STATUS-CODE\",\n        \"start\": 6,\n        \"end\": 7,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\",\n        \"level_88\": {\n          \"ACTIVE-ACCT\": \"A\",\n          \"FROZEN-ACCT\": \"F\",\n          \"CLOSED-ACCT\": \"C\"\n        }\n      },\n      {\n        \"name\": \"BALANCE\",\n        \"start\": 7,\n        \"end\": 16,\n        \"length\": 9,\n        \"pic\": \"S9(6)V99 SIGN LEADING SEPARATE\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"DAYS-PAST-DUE\",\n        \"start\": 16,\n        \"end\": 19,\n        \"length\": 3,\n        \"pic\": \"9(3)\",\n        \"python_type\": \"int\"\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"CLOSED-ACCT and FROZEN-ACCT are level-88 names over STATUS-CODE.\",\n  \"Closed and frozen branches precede delinquency and overdrawn checks.\"\n]",
+          "response": "{\"code\":\"from decimal import Decimal, InvalidOperation\\n\\ndef migrate(input_record: str) -> str:\\n    rec = (input_record or \\\"\\\")\\n    rec = rec.ljust(19)[:19]\\n\\n    status_code = rec[6:7]\\n    balance_raw = rec[7:16].strip()\\n    days_raw = rec[16:19].strip()\\n\\n    try:\\n        balance = Decimal(balance_raw) if balance_raw else Decimal('0')\\n    except InvalidOperation:\\n        balance = Decimal('0')\\n\\n    try:\\n        days_past_due = int(days_raw) if days_raw else 0\\n    except ValueError:\\n        days_past_due = 0\\n\\n    category = 'OK'\\n    action = 'N'\\n\\n    # Preserve COBOL EVALUATE TRUE branch precedence.\\n    if status_code == 'C':  # CLOSED-ACCT\\n        category = 'CL'\\n        action = 'N'\\n    elif status_code == 'F':  # FROZEN-ACCT\\n        category = 'FR'\\n        action = 'H'\\n    elif days_past_due >= 90:\\n        category = 'DL'\\n        action = 'C'\\n    elif balance < 0:\\n        category = 'OD'\\n        action = 'R'\\n\\n    return f\\\"{category}{action}\\\"\\n\"}",
+          "code_chars": 947
+        }
+      ],
+      "visible": {
+        "passed": 0,
+        "total": 3,
+        "pass_rate": 0.0,
+        "failures": [
+          {
+            "case_id": "visible_1",
+            "input_summary": "active account in good standing",
+            "expected_summary": "OUT-ACCOUNT-ID=A00001, OUT-CATEGORY=OK, OUT-ACTION=N",
+            "actual_summary": "length 3 output: 'OKN'",
+            "error": null
+          },
+          {
+            "case_id": "visible_2",
+            "input_summary": "overdrawn active account",
+            "expected_summary": "OUT-ACCOUNT-ID=A00002, OUT-CATEGORY=OD, OUT-ACTION=R",
+            "actual_summary": "length 3 output: 'ODR'",
+            "error": null
+          },
+          {
+            "case_id": "visible_3",
+            "input_summary": "frozen account",
+            "expected_summary": "OUT-ACCOUNT-ID=A00003, OUT-CATEGORY=FR, OUT-ACTION=H",
+            "actual_summary": "length 3 output: 'FRH'",
+            "error": null
+          }
+        ]
+      },
+      "final": {
+        "public_score": 0.15,
+        "accepted": false,
+        "components": {
+          "hidden_correctness": 0.0,
+          "fresh_correctness": 0.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 0.0,
+          "anti_hardcoding": 0.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 0,
+        "hidden_total": 3,
+        "fresh_passed": 0,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "ACCTSTAT.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCTSTAT.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. ACCTSTAT.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY ACCOUNT_REC.\n       PROCEDURE DIVISION.\n          EVALUATE TRUE\n             WHEN CLOSED-ACCT MOVE 'CL' TO CATEGORY MOVE 'N' TO ACTION\n             WHEN FROZEN-ACCT MOVE 'FR' TO CATEGORY MOVE 'H' TO ACTION\n             WHEN DAYS-PAST-DUE >= 090 MOVE 'DL' TO CATEGORY MOVE 'C' TO ACTION\n             WHEN BALANCE < 0 MOVE 'OD' TO CATEGORY MOVE 'R' TO ACTION\n             WHEN OTHER MOVE 'OK' TO CATEGORY MOVE 'N' TO ACTION\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "ACCOUNT_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCOUNT_REC.cpy",
+            "content": "       01  ACCOUNT-RECORD.\n           05 ACCOUNT-ID             PIC X(6).\n           05 STATUS-CODE            PIC X.\n              88 ACTIVE-ACCT         VALUE 'A'.\n              88 FROZEN-ACCT         VALUE 'F'.\n              88 CLOSED-ACCT         VALUE 'C'.\n           05 BALANCE                PIC S9(6)V99 SIGN LEADING SEPARATE.\n           05 DAYS-PAST-DUE          PIC 9(3).\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "ACCOUNT_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCOUNT_REC.cpy",
+            "record_name": "ACCOUNT-RECORD",
+            "total_width": 19,
+            "fields": [
+              {
+                "name": "ACCOUNT-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "STATUS-CODE",
+                "start": 6,
+                "end": 7,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str",
+                "level_88": {
+                  "ACTIVE-ACCT": "A",
+                  "FROZEN-ACCT": "F",
+                  "CLOSED-ACCT": "C"
+                }
+              },
+              {
+                "name": "BALANCE",
+                "start": 7,
+                "end": 16,
+                "length": 9,
+                "pic": "S9(6)V99 SIGN LEADING SEPARATE",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "DAYS-PAST-DUE",
+                "start": 16,
+                "end": 19,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "CLOSED-ACCT and FROZEN-ACCT are level-88 names over STATUS-CODE.",
+              "Closed and frozen branches precede delinquency and overdrawn checks."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 947
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.0,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 0,
+            "total": 3,
+            "pass_rate": 0.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": [
+              {
+                "case_id": "visible_1",
+                "input_summary": "active account in good standing",
+                "expected_summary": "OUT-ACCOUNT-ID=A00001, OUT-CATEGORY=OK, OUT-ACTION=N",
+                "actual_summary": "length 3 output: 'OKN'",
+                "error": null
+              },
+              {
+                "case_id": "visible_2",
+                "input_summary": "overdrawn active account",
+                "expected_summary": "OUT-ACCOUNT-ID=A00002, OUT-CATEGORY=OD, OUT-ACTION=R",
+                "actual_summary": "length 3 output: 'ODR'",
+                "error": null
+              },
+              {
+                "case_id": "visible_3",
+                "input_summary": "frozen account",
+                "expected_summary": "OUT-ACCOUNT-ID=A00003, OUT-CATEGORY=FR, OUT-ACTION=H",
+                "actual_summary": "length 3 output: 'FRH'",
+                "error": null
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.15,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": false,
+            "episode_done": true,
+            "public_score": 0.15,
+            "components": {
+              "hidden_correctness": 0.0,
+              "fresh_correctness": 0.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 0.0,
+              "anti_hardcoding": 0.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 0,
+            "hidden_total": 3,
+            "fresh_passed": 0,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "azure-openai",
+      "task_id": "invoice_occurs_001",
+      "family_id": "invoice_occurs_totals",
+      "ticket": {
+        "task_id": "invoice_occurs_001",
+        "family_id": "invoice_occurs_totals",
+        "domain": "invoicing",
+        "ticket": "Migrate INVTOTAL.cbl. Preserve OCCURS stride parsing, item-count bounds, per-line tax rounding, and fixed-width total output.",
+        "available_files": [
+          "INVTOTAL.cbl"
+        ],
+        "available_copybooks": [
+          "INVOICE_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 44,
+        "output_width": 18,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "azure-openai",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"invoice_occurs_001\",\n  \"family_id\": \"invoice_occurs_totals\",\n  \"domain\": \"invoicing\",\n  \"ticket\": \"Migrate INVTOTAL.cbl. Preserve OCCURS stride parsing, item-count bounds, per-line tax rounding, and fixed-width total output.\",\n  \"available_files\": [\n    \"INVTOTAL.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"INVOICE_REC.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 44,\n  \"output_width\": 18,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"INVTOTAL.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. INVTOTAL.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY INVOICE_REC.\\n       PROCEDURE DIVISION.\\n          PERFORM VARYING IDX FROM 1 BY 1 UNTIL IDX > ITEM-COUNT\\n             COMPUTE LINE-AMOUNT ROUNDED = ITEM-QTY(IDX) * ITEM-PRICE(IDX)\\n             IF TAXABLE-FLAG(IDX) = 'Y'\\n                COMPUTE LINE-AMOUNT ROUNDED = LINE-AMOUNT * 1.075\\n             END-IF\\n             ADD LINE-AMOUNT TO INVOICE-TOTAL\\n          END-PERFORM\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"INVOICE_REC.cpy\": \"       01  INVOICE-RECORD.\\n           05 INVOICE-ID             PIC X(6).\\n           05 ITEM-COUNT             PIC 9(2).\\n           05 LINE-ITEM OCCURS 4 TIMES.\\n              10 ITEM-QTY            PIC 9(2).\\n              10 ITEM-PRICE          PIC 9(4)V99.\\n              10 TAXABLE-FLAG        PIC X.\\n\"\n}\n\nParsed layouts:\n{\n  \"INVOICE_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"INVOICE_REC.cpy\",\n    \"record_name\": \"INVOICE-RECORD\",\n    \"total_width\": 44,\n    \"fields\": [\n      {\n        \"name\": \"INVOICE-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"ITEM-COUNT\",\n        \"start\": 6,\n        \"end\": 8,\n        \"length\": 2,\n        \"pic\": \"9(2)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"LINE-ITEMS\",\n        \"start\": 8,\n        \"end\": 44,\n        \"length\": 36,\n        \"pic\": \"OCCURS 4 TIMES\",\n        \"python_type\": \"group\",\n        \"occurs\": 4,\n        \"stride\": 9\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"LINE-ITEM OCCURS 4 TIMES with a 9-byte stride.\",\n  \"Only the first ITEM-COUNT entries contribute to the total.\",\n  \"Taxable line amounts multiply by 1.075 and round half up per line.\"\n]",
+          "response": "{\"code\":\"from decimal import Decimal, ROUND_HALF_UP\\n\\ndef migrate(input_record: str) -> str:\\n    record = (input_record or \\\"\\\")[:44].ljust(44)\\n\\n    def parse_int(s: str) -> int:\\n        s = s.strip()\\n        return int(s) if s else 0\\n\\n    def parse_money(s: str) -> Decimal:\\n        s = s.strip()\\n        if not s:\\n            return Decimal('0')\\n        return Decimal(s) / Decimal('100')\\n\\n    item_count = parse_int(record[6:8])\\n    if item_count < 0:\\n        item_count = 0\\n    if item_count > 4:\\n        item_count = 4\\n\\n    total = Decimal('0')\\n    base = 8\\n    stride = 9\\n\\n    for idx in range(item_count):\\n        off = base + idx * stride\\n        qty = parse_int(record[off:off + 2])\\n        price = parse_money(record[off + 2:off + 8])\\n        taxable = record[off + 8:off + 9]\\n\\n        line_amount = (Decimal(qty) * price).quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)\\n        if taxable == 'Y':\\n            line_amount = (line_amount * Decimal('1.075')).quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)\\n        total += line_amount\\n\\n    cents = int(total.quantize(Decimal('0.01'), rounding=ROUND_HALF_UP) * 100)\\n    return f\\\"{cents:018d}\\\"\\n\"}",
+          "code_chars": 1143
+        }
+      ],
+      "visible": {
+        "passed": 0,
+        "total": 3,
+        "pass_rate": 0.0,
+        "failures": [
+          {
+            "case_id": "visible_1",
+            "input_summary": "two taxable and non-taxable lines",
+            "expected_summary": "OUT-INVOICE-ID=I00001, OUT-TOTAL=000036542, OUT-ITEM-COUNT=02, OUT-FLAG=L",
+            "actual_summary": "OUT-INVOICE-ID=000000, OUT-TOTAL=000000036, OUT-ITEM-COUNT=54, OUT-FLAG=2",
+            "error": null
+          },
+          {
+            "case_id": "visible_2",
+            "input_summary": "zero item invoice",
+            "expected_summary": "OUT-INVOICE-ID=I00002, OUT-TOTAL=000000000, OUT-ITEM-COUNT=00, OUT-FLAG=L",
+            "actual_summary": "OUT-INVOICE-ID=000000, OUT-TOTAL=000000000, OUT-ITEM-COUNT=00, OUT-FLAG=0",
+            "error": null
+          },
+          {
+            "case_id": "visible_3",
+            "input_summary": "four item high invoice",
+            "expected_summary": "OUT-INVOICE-ID=I00003, OUT-TOTAL=000351035, OUT-ITEM-COUNT=04, OUT-FLAG=H",
+            "actual_summary": "OUT-INVOICE-ID=000000, OUT-TOTAL=000000351, OUT-ITEM-COUNT=03, OUT-FLAG=5",
+            "error": null
+          }
+        ]
+      },
+      "final": {
+        "public_score": 0.23,
+        "accepted": false,
+        "components": {
+          "hidden_correctness": 0.0,
+          "fresh_correctness": 0.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 0.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 0,
+        "hidden_total": 3,
+        "fresh_passed": 0,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "INVTOTAL.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVTOTAL.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. INVTOTAL.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY INVOICE_REC.\n       PROCEDURE DIVISION.\n          PERFORM VARYING IDX FROM 1 BY 1 UNTIL IDX > ITEM-COUNT\n             COMPUTE LINE-AMOUNT ROUNDED = ITEM-QTY(IDX) * ITEM-PRICE(IDX)\n             IF TAXABLE-FLAG(IDX) = 'Y'\n                COMPUTE LINE-AMOUNT ROUNDED = LINE-AMOUNT * 1.075\n             END-IF\n             ADD LINE-AMOUNT TO INVOICE-TOTAL\n          END-PERFORM\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "INVOICE_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVOICE_REC.cpy",
+            "content": "       01  INVOICE-RECORD.\n           05 INVOICE-ID             PIC X(6).\n           05 ITEM-COUNT             PIC 9(2).\n           05 LINE-ITEM OCCURS 4 TIMES.\n              10 ITEM-QTY            PIC 9(2).\n              10 ITEM-PRICE          PIC 9(4)V99.\n              10 TAXABLE-FLAG        PIC X.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "INVOICE_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVOICE_REC.cpy",
+            "record_name": "INVOICE-RECORD",
+            "total_width": 44,
+            "fields": [
+              {
+                "name": "INVOICE-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "ITEM-COUNT",
+                "start": 6,
+                "end": 8,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "LINE-ITEMS",
+                "start": 8,
+                "end": 44,
+                "length": 36,
+                "pic": "OCCURS 4 TIMES",
+                "python_type": "group",
+                "occurs": 4,
+                "stride": 9
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "LINE-ITEM OCCURS 4 TIMES with a 9-byte stride.",
+              "Only the first ITEM-COUNT entries contribute to the total.",
+              "Taxable line amounts multiply by 1.075 and round half up per line."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 1143
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.0,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 0,
+            "total": 3,
+            "pass_rate": 0.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": [
+              {
+                "case_id": "visible_1",
+                "input_summary": "two taxable and non-taxable lines",
+                "expected_summary": "OUT-INVOICE-ID=I00001, OUT-TOTAL=000036542, OUT-ITEM-COUNT=02, OUT-FLAG=L",
+                "actual_summary": "OUT-INVOICE-ID=000000, OUT-TOTAL=000000036, OUT-ITEM-COUNT=54, OUT-FLAG=2",
+                "error": null
+              },
+              {
+                "case_id": "visible_2",
+                "input_summary": "zero item invoice",
+                "expected_summary": "OUT-INVOICE-ID=I00002, OUT-TOTAL=000000000, OUT-ITEM-COUNT=00, OUT-FLAG=L",
+                "actual_summary": "OUT-INVOICE-ID=000000, OUT-TOTAL=000000000, OUT-ITEM-COUNT=00, OUT-FLAG=0",
+                "error": null
+              },
+              {
+                "case_id": "visible_3",
+                "input_summary": "four item high invoice",
+                "expected_summary": "OUT-INVOICE-ID=I00003, OUT-TOTAL=000351035, OUT-ITEM-COUNT=04, OUT-FLAG=H",
+                "actual_summary": "OUT-INVOICE-ID=000000, OUT-TOTAL=000000351, OUT-ITEM-COUNT=03, OUT-FLAG=5",
+                "error": null
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.23,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": false,
+            "episode_done": true,
+            "public_score": 0.23,
+            "components": {
+              "hidden_correctness": 0.0,
+              "fresh_correctness": 0.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 0.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 0,
+            "hidden_total": 3,
+            "fresh_passed": 0,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "azure-openai",
+      "task_id": "date_normalization_001",
+      "family_id": "date_normalization",
+      "ticket": {
+        "task_id": "date_normalization_001",
+        "family_id": "date_normalization",
+        "domain": "claims and billing",
+        "ticket": "Migrate DATENORM.cbl. Preserve legacy YYMMDD century-window rules and invalid-date behavior instead of blindly using modern date parsing.",
+        "available_files": [
+          "DATENORM.cbl"
+        ],
+        "available_copybooks": [
+          "POLICY_DATE.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 21,
+        "output_width": 22,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "azure-openai",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"date_normalization_001\",\n  \"family_id\": \"date_normalization\",\n  \"domain\": \"claims and billing\",\n  \"ticket\": \"Migrate DATENORM.cbl. Preserve legacy YYMMDD century-window rules and invalid-date behavior instead of blindly using modern date parsing.\",\n  \"available_files\": [\n    \"DATENORM.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"POLICY_DATE.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 21,\n  \"output_width\": 22,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"DATENORM.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. DATENORM.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY POLICY_DATE.\\n       PROCEDURE DIVISION.\\n          IF RAW-YY >= WINDOW-YEAR\\n             MOVE 19 TO OUT-CENTURY\\n          ELSE\\n             MOVE 20 TO OUT-CENTURY\\n          END-IF\\n          PERFORM VALIDATE-MONTH-DAY\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"POLICY_DATE.cpy\": \"       01  POLICY-DATE-RECORD.\\n           05 POLICY-ID              PIC X(6).\\n           05 RAW-DATE               PIC 9(6).\\n           05 WINDOW-YEAR            PIC 9(2).\\n           05 AMOUNT                 PIC 9(5)V99.\\n\"\n}\n\nParsed layouts:\n{\n  \"POLICY_DATE.cpy\": {\n    \"ok\": true,\n    \"filename\": \"POLICY_DATE.cpy\",\n    \"record_name\": \"POLICY-DATE-RECORD\",\n    \"total_width\": 21,\n    \"fields\": [\n      {\n        \"name\": \"POLICY-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"RAW-DATE\",\n        \"start\": 6,\n        \"end\": 12,\n        \"length\": 6,\n        \"pic\": \"9(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"WINDOW-YEAR\",\n        \"start\": 12,\n        \"end\": 14,\n        \"length\": 2,\n        \"pic\": \"9(2)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"AMOUNT\",\n        \"start\": 14,\n        \"end\": 21,\n        \"length\": 7,\n        \"pic\": \"9(5)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"RAW-DATE is YYMMDD.\",\n  \"If YY >= WINDOW-YEAR, century is 19; otherwise century is 20.\",\n  \"Invalid dates output 00000000 and valid flag N; amount is preserved.\"\n]",
+          "response": "{\"code\":\"def migrate(input_record: str) -> str:\\n    \\\"\\\"\\\"Migrate DATENORM legacy COBOL routine.\\n\\n    Input layout (21 chars):\\n      0-5   POLICY-ID\\n      6-11  RAW-DATE   (YYMMDD)\\n      12-13 WINDOW-YEAR\\n      14-20 AMOUNT\\n\\n    Output width: 22 chars.\\n    Legacy behavior preserved:\\n      - Century window: if YY >= WINDOW-YEAR => 19xx else 20xx\\n      - Invalid dates become 00000000 and valid flag N\\n      - Amount is preserved\\n    \\\"\\\"\\\"\\n\\n    s = input_record or \\\"\\\"\\n    s = s.ljust(21)\\n\\n    policy_id = s[0:6]\\n    raw_date = s[6:12]\\n    window_year_str = s[12:14]\\n    amount = s[14:21]\\n\\n    yy = raw_date[0:2]\\n    mm = raw_date[2:4]\\n    dd = raw_date[4:6]\\n\\n    try:\\n        window_year = int(window_year_str)\\n    except ValueError:\\n        window_year = 0\\n\\n    try:\\n        yy_int = int(yy)\\n    except ValueError:\\n        yy_int = -1\\n\\n    if yy_int >= window_year:\\n        century = \\\"19\\\"\\n    else:\\n        century = \\\"20\\\"\\n\\n    # Validate month/day with leap-year awareness after century expansion.\\n    valid = True\\n    try:\\n        month = int(mm)\\n        day = int(dd)\\n        year = int(century + yy)\\n        if not (1 <= month <= 12):\\n            valid = False\\n        else:\\n            month_lengths = [31, 29 if (year % 400 == 0 or (year % 4 == 0 and year % 100 != 0)) else 28,\\n                             31, 30, 31, 30, 31, 31, 30, 31, 30, 31]\\n            if not (1 <= day <= month_lengths[month - 1]):\\n                valid = False\\n    except ValueError:\\n        valid = False\\n\\n    if valid:\\n        out_date = century + raw_date\\n        valid_flag = \\\"Y\\\"\\n    else:\\n        out_date = \\\"00000000\\\"\\n        valid_flag = \\\"N\\\"\\n\\n    # Output width 22: 8-char date + 1-char valid flag + 6-char policy id + 7-char amount = 22\\n    return f\\\"{out_date}{valid_flag}{policy_id}{amount}\\\"\\n\"}",
+          "code_chars": 1768
+        }
+      ],
+      "visible": {
+        "passed": 0,
+        "total": 3,
+        "pass_rate": 0.0,
+        "failures": [
+          {
+            "case_id": "visible_1",
+            "input_summary": "twentieth century date by window",
+            "expected_summary": "OUT-POLICY-ID=P00001, OUT-DATE=19991231, OUT-VALID=Y, OUT-AMOUNT=0012345",
+            "actual_summary": "OUT-POLICY-ID=199912, OUT-DATE=31YP0000, OUT-VALID=1, OUT-AMOUNT=0012345",
+            "error": null
+          },
+          {
+            "case_id": "visible_2",
+            "input_summary": "twenty-first century date by window",
+            "expected_summary": "OUT-POLICY-ID=P00002, OUT-DATE=20240229, OUT-VALID=Y, OUT-AMOUNT=0000100",
+            "actual_summary": "OUT-POLICY-ID=202402, OUT-DATE=29YP0000, OUT-VALID=2, OUT-AMOUNT=0000100",
+            "error": null
+          },
+          {
+            "case_id": "visible_3",
+            "input_summary": "invalid month",
+            "expected_summary": "OUT-POLICY-ID=P00003, OUT-DATE=00000000, OUT-VALID=N, OUT-AMOUNT=0000999",
+            "actual_summary": "OUT-POLICY-ID=000000, OUT-DATE=00NP0000, OUT-VALID=3, OUT-AMOUNT=0000999",
+            "error": null
+          }
+        ]
+      },
+      "final": {
+        "public_score": 0.15,
+        "accepted": false,
+        "components": {
+          "hidden_correctness": 0.0,
+          "fresh_correctness": 0.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 0.0,
+          "anti_hardcoding": 0.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 0,
+        "hidden_total": 3,
+        "fresh_passed": 0,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "DATENORM.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "DATENORM.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. DATENORM.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY POLICY_DATE.\n       PROCEDURE DIVISION.\n          IF RAW-YY >= WINDOW-YEAR\n             MOVE 19 TO OUT-CENTURY\n          ELSE\n             MOVE 20 TO OUT-CENTURY\n          END-IF\n          PERFORM VALIDATE-MONTH-DAY\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "POLICY_DATE.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "POLICY_DATE.cpy",
+            "content": "       01  POLICY-DATE-RECORD.\n           05 POLICY-ID              PIC X(6).\n           05 RAW-DATE               PIC 9(6).\n           05 WINDOW-YEAR            PIC 9(2).\n           05 AMOUNT                 PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "POLICY_DATE.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "POLICY_DATE.cpy",
+            "record_name": "POLICY-DATE-RECORD",
+            "total_width": 21,
+            "fields": [
+              {
+                "name": "POLICY-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "RAW-DATE",
+                "start": 6,
+                "end": 12,
+                "length": 6,
+                "pic": "9(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "WINDOW-YEAR",
+                "start": 12,
+                "end": 14,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "AMOUNT",
+                "start": 14,
+                "end": 21,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "RAW-DATE is YYMMDD.",
+              "If YY >= WINDOW-YEAR, century is 19; otherwise century is 20.",
+              "Invalid dates output 00000000 and valid flag N; amount is preserved."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 1768
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.0,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 0,
+            "total": 3,
+            "pass_rate": 0.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": [
+              {
+                "case_id": "visible_1",
+                "input_summary": "twentieth century date by window",
+                "expected_summary": "OUT-POLICY-ID=P00001, OUT-DATE=19991231, OUT-VALID=Y, OUT-AMOUNT=0012345",
+                "actual_summary": "OUT-POLICY-ID=199912, OUT-DATE=31YP0000, OUT-VALID=1, OUT-AMOUNT=0012345",
+                "error": null
+              },
+              {
+                "case_id": "visible_2",
+                "input_summary": "twenty-first century date by window",
+                "expected_summary": "OUT-POLICY-ID=P00002, OUT-DATE=20240229, OUT-VALID=Y, OUT-AMOUNT=0000100",
+                "actual_summary": "OUT-POLICY-ID=202402, OUT-DATE=29YP0000, OUT-VALID=2, OUT-AMOUNT=0000100",
+                "error": null
+              },
+              {
+                "case_id": "visible_3",
+                "input_summary": "invalid month",
+                "expected_summary": "OUT-POLICY-ID=P00003, OUT-DATE=00000000, OUT-VALID=N, OUT-AMOUNT=0000999",
+                "actual_summary": "OUT-POLICY-ID=000000, OUT-DATE=00NP0000, OUT-VALID=3, OUT-AMOUNT=0000999",
+                "error": null
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.15,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": false,
+            "episode_done": true,
+            "public_score": 0.15,
+            "components": {
+              "hidden_correctness": 0.0,
+              "fresh_correctness": 0.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 0.0,
+              "anti_hardcoding": 0.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 0,
+            "hidden_total": 3,
+            "fresh_passed": 0,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    }
+  ]
+}

outputs/evals/oracle_model_rollouts.json

@@ -0,0 +1,1448 @@
+{
+  "provider": "oracle-model",
+  "max_repairs": 0,
+  "task_count": 6,
+  "mean_public_score": 1.0,
+  "accepted_count": 6,
+  "trajectories": [
+    {
+      "policy": "oracle-model",
+      "task_id": "payroll_net_pay_001",
+      "family_id": "decimal_copybook_payroll",
+      "ticket": {
+        "task_id": "payroll_net_pay_001",
+        "family_id": "decimal_copybook_payroll",
+        "domain": "payroll",
+        "ticket": "Migrate PAYROLL.cbl while preserving copybook offsets, implied decimals, signed deductions, bonus condition names, and fixed-width ACH output.",
+        "available_files": [
+          "PAYROLL.cbl"
+        ],
+        "available_copybooks": [
+          "EMPLOYEE_PAY.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 5,
+        "input_width": 42,
+        "output_width": 28,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "oracle-model",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"payroll_net_pay_001\",\n  \"family_id\": \"decimal_copybook_payroll\",\n  \"domain\": \"payroll\",\n  \"ticket\": \"Migrate PAYROLL.cbl while preserving copybook offsets, implied decimals, signed deductions, bonus condition names, and fixed-width ACH output.\",\n  \"available_files\": [\n    \"PAYROLL.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"EMPLOYEE_PAY.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 5,\n  \"input_width\": 42,\n  \"output_width\": 28,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"PAYROLL.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. PAYROLLNET.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY EMPLOYEE_PAY.\\n       01 TAX-AMOUNT                 PIC 9(7)V99.\\n       01 NET-AMOUNT                 PIC 9(7)V99.\\n       01 PAY-CATEGORY               PIC X.\\n       PROCEDURE DIVISION.\\n           COMPUTE TAX-AMOUNT ROUNDED = GROSS-PAY * TAX-RATE\\n           COMPUTE NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS\\n           IF BONUS-ELIGIBLE ADD 50.00 TO NET-AMOUNT END-IF\\n           IF NET-AMOUNT < 0 MOVE 0 TO NET-AMOUNT END-IF\\n           EVALUATE TRUE\\n             WHEN NET-AMOUNT >= 5000.00 MOVE 'H' TO PAY-CATEGORY\\n             WHEN NET-AMOUNT >= 2500.00 MOVE 'M' TO PAY-CATEGORY\\n             WHEN OTHER MOVE 'L' TO PAY-CATEGORY\\n           END-EVALUATE\\n           GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"EMPLOYEE_PAY.cpy\": \"       01  EMPLOYEE-PAY-RECORD.\\n           05 EMP-ID                 PIC X(6).\\n           05 EMP-NAME               PIC X(12).\\n           05 GROSS-PAY              PIC 9(7)V99.\\n           05 TAX-RATE               PIC 9V999.\\n           05 DEDUCTIONS             PIC S9(5)V99 SIGN LEADING SEPARATE.\\n           05 BONUS-FLAG             PIC X(1).\\n              88 BONUS-ELIGIBLE      VALUE 'Y'.\\n              88 NO-BONUS            VALUE 'N'.\\n           05 FILLER                 PIC X(2).\\n\"\n}\n\nParsed layouts:\n{\n  \"EMPLOYEE_PAY.cpy\": {\n    \"ok\": true,\n    \"filename\": \"EMPLOYEE_PAY.cpy\",\n    \"record_name\": \"EMPLOYEE-PAY-RECORD\",\n    \"total_width\": 42,\n    \"fields\": [\n      {\n        \"name\": \"EMP-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"EMP-NAME\",\n        \"start\": 6,\n        \"end\": 18,\n        \"length\": 12,\n        \"pic\": \"X(12)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"GROSS-PAY\",\n        \"start\": 18,\n        \"end\": 27,\n        \"length\": 9,\n        \"pic\": \"9(7)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"TAX-RATE\",\n        \"start\": 27,\n        \"end\": 31,\n        \"length\": 4,\n        \"pic\": \"9V999\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 3\n      },\n      {\n        \"name\": \"DEDUCTIONS\",\n        \"start\": 31,\n        \"end\": 39,\n        \"length\": 8,\n        \"pic\": \"S9(5)V99 SIGN LEADING SEPARATE\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"BONUS-FLAG\",\n        \"start\": 39,\n        \"end\": 40,\n        \"length\": 1,\n        \"pic\": \"X(1)\",\n        \"python_type\": \"str\",\n        \"level_88\": {\n          \"BONUS-ELIGIBLE\": \"Y\",\n          \"NO-BONUS\": \"N\"\n        }\n      },\n      {\n        \"name\": \"FILLER\",\n        \"start\": 40,\n        \"end\": 42,\n        \"length\": 2,\n        \"pic\": \"X(2)\",\n        \"python_type\": \"str\"\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"Tax is rounded half up to cents after GROSS-PAY * TAX-RATE.\",\n  \"NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS.\",\n  \"BONUS-ELIGIBLE adds 50.00; negative net pay floors to zero.\",\n  \"Output net pay is a 9-digit zero-padded cents field.\"\n]",
+          "response": "{\"code\": \"from decimal import Decimal, ROUND_HALF_UP\\n\\n\\ndef migrate(input_record: str) -> str:\\n    emp_id = input_record[0:6]\\n    emp_name = input_record[6:18]\\n    gross = Decimal(int(input_record[18:27])) / Decimal(\\\"100\\\")\\n    tax_rate = Decimal(int(input_record[27:31])) / Decimal(\\\"1000\\\")\\n    raw_deductions = input_record[31:39]\\n    sign = -1 if raw_deductions[0] == \\\"-\\\" else 1\\n    deductions = Decimal(sign * int(raw_deductions[1:])) / Decimal(\\\"100\\\")\\n    tax = (gross * tax_rate).quantize(Decimal(\\\"0.01\\\"), rounding=ROUND_HALF_UP)\\n    net = gross - tax - deductions\\n    if input_record[39:40] == \\\"Y\\\":\\n        net += Decimal(\\\"50.00\\\")\\n    if net < 0:\\n        net = Decimal(\\\"0.00\\\")\\n    if net >= Decimal(\\\"5000.00\\\"):\\n        category = \\\"H\\\"\\n    elif net >= Decimal(\\\"2500.00\\\"):\\n        category = \\\"M\\\"\\n    else:\\n        category = \\\"L\\\"\\n    cents = int((net * 100).quantize(Decimal(\\\"1\\\"), rounding=ROUND_HALF_UP))\\n    return f\\\"{emp_id}{emp_name[:12].ljust(12)}{cents:09d}{category}\\\"\\n\"}",
+          "code_chars": 964
+        }
+      ],
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 5,
+        "hidden_total": 5,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "PAYROLL.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "PAYROLL.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. PAYROLLNET.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY EMPLOYEE_PAY.\n       01 TAX-AMOUNT                 PIC 9(7)V99.\n       01 NET-AMOUNT                 PIC 9(7)V99.\n       01 PAY-CATEGORY               PIC X.\n       PROCEDURE DIVISION.\n           COMPUTE TAX-AMOUNT ROUNDED = GROSS-PAY * TAX-RATE\n           COMPUTE NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS\n           IF BONUS-ELIGIBLE ADD 50.00 TO NET-AMOUNT END-IF\n           IF NET-AMOUNT < 0 MOVE 0 TO NET-AMOUNT END-IF\n           EVALUATE TRUE\n             WHEN NET-AMOUNT >= 5000.00 MOVE 'H' TO PAY-CATEGORY\n             WHEN NET-AMOUNT >= 2500.00 MOVE 'M' TO PAY-CATEGORY\n             WHEN OTHER MOVE 'L' TO PAY-CATEGORY\n           END-EVALUATE\n           GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "content": "       01  EMPLOYEE-PAY-RECORD.\n           05 EMP-ID                 PIC X(6).\n           05 EMP-NAME               PIC X(12).\n           05 GROSS-PAY              PIC 9(7)V99.\n           05 TAX-RATE               PIC 9V999.\n           05 DEDUCTIONS             PIC S9(5)V99 SIGN LEADING SEPARATE.\n           05 BONUS-FLAG             PIC X(1).\n              88 BONUS-ELIGIBLE      VALUE 'Y'.\n              88 NO-BONUS            VALUE 'N'.\n           05 FILLER                 PIC X(2).\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "record_name": "EMPLOYEE-PAY-RECORD",
+            "total_width": 42,
+            "fields": [
+              {
+                "name": "EMP-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "EMP-NAME",
+                "start": 6,
+                "end": 18,
+                "length": 12,
+                "pic": "X(12)",
+                "python_type": "str"
+              },
+              {
+                "name": "GROSS-PAY",
+                "start": 18,
+                "end": 27,
+                "length": 9,
+                "pic": "9(7)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "TAX-RATE",
+                "start": 27,
+                "end": 31,
+                "length": 4,
+                "pic": "9V999",
+                "python_type": "Decimal",
+                "scale": 3
+              },
+              {
+                "name": "DEDUCTIONS",
+                "start": 31,
+                "end": 39,
+                "length": 8,
+                "pic": "S9(5)V99 SIGN LEADING SEPARATE",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "BONUS-FLAG",
+                "start": 39,
+                "end": 40,
+                "length": 1,
+                "pic": "X(1)",
+                "python_type": "str",
+                "level_88": {
+                  "BONUS-ELIGIBLE": "Y",
+                  "NO-BONUS": "N"
+                }
+              },
+              {
+                "name": "FILLER",
+                "start": 40,
+                "end": 42,
+                "length": 2,
+                "pic": "X(2)",
+                "python_type": "str"
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "Tax is rounded half up to cents after GROSS-PAY * TAX-RATE.",
+              "NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS.",
+              "BONUS-ELIGIBLE adds 50.00; negative net pay floors to zero.",
+              "Output net pay is a 9-digit zero-padded cents field."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 964
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 5,
+            "hidden_total": 5,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle-model",
+      "task_id": "customer_format_001",
+      "family_id": "fixed_width_customer",
+      "ticket": {
+        "task_id": "customer_format_001",
+        "family_id": "fixed_width_customer",
+        "domain": "customer master data",
+        "ticket": "Migrate CUSTFMT.cbl. Preserve fixed-width name formatting, status mapping, ZIP leading zeros, and zero-padded balance output.",
+        "available_files": [
+          "CUSTFMT.cbl"
+        ],
+        "available_copybooks": [
+          "CUSTOMER_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 40,
+        "output_width": 41,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "oracle-model",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"customer_format_001\",\n  \"family_id\": \"fixed_width_customer\",\n  \"domain\": \"customer master data\",\n  \"ticket\": \"Migrate CUSTFMT.cbl. Preserve fixed-width name formatting, status mapping, ZIP leading zeros, and zero-padded balance output.\",\n  \"available_files\": [\n    \"CUSTFMT.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"CUSTOMER_REC.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 40,\n  \"output_width\": 41,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"CUSTFMT.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. CUSTFMT.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY CUSTOMER_REC.\\n       01 OUTPUT-RECORD.\\n          05 OUT-CUST-ID             PIC X(5).\\n          05 OUT-FULL-NAME           PIC X(22).\\n          05 OUT-ZIP-CODE            PIC 9(5).\\n          05 OUT-STATUS              PIC X.\\n          05 OUT-BALANCE             PIC 9(6)V99.\\n       PROCEDURE DIVISION.\\n          STRING LAST-NAME DELIMITED BY SIZE ', ' FIRST-NAME DELIMITED BY SIZE\\n             INTO OUT-FULL-NAME\\n          EVALUATE STATUS-CODE\\n             WHEN 'A' MOVE 'O' TO OUT-STATUS\\n             WHEN 'S' MOVE 'S' TO OUT-STATUS\\n             WHEN OTHER MOVE 'C' TO OUT-STATUS\\n          END-EVALUATE\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"CUSTOMER_REC.cpy\": \"       01  CUSTOMER-RECORD.\\n           05 CUST-ID                PIC X(5).\\n           05 FIRST-NAME             PIC X(10).\\n           05 LAST-NAME              PIC X(12).\\n           05 ZIP-CODE               PIC 9(5).\\n           05 STATUS-CODE            PIC X.\\n              88 ACTIVE-CUSTOMER     VALUE 'A'.\\n              88 SUSPENDED-CUSTOMER  VALUE 'S'.\\n              88 CLOSED-CUSTOMER     VALUE 'C'.\\n           05 BALANCE-DUE            PIC 9(5)V99.\\n\"\n}\n\nParsed layouts:\n{\n  \"CUSTOMER_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"CUSTOMER_REC.cpy\",\n    \"record_name\": \"CUSTOMER-RECORD\",\n    \"total_width\": 40,\n    \"fields\": [\n      {\n        \"name\": \"CUST-ID\",\n        \"start\": 0,\n        \"end\": 5,\n        \"length\": 5,\n        \"pic\": \"X(5)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"FIRST-NAME\",\n        \"start\": 5,\n        \"end\": 15,\n        \"length\": 10,\n        \"pic\": \"X(10)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"LAST-NAME\",\n        \"start\": 15,\n        \"end\": 27,\n        \"length\": 12,\n        \"pic\": \"X(12)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"ZIP-CODE\",\n        \"start\": 27,\n        \"end\": 32,\n        \"length\": 5,\n        \"pic\": \"9(5)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"STATUS-CODE\",\n        \"start\": 32,\n        \"end\": 33,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"BALANCE-DUE\",\n        \"start\": 33,\n        \"end\": 40,\n        \"length\": 7,\n        \"pic\": \"9(5)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"OUT-FULL-NAME is LAST-NAME, comma-space, FIRST-NAME, truncated/padded to 22.\",\n  \"A maps to O, S maps to S, anything else maps to C.\",\n  \"ZIP stays five digits and OUT-BALANCE is eight zero-padded digits.\"\n]",
+          "response": "{\"code\": \"def migrate(input_record: str) -> str:\\n    cust_id = input_record[0:5]\\n    first = input_record[5:15].rstrip()\\n    last = input_record[15:27].rstrip()\\n    zip_code = input_record[27:32]\\n    status = {\\\"A\\\": \\\"O\\\", \\\"S\\\": \\\"S\\\"}.get(input_record[32:33], \\\"C\\\")\\n    balance = int(input_record[33:40])\\n    full_name = f\\\"{last}, {first}\\\"[:22].ljust(22)\\n    return f\\\"{cust_id}{full_name}{zip_code}{status}{balance:08d}\\\"\\n\"}",
+          "code_chars": 405
+        }
+      ],
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "CUSTFMT.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTFMT.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. CUSTFMT.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY CUSTOMER_REC.\n       01 OUTPUT-RECORD.\n          05 OUT-CUST-ID             PIC X(5).\n          05 OUT-FULL-NAME           PIC X(22).\n          05 OUT-ZIP-CODE            PIC 9(5).\n          05 OUT-STATUS              PIC X.\n          05 OUT-BALANCE             PIC 9(6)V99.\n       PROCEDURE DIVISION.\n          STRING LAST-NAME DELIMITED BY SIZE ', ' FIRST-NAME DELIMITED BY SIZE\n             INTO OUT-FULL-NAME\n          EVALUATE STATUS-CODE\n             WHEN 'A' MOVE 'O' TO OUT-STATUS\n             WHEN 'S' MOVE 'S' TO OUT-STATUS\n             WHEN OTHER MOVE 'C' TO OUT-STATUS\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "CUSTOMER_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTOMER_REC.cpy",
+            "content": "       01  CUSTOMER-RECORD.\n           05 CUST-ID                PIC X(5).\n           05 FIRST-NAME             PIC X(10).\n           05 LAST-NAME              PIC X(12).\n           05 ZIP-CODE               PIC 9(5).\n           05 STATUS-CODE            PIC X.\n              88 ACTIVE-CUSTOMER     VALUE 'A'.\n              88 SUSPENDED-CUSTOMER  VALUE 'S'.\n              88 CLOSED-CUSTOMER     VALUE 'C'.\n           05 BALANCE-DUE            PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "CUSTOMER_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTOMER_REC.cpy",
+            "record_name": "CUSTOMER-RECORD",
+            "total_width": 40,
+            "fields": [
+              {
+                "name": "CUST-ID",
+                "start": 0,
+                "end": 5,
+                "length": 5,
+                "pic": "X(5)",
+                "python_type": "str"
+              },
+              {
+                "name": "FIRST-NAME",
+                "start": 5,
+                "end": 15,
+                "length": 10,
+                "pic": "X(10)",
+                "python_type": "str"
+              },
+              {
+                "name": "LAST-NAME",
+                "start": 15,
+                "end": 27,
+                "length": 12,
+                "pic": "X(12)",
+                "python_type": "str"
+              },
+              {
+                "name": "ZIP-CODE",
+                "start": 27,
+                "end": 32,
+                "length": 5,
+                "pic": "9(5)",
+                "python_type": "str"
+              },
+              {
+                "name": "STATUS-CODE",
+                "start": 32,
+                "end": 33,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "BALANCE-DUE",
+                "start": 33,
+                "end": 40,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "OUT-FULL-NAME is LAST-NAME, comma-space, FIRST-NAME, truncated/padded to 22.",
+              "A maps to O, S maps to S, anything else maps to C.",
+              "ZIP stays five digits and OUT-BALANCE is eight zero-padded digits."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 405
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle-model",
+      "task_id": "claims_eligibility_001",
+      "family_id": "claims_eligibility_branching",
+      "ticket": {
+        "task_id": "claims_eligibility_001",
+        "family_id": "claims_eligibility_branching",
+        "domain": "insurance claims",
+        "ticket": "Migrate CLMELIG.cbl. Preserve EVALUATE TRUE branch precedence for claim approvals, denials, and manual review reasons.",
+        "available_files": [
+          "CLMELIG.cbl"
+        ],
+        "available_copybooks": [
+          "CLAIM_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 21,
+        "output_width": 9,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "oracle-model",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"claims_eligibility_001\",\n  \"family_id\": \"claims_eligibility_branching\",\n  \"domain\": \"insurance claims\",\n  \"ticket\": \"Migrate CLMELIG.cbl. Preserve EVALUATE TRUE branch precedence for claim approvals, denials, and manual review reasons.\",\n  \"available_files\": [\n    \"CLMELIG.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"CLAIM_REC.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 21,\n  \"output_width\": 9,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"CLMELIG.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. CLMELIG.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY CLAIM_REC.\\n       PROCEDURE DIVISION.\\n          EVALUATE TRUE\\n             WHEN AGE < 18 MOVE 'D' TO DECISION MOVE 'A1' TO REASON\\n             WHEN PLAN-CODE = 'B' AND CLAIM-AMOUNT > 1500.00 MOVE 'R' TO DECISION MOVE 'B2' TO REASON\\n             WHEN PREAUTH-FLAG = 'N' AND CLAIM-AMOUNT > 1000.00 MOVE 'D' TO DECISION MOVE 'P1' TO REASON\\n             WHEN SERVICE-DAYS > 030 MOVE 'R' TO DECISION MOVE 'L1' TO REASON\\n             WHEN OTHER MOVE 'A' TO DECISION MOVE 'OK' TO REASON\\n          END-EVALUATE\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"CLAIM_REC.cpy\": \"       01  CLAIM-RECORD.\\n           05 CLAIM-ID               PIC X(6).\\n           05 AGE                    PIC 9(3).\\n           05 PLAN-CODE              PIC X.\\n           05 SERVICE-DAYS           PIC 9(3).\\n           05 PREAUTH-FLAG           PIC X.\\n           05 CLAIM-AMOUNT           PIC 9(5)V99.\\n\"\n}\n\nParsed layouts:\n{\n  \"CLAIM_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"CLAIM_REC.cpy\",\n    \"record_name\": \"CLAIM-RECORD\",\n    \"total_width\": 21,\n    \"fields\": [\n      {\n        \"name\": \"CLAIM-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"AGE\",\n        \"start\": 6,\n        \"end\": 9,\n        \"length\": 3,\n        \"pic\": \"9(3)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"PLAN-CODE\",\n        \"start\": 9,\n        \"end\": 10,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"SERVICE-DAYS\",\n        \"start\": 10,\n        \"end\": 13,\n        \"length\": 3,\n        \"pic\": \"9(3)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"PREAUTH-FLAG\",\n        \"start\": 13,\n        \"end\": 14,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"CLAIM-AMOUNT\",\n        \"start\": 14,\n        \"end\": 21,\n        \"length\": 7,\n        \"pic\": \"9(5)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"Branch order matters: age, plan-B large claim, missing preauth, long service, default approve.\",\n  \"Amounts are implied cents; 100000 means 1000.00.\"\n]",
+          "response": "{\"code\": \"def migrate(input_record: str) -> str:\\n    claim_id = input_record[0:6]\\n    age = int(input_record[6:9])\\n    plan = input_record[9:10]\\n    days = int(input_record[10:13])\\n    preauth = input_record[13:14]\\n    amount_cents = int(input_record[14:21])\\n    if age < 18:\\n        decision, reason = \\\"D\\\", \\\"A1\\\"\\n    elif plan == \\\"B\\\" and amount_cents > 150000:\\n        decision, reason = \\\"R\\\", \\\"B2\\\"\\n    elif preauth == \\\"N\\\" and amount_cents > 100000:\\n        decision, reason = \\\"D\\\", \\\"P1\\\"\\n    elif days > 30:\\n        decision, reason = \\\"R\\\", \\\"L1\\\"\\n    else:\\n        decision, reason = \\\"A\\\", \\\"OK\\\"\\n    return f\\\"{claim_id}{decision}{reason}\\\"\\n\"}",
+          "code_chars": 623
+        }
+      ],
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "CLMELIG.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLMELIG.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. CLMELIG.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY CLAIM_REC.\n       PROCEDURE DIVISION.\n          EVALUATE TRUE\n             WHEN AGE < 18 MOVE 'D' TO DECISION MOVE 'A1' TO REASON\n             WHEN PLAN-CODE = 'B' AND CLAIM-AMOUNT > 1500.00 MOVE 'R' TO DECISION MOVE 'B2' TO REASON\n             WHEN PREAUTH-FLAG = 'N' AND CLAIM-AMOUNT > 1000.00 MOVE 'D' TO DECISION MOVE 'P1' TO REASON\n             WHEN SERVICE-DAYS > 030 MOVE 'R' TO DECISION MOVE 'L1' TO REASON\n             WHEN OTHER MOVE 'A' TO DECISION MOVE 'OK' TO REASON\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "CLAIM_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLAIM_REC.cpy",
+            "content": "       01  CLAIM-RECORD.\n           05 CLAIM-ID               PIC X(6).\n           05 AGE                    PIC 9(3).\n           05 PLAN-CODE              PIC X.\n           05 SERVICE-DAYS           PIC 9(3).\n           05 PREAUTH-FLAG           PIC X.\n           05 CLAIM-AMOUNT           PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "CLAIM_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLAIM_REC.cpy",
+            "record_name": "CLAIM-RECORD",
+            "total_width": 21,
+            "fields": [
+              {
+                "name": "CLAIM-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "AGE",
+                "start": 6,
+                "end": 9,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              },
+              {
+                "name": "PLAN-CODE",
+                "start": 9,
+                "end": 10,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "SERVICE-DAYS",
+                "start": 10,
+                "end": 13,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              },
+              {
+                "name": "PREAUTH-FLAG",
+                "start": 13,
+                "end": 14,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "CLAIM-AMOUNT",
+                "start": 14,
+                "end": 21,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "Branch order matters: age, plan-B large claim, missing preauth, long service, default approve.",
+              "Amounts are implied cents; 100000 means 1000.00."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 623
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle-model",
+      "task_id": "account_status_001",
+      "family_id": "account_status_level88",
+      "ticket": {
+        "task_id": "account_status_001",
+        "family_id": "account_status_level88",
+        "domain": "banking",
+        "ticket": "Migrate ACCTSTAT.cbl. Correctly interpret level-88 status names and preserve branch precedence for account servicing actions.",
+        "available_files": [
+          "ACCTSTAT.cbl"
+        ],
+        "available_copybooks": [
+          "ACCOUNT_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 19,
+        "output_width": 9,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "oracle-model",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"account_status_001\",\n  \"family_id\": \"account_status_level88\",\n  \"domain\": \"banking\",\n  \"ticket\": \"Migrate ACCTSTAT.cbl. Correctly interpret level-88 status names and preserve branch precedence for account servicing actions.\",\n  \"available_files\": [\n    \"ACCTSTAT.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"ACCOUNT_REC.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 19,\n  \"output_width\": 9,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"ACCTSTAT.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. ACCTSTAT.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY ACCOUNT_REC.\\n       PROCEDURE DIVISION.\\n          EVALUATE TRUE\\n             WHEN CLOSED-ACCT MOVE 'CL' TO CATEGORY MOVE 'N' TO ACTION\\n             WHEN FROZEN-ACCT MOVE 'FR' TO CATEGORY MOVE 'H' TO ACTION\\n             WHEN DAYS-PAST-DUE >= 090 MOVE 'DL' TO CATEGORY MOVE 'C' TO ACTION\\n             WHEN BALANCE < 0 MOVE 'OD' TO CATEGORY MOVE 'R' TO ACTION\\n             WHEN OTHER MOVE 'OK' TO CATEGORY MOVE 'N' TO ACTION\\n          END-EVALUATE\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"ACCOUNT_REC.cpy\": \"       01  ACCOUNT-RECORD.\\n           05 ACCOUNT-ID             PIC X(6).\\n           05 STATUS-CODE            PIC X.\\n              88 ACTIVE-ACCT         VALUE 'A'.\\n              88 FROZEN-ACCT         VALUE 'F'.\\n              88 CLOSED-ACCT         VALUE 'C'.\\n           05 BALANCE                PIC S9(6)V99 SIGN LEADING SEPARATE.\\n           05 DAYS-PAST-DUE          PIC 9(3).\\n\"\n}\n\nParsed layouts:\n{\n  \"ACCOUNT_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"ACCOUNT_REC.cpy\",\n    \"record_name\": \"ACCOUNT-RECORD\",\n    \"total_width\": 19,\n    \"fields\": [\n      {\n        \"name\": \"ACCOUNT-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"STATUS-CODE\",\n        \"start\": 6,\n        \"end\": 7,\n        \"length\": 1,\n        \"pic\": \"X\",\n        \"python_type\": \"str\",\n        \"level_88\": {\n          \"ACTIVE-ACCT\": \"A\",\n          \"FROZEN-ACCT\": \"F\",\n          \"CLOSED-ACCT\": \"C\"\n        }\n      },\n      {\n        \"name\": \"BALANCE\",\n        \"start\": 7,\n        \"end\": 16,\n        \"length\": 9,\n        \"pic\": \"S9(6)V99 SIGN LEADING SEPARATE\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      },\n      {\n        \"name\": \"DAYS-PAST-DUE\",\n        \"start\": 16,\n        \"end\": 19,\n        \"length\": 3,\n        \"pic\": \"9(3)\",\n        \"python_type\": \"int\"\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"CLOSED-ACCT and FROZEN-ACCT are level-88 names over STATUS-CODE.\",\n  \"Closed and frozen branches precede delinquency and overdrawn checks.\"\n]",
+          "response": "{\"code\": \"def migrate(input_record: str) -> str:\\n    account_id = input_record[0:6]\\n    status = input_record[6:7]\\n    raw_balance = input_record[7:16]\\n    sign = -1 if raw_balance[0] == \\\"-\\\" else 1\\n    balance_cents = sign * int(raw_balance[1:])\\n    days = int(input_record[16:19])\\n    if status == \\\"C\\\":\\n        category, action = \\\"CL\\\", \\\"N\\\"\\n    elif status == \\\"F\\\":\\n        category, action = \\\"FR\\\", \\\"H\\\"\\n    elif days >= 90:\\n        category, action = \\\"DL\\\", \\\"C\\\"\\n    elif balance_cents < 0:\\n        category, action = \\\"OD\\\", \\\"R\\\"\\n    else:\\n        category, action = \\\"OK\\\", \\\"N\\\"\\n    return f\\\"{account_id}{category}{action}\\\"\\n\"}",
+          "code_chars": 607
+        }
+      ],
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "ACCTSTAT.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCTSTAT.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. ACCTSTAT.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY ACCOUNT_REC.\n       PROCEDURE DIVISION.\n          EVALUATE TRUE\n             WHEN CLOSED-ACCT MOVE 'CL' TO CATEGORY MOVE 'N' TO ACTION\n             WHEN FROZEN-ACCT MOVE 'FR' TO CATEGORY MOVE 'H' TO ACTION\n             WHEN DAYS-PAST-DUE >= 090 MOVE 'DL' TO CATEGORY MOVE 'C' TO ACTION\n             WHEN BALANCE < 0 MOVE 'OD' TO CATEGORY MOVE 'R' TO ACTION\n             WHEN OTHER MOVE 'OK' TO CATEGORY MOVE 'N' TO ACTION\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "ACCOUNT_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCOUNT_REC.cpy",
+            "content": "       01  ACCOUNT-RECORD.\n           05 ACCOUNT-ID             PIC X(6).\n           05 STATUS-CODE            PIC X.\n              88 ACTIVE-ACCT         VALUE 'A'.\n              88 FROZEN-ACCT         VALUE 'F'.\n              88 CLOSED-ACCT         VALUE 'C'.\n           05 BALANCE                PIC S9(6)V99 SIGN LEADING SEPARATE.\n           05 DAYS-PAST-DUE          PIC 9(3).\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "ACCOUNT_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCOUNT_REC.cpy",
+            "record_name": "ACCOUNT-RECORD",
+            "total_width": 19,
+            "fields": [
+              {
+                "name": "ACCOUNT-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "STATUS-CODE",
+                "start": 6,
+                "end": 7,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str",
+                "level_88": {
+                  "ACTIVE-ACCT": "A",
+                  "FROZEN-ACCT": "F",
+                  "CLOSED-ACCT": "C"
+                }
+              },
+              {
+                "name": "BALANCE",
+                "start": 7,
+                "end": 16,
+                "length": 9,
+                "pic": "S9(6)V99 SIGN LEADING SEPARATE",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "DAYS-PAST-DUE",
+                "start": 16,
+                "end": 19,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "CLOSED-ACCT and FROZEN-ACCT are level-88 names over STATUS-CODE.",
+              "Closed and frozen branches precede delinquency and overdrawn checks."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 607
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle-model",
+      "task_id": "invoice_occurs_001",
+      "family_id": "invoice_occurs_totals",
+      "ticket": {
+        "task_id": "invoice_occurs_001",
+        "family_id": "invoice_occurs_totals",
+        "domain": "invoicing",
+        "ticket": "Migrate INVTOTAL.cbl and TAXRATE.cbl. Preserve OCCURS stride parsing, item-count bounds, tax-code lookup behavior, and fixed-width total output.",
+        "available_files": [
+          "INVTOTAL.cbl",
+          "TAXRATE.cbl"
+        ],
+        "available_copybooks": [
+          "INVOICE_REC.cpy",
+          "TAX_CODE.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 4,
+        "input_width": 44,
+        "output_width": 18,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "oracle-model",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"invoice_occurs_001\",\n  \"family_id\": \"invoice_occurs_totals\",\n  \"domain\": \"invoicing\",\n  \"ticket\": \"Migrate INVTOTAL.cbl and TAXRATE.cbl. Preserve OCCURS stride parsing, item-count bounds, tax-code lookup behavior, and fixed-width total output.\",\n  \"available_files\": [\n    \"INVTOTAL.cbl\",\n    \"TAXRATE.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"INVOICE_REC.cpy\",\n    \"TAX_CODE.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 4,\n  \"input_width\": 44,\n  \"output_width\": 18,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"INVTOTAL.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. INVTOTAL.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY INVOICE_REC.\\n       COPY TAX_CODE.\\n       01 IDX                        PIC 9 VALUE 0.\\n       01 LINE-AMOUNT                PIC 9(7)V99 VALUE 0.\\n       01 TAX-AMOUNT                 PIC 9(7)V99 VALUE 0.\\n       01 TAX-PERCENT                PIC 9V9999 VALUE 0.\\n       01 INVOICE-TOTAL              PIC 9(7)V99 VALUE 0.\\n       PROCEDURE DIVISION.\\n          PERFORM VARYING IDX FROM 1 BY 1 UNTIL IDX > ITEM-COUNT\\n             COMPUTE LINE-AMOUNT ROUNDED = ITEM-QTY(IDX) * ITEM-PRICE(IDX)\\n             CALL 'TAXRATE' USING TAX-CODE(IDX) TAX-PERCENT\\n             COMPUTE TAX-AMOUNT ROUNDED = LINE-AMOUNT * TAX-PERCENT\\n             ADD TAX-AMOUNT TO LINE-AMOUNT\\n             ADD LINE-AMOUNT TO INVOICE-TOTAL\\n          END-PERFORM\\n          GOBACK.\\n\",\n  \"TAXRATE.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. TAXRATE.\\n       DATA DIVISION.\\n       LINKAGE SECTION.\\n       01 LK-TAX-CODE                PIC X.\\n       01 LK-TAX-PERCENT             PIC 9V9999.\\n       PROCEDURE DIVISION USING LK-TAX-CODE LK-TAX-PERCENT.\\n          EVALUATE LK-TAX-CODE\\n             WHEN 'S' MOVE 0.0725 TO LK-TAX-PERCENT\\n             WHEN 'R' MOVE 0.0250 TO LK-TAX-PERCENT\\n             WHEN 'L' MOVE 0.1000 TO LK-TAX-PERCENT\\n             WHEN OTHER MOVE 0.0000 TO LK-TAX-PERCENT\\n          END-EVALUATE\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"INVOICE_REC.cpy\": \"       01  INVOICE-RECORD.\\n           05 INVOICE-ID             PIC X(6).\\n           05 ITEM-COUNT             PIC 9(2).\\n           05 LINE-ITEM OCCURS 4 TIMES.\\n              10 ITEM-QTY            PIC 9(2).\\n              10 ITEM-PRICE          PIC 9(4)V99.\\n              10 TAX-CODE            PIC X.\\n\",\n  \"TAX_CODE.cpy\": \"       01  TAX-CODE-TABLE.\\n           05 TAX-CODE-ENTRY OCCURS 4 TIMES.\\n              10 TAX-CODE-KEY        PIC X.\\n              10 TAX-RATE            PIC 9V9999.\\n\"\n}\n\nParsed layouts:\n{\n  \"INVOICE_REC.cpy\": {\n    \"ok\": true,\n    \"filename\": \"INVOICE_REC.cpy\",\n    \"record_name\": \"INVOICE-RECORD\",\n    \"total_width\": 44,\n    \"fields\": [\n      {\n        \"name\": \"INVOICE-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"ITEM-COUNT\",\n        \"start\": 6,\n        \"end\": 8,\n        \"length\": 2,\n        \"pic\": \"9(2)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"LINE-ITEMS\",\n        \"start\": 8,\n        \"end\": 44,\n        \"length\": 36,\n        \"pic\": \"OCCURS 4 TIMES\",\n        \"python_type\": \"group\",\n        \"occurs\": 4,\n        \"stride\": 9\n      }\n    ]\n  },\n  \"TAX_CODE.cpy\": {\n    \"ok\": true,\n    \"filename\": \"TAX_CODE.cpy\",\n    \"record_name\": \"INVOICE-RECORD\",\n    \"total_width\": 44,\n    \"fields\": [\n      {\n        \"name\": \"INVOICE-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"ITEM-COUNT\",\n        \"start\": 6,\n        \"end\": 8,\n        \"length\": 2,\n        \"pic\": \"9(2)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"LINE-ITEMS\",\n        \"start\": 8,\n        \"end\": 44,\n        \"length\": 36,\n        \"pic\": \"OCCURS 4 TIMES\",\n        \"python_type\": \"group\",\n        \"occurs\": 4,\n        \"stride\": 9\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"LINE-ITEM OCCURS 4 TIMES with a 9-byte stride.\",\n  \"Only the first ITEM-COUNT entries contribute to the total.\",\n  \"Tax codes are resolved through the separate TAXRATE program; handle all code branches found there.\",\n  \"Keep the total, item count, and high/low flag in the fixed-width output layout.\"\n]",
+          "response": "{\"code\": \"from decimal import Decimal, ROUND_HALF_UP\\n\\n\\ndef migrate(input_record: str) -> str:\\n    invoice_id = input_record[0:6]\\n    count = min(int(input_record[6:8]), 4)\\n    tax_rates = {\\n        \\\"S\\\": Decimal(\\\"0.0725\\\"),\\n        \\\"R\\\": Decimal(\\\"0.0250\\\"),\\n        \\\"L\\\": Decimal(\\\"0.1000\\\"),\\n    }\\n    total = Decimal(\\\"0.00\\\")\\n    for idx in range(count):\\n        start = 8 + idx * 9\\n        qty = int(input_record[start:start + 2])\\n        price = Decimal(int(input_record[start + 2:start + 8])) / Decimal(\\\"100\\\")\\n        tax_code = input_record[start + 8:start + 9]\\n        line = (Decimal(qty) * price).quantize(Decimal(\\\"0.01\\\"), rounding=ROUND_HALF_UP)\\n        tax = (line * tax_rates.get(tax_code, Decimal(\\\"0.0000\\\"))).quantize(Decimal(\\\"0.01\\\"), rounding=ROUND_HALF_UP)\\n        line += tax\\n        total += line\\n    cents = int((total * 100).quantize(Decimal(\\\"1\\\"), rounding=ROUND_HALF_UP))\\n    flag = \\\"H\\\" if total >= Decimal(\\\"1000.00\\\") else \\\"L\\\"\\n    return f\\\"{invoice_id}{cents:09d}{count:02d}{flag}\\\"\\n\"}",
+          "code_chars": 984
+        }
+      ],
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 4,
+        "hidden_total": 4,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "INVTOTAL.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVTOTAL.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. INVTOTAL.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY INVOICE_REC.\n       COPY TAX_CODE.\n       01 IDX                        PIC 9 VALUE 0.\n       01 LINE-AMOUNT                PIC 9(7)V99 VALUE 0.\n       01 TAX-AMOUNT                 PIC 9(7)V99 VALUE 0.\n       01 TAX-PERCENT                PIC 9V9999 VALUE 0.\n       01 INVOICE-TOTAL              PIC 9(7)V99 VALUE 0.\n       PROCEDURE DIVISION.\n          PERFORM VARYING IDX FROM 1 BY 1 UNTIL IDX > ITEM-COUNT\n             COMPUTE LINE-AMOUNT ROUNDED = ITEM-QTY(IDX) * ITEM-PRICE(IDX)\n             CALL 'TAXRATE' USING TAX-CODE(IDX) TAX-PERCENT\n             COMPUTE TAX-AMOUNT ROUNDED = LINE-AMOUNT * TAX-PERCENT\n             ADD TAX-AMOUNT TO LINE-AMOUNT\n             ADD LINE-AMOUNT TO INVOICE-TOTAL\n          END-PERFORM\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "TAXRATE.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "TAXRATE.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. TAXRATE.\n       DATA DIVISION.\n       LINKAGE SECTION.\n       01 LK-TAX-CODE                PIC X.\n       01 LK-TAX-PERCENT             PIC 9V9999.\n       PROCEDURE DIVISION USING LK-TAX-CODE LK-TAX-PERCENT.\n          EVALUATE LK-TAX-CODE\n             WHEN 'S' MOVE 0.0725 TO LK-TAX-PERCENT\n             WHEN 'R' MOVE 0.0250 TO LK-TAX-PERCENT\n             WHEN 'L' MOVE 0.1000 TO LK-TAX-PERCENT\n             WHEN OTHER MOVE 0.0000 TO LK-TAX-PERCENT\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "INVOICE_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVOICE_REC.cpy",
+            "content": "       01  INVOICE-RECORD.\n           05 INVOICE-ID             PIC X(6).\n           05 ITEM-COUNT             PIC 9(2).\n           05 LINE-ITEM OCCURS 4 TIMES.\n              10 ITEM-QTY            PIC 9(2).\n              10 ITEM-PRICE          PIC 9(4)V99.\n              10 TAX-CODE            PIC X.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "INVOICE_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVOICE_REC.cpy",
+            "record_name": "INVOICE-RECORD",
+            "total_width": 44,
+            "fields": [
+              {
+                "name": "INVOICE-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "ITEM-COUNT",
+                "start": 6,
+                "end": 8,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "LINE-ITEMS",
+                "start": 8,
+                "end": 44,
+                "length": 36,
+                "pic": "OCCURS 4 TIMES",
+                "python_type": "group",
+                "occurs": 4,
+                "stride": 9
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "TAX_CODE.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "TAX_CODE.cpy",
+            "content": "       01  TAX-CODE-TABLE.\n           05 TAX-CODE-ENTRY OCCURS 4 TIMES.\n              10 TAX-CODE-KEY        PIC X.\n              10 TAX-RATE            PIC 9V9999.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "TAX_CODE.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "TAX_CODE.cpy",
+            "record_name": "INVOICE-RECORD",
+            "total_width": 44,
+            "fields": [
+              {
+                "name": "INVOICE-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "ITEM-COUNT",
+                "start": 6,
+                "end": 8,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "LINE-ITEMS",
+                "start": 8,
+                "end": 44,
+                "length": 36,
+                "pic": "OCCURS 4 TIMES",
+                "python_type": "group",
+                "occurs": 4,
+                "stride": 9
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "LINE-ITEM OCCURS 4 TIMES with a 9-byte stride.",
+              "Only the first ITEM-COUNT entries contribute to the total.",
+              "Tax codes are resolved through the separate TAXRATE program; handle all code branches found there.",
+              "Keep the total, item count, and high/low flag in the fixed-width output layout."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 984
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 4,
+            "hidden_total": 4,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle-model",
+      "task_id": "date_normalization_001",
+      "family_id": "date_normalization",
+      "ticket": {
+        "task_id": "date_normalization_001",
+        "family_id": "date_normalization",
+        "domain": "claims and billing",
+        "ticket": "Migrate DATENORM.cbl. Preserve legacy YYMMDD century-window rules and invalid-date behavior instead of blindly using modern date parsing.",
+        "available_files": [
+          "DATENORM.cbl"
+        ],
+        "available_copybooks": [
+          "POLICY_DATE.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 21,
+        "output_width": 22,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "model_turns": [
+        {
+          "provider": "oracle-model",
+          "prompt": "You are migrating a legacy COBOL routine into Python.\n\nReturn only JSON in this shape: {\"code\": \"...python source...\"}.\n\nThe Python source must define migrate(input_record: str) -> str.\n\nTicket:\n{\n  \"task_id\": \"date_normalization_001\",\n  \"family_id\": \"date_normalization\",\n  \"domain\": \"claims and billing\",\n  \"ticket\": \"Migrate DATENORM.cbl. Preserve legacy YYMMDD century-window rules and invalid-date behavior instead of blindly using modern date parsing.\",\n  \"available_files\": [\n    \"DATENORM.cbl\"\n  ],\n  \"available_copybooks\": [\n    \"POLICY_DATE.cpy\"\n  ],\n  \"expected_callable\": \"migrate(input_record: str) -> str\",\n  \"visible_tests\": 3,\n  \"hidden_tests\": 3,\n  \"input_width\": 21,\n  \"output_width\": 22,\n  \"max_steps\": 12,\n  \"allowed_tools\": [\n    \"read_cobol_file\",\n    \"read_copybook\",\n    \"parse_copybook_layout\",\n    \"inspect_business_rules\",\n    \"write_python_solution\",\n    \"run_visible_tests\",\n    \"inspect_diff\",\n    \"submit_final\"\n  ]\n}\n\nCOBOL files:\n{\n  \"DATENORM.cbl\": \"       IDENTIFICATION DIVISION.\\n       PROGRAM-ID. DATENORM.\\n       DATA DIVISION.\\n       WORKING-STORAGE SECTION.\\n       COPY POLICY_DATE.\\n       PROCEDURE DIVISION.\\n          IF RAW-YY >= WINDOW-YEAR\\n             MOVE 19 TO OUT-CENTURY\\n          ELSE\\n             MOVE 20 TO OUT-CENTURY\\n          END-IF\\n          PERFORM VALIDATE-MONTH-DAY\\n          GOBACK.\\n\"\n}\n\nCopybooks:\n{\n  \"POLICY_DATE.cpy\": \"       01  POLICY-DATE-RECORD.\\n           05 POLICY-ID              PIC X(6).\\n           05 RAW-DATE               PIC 9(6).\\n           05 WINDOW-YEAR            PIC 9(2).\\n           05 AMOUNT                 PIC 9(5)V99.\\n\"\n}\n\nParsed layouts:\n{\n  \"POLICY_DATE.cpy\": {\n    \"ok\": true,\n    \"filename\": \"POLICY_DATE.cpy\",\n    \"record_name\": \"POLICY-DATE-RECORD\",\n    \"total_width\": 21,\n    \"fields\": [\n      {\n        \"name\": \"POLICY-ID\",\n        \"start\": 0,\n        \"end\": 6,\n        \"length\": 6,\n        \"pic\": \"X(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"RAW-DATE\",\n        \"start\": 6,\n        \"end\": 12,\n        \"length\": 6,\n        \"pic\": \"9(6)\",\n        \"python_type\": \"str\"\n      },\n      {\n        \"name\": \"WINDOW-YEAR\",\n        \"start\": 12,\n        \"end\": 14,\n        \"length\": 2,\n        \"pic\": \"9(2)\",\n        \"python_type\": \"int\"\n      },\n      {\n        \"name\": \"AMOUNT\",\n        \"start\": 14,\n        \"end\": 21,\n        \"length\": 7,\n        \"pic\": \"9(5)V99\",\n        \"python_type\": \"Decimal\",\n        \"scale\": 2\n      }\n    ]\n  }\n}\n\nBusiness rules:\n[\n  \"RAW-DATE is YYMMDD.\",\n  \"If YY >= WINDOW-YEAR, century is 19; otherwise century is 20.\",\n  \"Invalid dates output 00000000 and valid flag N; amount is preserved.\"\n]",
+          "response": "{\"code\": \"def is_leap(year: int) -> bool:\\n    return year % 4 == 0 and (year % 100 != 0 or year % 400 == 0)\\n\\n\\ndef valid_date(year: int, month: int, day: int) -> bool:\\n    month_lengths = [31, 29 if is_leap(year) else 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]\\n    return 1 <= month <= 12 and 1 <= day <= month_lengths[month - 1]\\n\\n\\ndef migrate(input_record: str) -> str:\\n    policy_id = input_record[0:6]\\n    raw = input_record[6:12]\\n    window = int(input_record[12:14])\\n    amount = input_record[14:21]\\n    yy = int(raw[0:2])\\n    mm = int(raw[2:4])\\n    dd = int(raw[4:6])\\n    year = 1900 + yy if yy >= window else 2000 + yy\\n    normalized = f\\\"{year:04d}{mm:02d}{dd:02d}\\\" if valid_date(year, mm, dd) else \\\"00000000\\\"\\n    valid = \\\"Y\\\" if normalized != \\\"00000000\\\" else \\\"N\\\"\\n    return f\\\"{policy_id}{normalized}{valid}{amount}\\\"\\n\"}",
+          "code_chars": 813
+        }
+      ],
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "DATENORM.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "DATENORM.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. DATENORM.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY POLICY_DATE.\n       PROCEDURE DIVISION.\n          IF RAW-YY >= WINDOW-YEAR\n             MOVE 19 TO OUT-CENTURY\n          ELSE\n             MOVE 20 TO OUT-CENTURY\n          END-IF\n          PERFORM VALIDATE-MONTH-DAY\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "POLICY_DATE.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "POLICY_DATE.cpy",
+            "content": "       01  POLICY-DATE-RECORD.\n           05 POLICY-ID              PIC X(6).\n           05 RAW-DATE               PIC 9(6).\n           05 WINDOW-YEAR            PIC 9(2).\n           05 AMOUNT                 PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "POLICY_DATE.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "POLICY_DATE.cpy",
+            "record_name": "POLICY-DATE-RECORD",
+            "total_width": 21,
+            "fields": [
+              {
+                "name": "POLICY-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "RAW-DATE",
+                "start": 6,
+                "end": 12,
+                "length": 6,
+                "pic": "9(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "WINDOW-YEAR",
+                "start": 12,
+                "end": 14,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "AMOUNT",
+                "start": 14,
+                "end": 21,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "RAW-DATE is YYMMDD.",
+              "If YY >= WINDOW-YEAR, century is 19; otherwise century is 20.",
+              "Invalid dates output 00000000 and valid flag N; amount is preserved."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 813
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    }
+  ]
+}

outputs/evals/oracle_trajectories.json

@@ -0,0 +1,1399 @@
+{
+  "policy": "oracle",
+  "task_count": 6,
+  "mean_public_score": 1.0,
+  "accepted_count": 6,
+  "trajectories": [
+    {
+      "policy": "oracle",
+      "task_id": "payroll_net_pay_001",
+      "family_id": "decimal_copybook_payroll",
+      "ticket": {
+        "task_id": "payroll_net_pay_001",
+        "family_id": "decimal_copybook_payroll",
+        "domain": "payroll",
+        "ticket": "Migrate PAYROLL.cbl while preserving copybook offsets, implied decimals, signed deductions, bonus condition names, and fixed-width ACH output.",
+        "available_files": [
+          "PAYROLL.cbl"
+        ],
+        "available_copybooks": [
+          "EMPLOYEE_PAY.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 5,
+        "input_width": 42,
+        "output_width": 28,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 5,
+        "hidden_total": 5,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "PAYROLL.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "PAYROLL.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. PAYROLLNET.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY EMPLOYEE_PAY.\n       01 TAX-AMOUNT                 PIC 9(7)V99.\n       01 NET-AMOUNT                 PIC 9(7)V99.\n       01 PAY-CATEGORY               PIC X.\n       PROCEDURE DIVISION.\n           COMPUTE TAX-AMOUNT ROUNDED = GROSS-PAY * TAX-RATE\n           COMPUTE NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS\n           IF BONUS-ELIGIBLE ADD 50.00 TO NET-AMOUNT END-IF\n           IF NET-AMOUNT < 0 MOVE 0 TO NET-AMOUNT END-IF\n           EVALUATE TRUE\n             WHEN NET-AMOUNT >= 5000.00 MOVE 'H' TO PAY-CATEGORY\n             WHEN NET-AMOUNT >= 2500.00 MOVE 'M' TO PAY-CATEGORY\n             WHEN OTHER MOVE 'L' TO PAY-CATEGORY\n           END-EVALUATE\n           GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "content": "       01  EMPLOYEE-PAY-RECORD.\n           05 EMP-ID                 PIC X(6).\n           05 EMP-NAME               PIC X(12).\n           05 GROSS-PAY              PIC 9(7)V99.\n           05 TAX-RATE               PIC 9V999.\n           05 DEDUCTIONS             PIC S9(5)V99 SIGN LEADING SEPARATE.\n           05 BONUS-FLAG             PIC X(1).\n              88 BONUS-ELIGIBLE      VALUE 'Y'.\n              88 NO-BONUS            VALUE 'N'.\n           05 FILLER                 PIC X(2).\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "EMPLOYEE_PAY.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "EMPLOYEE_PAY.cpy",
+            "record_name": "EMPLOYEE-PAY-RECORD",
+            "total_width": 42,
+            "fields": [
+              {
+                "name": "EMP-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "EMP-NAME",
+                "start": 6,
+                "end": 18,
+                "length": 12,
+                "pic": "X(12)",
+                "python_type": "str"
+              },
+              {
+                "name": "GROSS-PAY",
+                "start": 18,
+                "end": 27,
+                "length": 9,
+                "pic": "9(7)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "TAX-RATE",
+                "start": 27,
+                "end": 31,
+                "length": 4,
+                "pic": "9V999",
+                "python_type": "Decimal",
+                "scale": 3
+              },
+              {
+                "name": "DEDUCTIONS",
+                "start": 31,
+                "end": 39,
+                "length": 8,
+                "pic": "S9(5)V99 SIGN LEADING SEPARATE",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "BONUS-FLAG",
+                "start": 39,
+                "end": 40,
+                "length": 1,
+                "pic": "X(1)",
+                "python_type": "str",
+                "level_88": {
+                  "BONUS-ELIGIBLE": "Y",
+                  "NO-BONUS": "N"
+                }
+              },
+              {
+                "name": "FILLER",
+                "start": 40,
+                "end": 42,
+                "length": 2,
+                "pic": "X(2)",
+                "python_type": "str"
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "Tax is rounded half up to cents after GROSS-PAY * TAX-RATE.",
+              "NET-AMOUNT = GROSS-PAY - TAX-AMOUNT - DEDUCTIONS.",
+              "BONUS-ELIGIBLE adds 50.00; negative net pay floors to zero.",
+              "Output net pay is a 9-digit zero-padded cents field."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 964
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 5,
+            "hidden_total": 5,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle",
+      "task_id": "customer_format_001",
+      "family_id": "fixed_width_customer",
+      "ticket": {
+        "task_id": "customer_format_001",
+        "family_id": "fixed_width_customer",
+        "domain": "customer master data",
+        "ticket": "Migrate CUSTFMT.cbl. Preserve fixed-width name formatting, status mapping, ZIP leading zeros, and zero-padded balance output.",
+        "available_files": [
+          "CUSTFMT.cbl"
+        ],
+        "available_copybooks": [
+          "CUSTOMER_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 40,
+        "output_width": 41,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "CUSTFMT.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTFMT.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. CUSTFMT.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY CUSTOMER_REC.\n       01 OUTPUT-RECORD.\n          05 OUT-CUST-ID             PIC X(5).\n          05 OUT-FULL-NAME           PIC X(22).\n          05 OUT-ZIP-CODE            PIC 9(5).\n          05 OUT-STATUS              PIC X.\n          05 OUT-BALANCE             PIC 9(6)V99.\n       PROCEDURE DIVISION.\n          STRING LAST-NAME DELIMITED BY SIZE ', ' FIRST-NAME DELIMITED BY SIZE\n             INTO OUT-FULL-NAME\n          EVALUATE STATUS-CODE\n             WHEN 'A' MOVE 'O' TO OUT-STATUS\n             WHEN 'S' MOVE 'S' TO OUT-STATUS\n             WHEN OTHER MOVE 'C' TO OUT-STATUS\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "CUSTOMER_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTOMER_REC.cpy",
+            "content": "       01  CUSTOMER-RECORD.\n           05 CUST-ID                PIC X(5).\n           05 FIRST-NAME             PIC X(10).\n           05 LAST-NAME              PIC X(12).\n           05 ZIP-CODE               PIC 9(5).\n           05 STATUS-CODE            PIC X.\n              88 ACTIVE-CUSTOMER     VALUE 'A'.\n              88 SUSPENDED-CUSTOMER  VALUE 'S'.\n              88 CLOSED-CUSTOMER     VALUE 'C'.\n           05 BALANCE-DUE            PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "CUSTOMER_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CUSTOMER_REC.cpy",
+            "record_name": "CUSTOMER-RECORD",
+            "total_width": 40,
+            "fields": [
+              {
+                "name": "CUST-ID",
+                "start": 0,
+                "end": 5,
+                "length": 5,
+                "pic": "X(5)",
+                "python_type": "str"
+              },
+              {
+                "name": "FIRST-NAME",
+                "start": 5,
+                "end": 15,
+                "length": 10,
+                "pic": "X(10)",
+                "python_type": "str"
+              },
+              {
+                "name": "LAST-NAME",
+                "start": 15,
+                "end": 27,
+                "length": 12,
+                "pic": "X(12)",
+                "python_type": "str"
+              },
+              {
+                "name": "ZIP-CODE",
+                "start": 27,
+                "end": 32,
+                "length": 5,
+                "pic": "9(5)",
+                "python_type": "str"
+              },
+              {
+                "name": "STATUS-CODE",
+                "start": 32,
+                "end": 33,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "BALANCE-DUE",
+                "start": 33,
+                "end": 40,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "OUT-FULL-NAME is LAST-NAME, comma-space, FIRST-NAME, truncated/padded to 22.",
+              "A maps to O, S maps to S, anything else maps to C.",
+              "ZIP stays five digits and OUT-BALANCE is eight zero-padded digits."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 405
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle",
+      "task_id": "claims_eligibility_001",
+      "family_id": "claims_eligibility_branching",
+      "ticket": {
+        "task_id": "claims_eligibility_001",
+        "family_id": "claims_eligibility_branching",
+        "domain": "insurance claims",
+        "ticket": "Migrate CLMELIG.cbl. Preserve EVALUATE TRUE branch precedence for claim approvals, denials, and manual review reasons.",
+        "available_files": [
+          "CLMELIG.cbl"
+        ],
+        "available_copybooks": [
+          "CLAIM_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 21,
+        "output_width": 9,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "CLMELIG.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLMELIG.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. CLMELIG.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY CLAIM_REC.\n       PROCEDURE DIVISION.\n          EVALUATE TRUE\n             WHEN AGE < 18 MOVE 'D' TO DECISION MOVE 'A1' TO REASON\n             WHEN PLAN-CODE = 'B' AND CLAIM-AMOUNT > 1500.00 MOVE 'R' TO DECISION MOVE 'B2' TO REASON\n             WHEN PREAUTH-FLAG = 'N' AND CLAIM-AMOUNT > 1000.00 MOVE 'D' TO DECISION MOVE 'P1' TO REASON\n             WHEN SERVICE-DAYS > 030 MOVE 'R' TO DECISION MOVE 'L1' TO REASON\n             WHEN OTHER MOVE 'A' TO DECISION MOVE 'OK' TO REASON\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "CLAIM_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLAIM_REC.cpy",
+            "content": "       01  CLAIM-RECORD.\n           05 CLAIM-ID               PIC X(6).\n           05 AGE                    PIC 9(3).\n           05 PLAN-CODE              PIC X.\n           05 SERVICE-DAYS           PIC 9(3).\n           05 PREAUTH-FLAG           PIC X.\n           05 CLAIM-AMOUNT           PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "CLAIM_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "CLAIM_REC.cpy",
+            "record_name": "CLAIM-RECORD",
+            "total_width": 21,
+            "fields": [
+              {
+                "name": "CLAIM-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "AGE",
+                "start": 6,
+                "end": 9,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              },
+              {
+                "name": "PLAN-CODE",
+                "start": 9,
+                "end": 10,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "SERVICE-DAYS",
+                "start": 10,
+                "end": 13,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              },
+              {
+                "name": "PREAUTH-FLAG",
+                "start": 13,
+                "end": 14,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str"
+              },
+              {
+                "name": "CLAIM-AMOUNT",
+                "start": 14,
+                "end": 21,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "Branch order matters: age, plan-B large claim, missing preauth, long service, default approve.",
+              "Amounts are implied cents; 100000 means 1000.00."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 623
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle",
+      "task_id": "account_status_001",
+      "family_id": "account_status_level88",
+      "ticket": {
+        "task_id": "account_status_001",
+        "family_id": "account_status_level88",
+        "domain": "banking",
+        "ticket": "Migrate ACCTSTAT.cbl. Correctly interpret level-88 status names and preserve branch precedence for account servicing actions.",
+        "available_files": [
+          "ACCTSTAT.cbl"
+        ],
+        "available_copybooks": [
+          "ACCOUNT_REC.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 19,
+        "output_width": 9,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "ACCTSTAT.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCTSTAT.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. ACCTSTAT.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY ACCOUNT_REC.\n       PROCEDURE DIVISION.\n          EVALUATE TRUE\n             WHEN CLOSED-ACCT MOVE 'CL' TO CATEGORY MOVE 'N' TO ACTION\n             WHEN FROZEN-ACCT MOVE 'FR' TO CATEGORY MOVE 'H' TO ACTION\n             WHEN DAYS-PAST-DUE >= 090 MOVE 'DL' TO CATEGORY MOVE 'C' TO ACTION\n             WHEN BALANCE < 0 MOVE 'OD' TO CATEGORY MOVE 'R' TO ACTION\n             WHEN OTHER MOVE 'OK' TO CATEGORY MOVE 'N' TO ACTION\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "ACCOUNT_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCOUNT_REC.cpy",
+            "content": "       01  ACCOUNT-RECORD.\n           05 ACCOUNT-ID             PIC X(6).\n           05 STATUS-CODE            PIC X.\n              88 ACTIVE-ACCT         VALUE 'A'.\n              88 FROZEN-ACCT         VALUE 'F'.\n              88 CLOSED-ACCT         VALUE 'C'.\n           05 BALANCE                PIC S9(6)V99 SIGN LEADING SEPARATE.\n           05 DAYS-PAST-DUE          PIC 9(3).\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "ACCOUNT_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "ACCOUNT_REC.cpy",
+            "record_name": "ACCOUNT-RECORD",
+            "total_width": 19,
+            "fields": [
+              {
+                "name": "ACCOUNT-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "STATUS-CODE",
+                "start": 6,
+                "end": 7,
+                "length": 1,
+                "pic": "X",
+                "python_type": "str",
+                "level_88": {
+                  "ACTIVE-ACCT": "A",
+                  "FROZEN-ACCT": "F",
+                  "CLOSED-ACCT": "C"
+                }
+              },
+              {
+                "name": "BALANCE",
+                "start": 7,
+                "end": 16,
+                "length": 9,
+                "pic": "S9(6)V99 SIGN LEADING SEPARATE",
+                "python_type": "Decimal",
+                "scale": 2
+              },
+              {
+                "name": "DAYS-PAST-DUE",
+                "start": 16,
+                "end": 19,
+                "length": 3,
+                "pic": "9(3)",
+                "python_type": "int"
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "CLOSED-ACCT and FROZEN-ACCT are level-88 names over STATUS-CODE.",
+              "Closed and frozen branches precede delinquency and overdrawn checks."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 607
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle",
+      "task_id": "invoice_occurs_001",
+      "family_id": "invoice_occurs_totals",
+      "ticket": {
+        "task_id": "invoice_occurs_001",
+        "family_id": "invoice_occurs_totals",
+        "domain": "invoicing",
+        "ticket": "Migrate INVTOTAL.cbl and TAXRATE.cbl. Preserve OCCURS stride parsing, item-count bounds, tax-code lookup behavior, and fixed-width total output.",
+        "available_files": [
+          "INVTOTAL.cbl",
+          "TAXRATE.cbl"
+        ],
+        "available_copybooks": [
+          "INVOICE_REC.cpy",
+          "TAX_CODE.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 4,
+        "input_width": 44,
+        "output_width": 18,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 4,
+        "hidden_total": 4,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "INVTOTAL.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVTOTAL.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. INVTOTAL.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY INVOICE_REC.\n       COPY TAX_CODE.\n       01 IDX                        PIC 9 VALUE 0.\n       01 LINE-AMOUNT                PIC 9(7)V99 VALUE 0.\n       01 TAX-AMOUNT                 PIC 9(7)V99 VALUE 0.\n       01 TAX-PERCENT                PIC 9V9999 VALUE 0.\n       01 INVOICE-TOTAL              PIC 9(7)V99 VALUE 0.\n       PROCEDURE DIVISION.\n          PERFORM VARYING IDX FROM 1 BY 1 UNTIL IDX > ITEM-COUNT\n             COMPUTE LINE-AMOUNT ROUNDED = ITEM-QTY(IDX) * ITEM-PRICE(IDX)\n             CALL 'TAXRATE' USING TAX-CODE(IDX) TAX-PERCENT\n             COMPUTE TAX-AMOUNT ROUNDED = LINE-AMOUNT * TAX-PERCENT\n             ADD TAX-AMOUNT TO LINE-AMOUNT\n             ADD LINE-AMOUNT TO INVOICE-TOTAL\n          END-PERFORM\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "TAXRATE.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "TAXRATE.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. TAXRATE.\n       DATA DIVISION.\n       LINKAGE SECTION.\n       01 LK-TAX-CODE                PIC X.\n       01 LK-TAX-PERCENT             PIC 9V9999.\n       PROCEDURE DIVISION USING LK-TAX-CODE LK-TAX-PERCENT.\n          EVALUATE LK-TAX-CODE\n             WHEN 'S' MOVE 0.0725 TO LK-TAX-PERCENT\n             WHEN 'R' MOVE 0.0250 TO LK-TAX-PERCENT\n             WHEN 'L' MOVE 0.1000 TO LK-TAX-PERCENT\n             WHEN OTHER MOVE 0.0000 TO LK-TAX-PERCENT\n          END-EVALUATE\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "INVOICE_REC.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVOICE_REC.cpy",
+            "content": "       01  INVOICE-RECORD.\n           05 INVOICE-ID             PIC X(6).\n           05 ITEM-COUNT             PIC 9(2).\n           05 LINE-ITEM OCCURS 4 TIMES.\n              10 ITEM-QTY            PIC 9(2).\n              10 ITEM-PRICE          PIC 9(4)V99.\n              10 TAX-CODE            PIC X.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "INVOICE_REC.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "INVOICE_REC.cpy",
+            "record_name": "INVOICE-RECORD",
+            "total_width": 44,
+            "fields": [
+              {
+                "name": "INVOICE-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "ITEM-COUNT",
+                "start": 6,
+                "end": 8,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "LINE-ITEMS",
+                "start": 8,
+                "end": 44,
+                "length": 36,
+                "pic": "OCCURS 4 TIMES",
+                "python_type": "group",
+                "occurs": 4,
+                "stride": 9
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "TAX_CODE.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "TAX_CODE.cpy",
+            "content": "       01  TAX-CODE-TABLE.\n           05 TAX-CODE-ENTRY OCCURS 4 TIMES.\n              10 TAX-CODE-KEY        PIC X.\n              10 TAX-RATE            PIC 9V9999.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "TAX_CODE.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "TAX_CODE.cpy",
+            "record_name": "INVOICE-RECORD",
+            "total_width": 44,
+            "fields": [
+              {
+                "name": "INVOICE-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "ITEM-COUNT",
+                "start": 6,
+                "end": 8,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "LINE-ITEMS",
+                "start": 8,
+                "end": 44,
+                "length": 36,
+                "pic": "OCCURS 4 TIMES",
+                "python_type": "group",
+                "occurs": 4,
+                "stride": 9
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "LINE-ITEM OCCURS 4 TIMES with a 9-byte stride.",
+              "Only the first ITEM-COUNT entries contribute to the total.",
+              "Tax codes are resolved through the separate TAXRATE program; handle all code branches found there.",
+              "Keep the total, item count, and high/low flag in the fixed-width output layout."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 984
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 4,
+            "hidden_total": 4,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    },
+    {
+      "policy": "oracle",
+      "task_id": "date_normalization_001",
+      "family_id": "date_normalization",
+      "ticket": {
+        "task_id": "date_normalization_001",
+        "family_id": "date_normalization",
+        "domain": "claims and billing",
+        "ticket": "Migrate DATENORM.cbl. Preserve legacy YYMMDD century-window rules and invalid-date behavior instead of blindly using modern date parsing.",
+        "available_files": [
+          "DATENORM.cbl"
+        ],
+        "available_copybooks": [
+          "POLICY_DATE.cpy"
+        ],
+        "expected_callable": "migrate(input_record: str) -> str",
+        "visible_tests": 3,
+        "hidden_tests": 3,
+        "input_width": 21,
+        "output_width": 22,
+        "max_steps": 12,
+        "allowed_tools": [
+          "read_cobol_file",
+          "read_copybook",
+          "parse_copybook_layout",
+          "inspect_business_rules",
+          "write_python_solution",
+          "run_visible_tests",
+          "inspect_diff",
+          "submit_final"
+        ]
+      },
+      "visible": {
+        "passed": 3,
+        "total": 3,
+        "pass_rate": 1.0,
+        "failures": []
+      },
+      "final": {
+        "public_score": 1.0,
+        "accepted": true,
+        "components": {
+          "hidden_correctness": 1.0,
+          "fresh_correctness": 1.0,
+          "interface_contract": 1.0,
+          "type_and_layout_fidelity": 1.0,
+          "anti_hardcoding": 1.0,
+          "safety": 1.0
+        },
+        "hidden_passed": 3,
+        "hidden_total": 3,
+        "fresh_passed": 6,
+        "fresh_total": 6
+      },
+      "steps": [
+        {
+          "tool_name": "read_cobol_file",
+          "arguments": {
+            "filename": "DATENORM.cbl"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "DATENORM.cbl",
+            "content": "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. DATENORM.\n       DATA DIVISION.\n       WORKING-STORAGE SECTION.\n       COPY POLICY_DATE.\n       PROCEDURE DIVISION.\n          IF RAW-YY >= WINDOW-YEAR\n             MOVE 19 TO OUT-CENTURY\n          ELSE\n             MOVE 20 TO OUT-CENTURY\n          END-IF\n          PERFORM VALIDATE-MONTH-DAY\n          GOBACK.\n",
+            "truncated": false
+          }
+        },
+        {
+          "tool_name": "read_copybook",
+          "arguments": {
+            "filename": "POLICY_DATE.cpy"
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "POLICY_DATE.cpy",
+            "content": "       01  POLICY-DATE-RECORD.\n           05 POLICY-ID              PIC X(6).\n           05 RAW-DATE               PIC 9(6).\n           05 WINDOW-YEAR            PIC 9(2).\n           05 AMOUNT                 PIC 9(5)V99.\n"
+          }
+        },
+        {
+          "tool_name": "parse_copybook_layout",
+          "arguments": {
+            "filename": "POLICY_DATE.cpy"
+          },
+          "reward": 0.03,
+          "done": false,
+          "result": {
+            "ok": true,
+            "filename": "POLICY_DATE.cpy",
+            "record_name": "POLICY-DATE-RECORD",
+            "total_width": 21,
+            "fields": [
+              {
+                "name": "POLICY-ID",
+                "start": 0,
+                "end": 6,
+                "length": 6,
+                "pic": "X(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "RAW-DATE",
+                "start": 6,
+                "end": 12,
+                "length": 6,
+                "pic": "9(6)",
+                "python_type": "str"
+              },
+              {
+                "name": "WINDOW-YEAR",
+                "start": 12,
+                "end": 14,
+                "length": 2,
+                "pic": "9(2)",
+                "python_type": "int"
+              },
+              {
+                "name": "AMOUNT",
+                "start": 14,
+                "end": 21,
+                "length": 7,
+                "pic": "9(5)V99",
+                "python_type": "Decimal",
+                "scale": 2
+              }
+            ]
+          }
+        },
+        {
+          "tool_name": "inspect_business_rules",
+          "arguments": {},
+          "reward": 0.01,
+          "done": false,
+          "result": {
+            "ok": true,
+            "rules": [
+              "RAW-DATE is YYMMDD.",
+              "If YY >= WINDOW-YEAR, century is 19; otherwise century is 20.",
+              "Invalid dates output 00000000 and valid flag N; amount is preserved."
+            ]
+          }
+        },
+        {
+          "tool_name": "write_python_solution",
+          "arguments": {
+            "code_chars": 813
+          },
+          "reward": 0.02,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "error": null
+          }
+        },
+        {
+          "tool_name": "run_visible_tests",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 0.05,
+          "done": false,
+          "result": {
+            "ok": true,
+            "draft_id": 1,
+            "passed": 3,
+            "total": 3,
+            "pass_rate": 1.0,
+            "syntax_ok": true,
+            "safety_ok": true,
+            "interface_ok": true,
+            "timed_out": false,
+            "error": null,
+            "failures": []
+          }
+        },
+        {
+          "tool_name": "submit_final",
+          "arguments": {
+            "draft_id": 1
+          },
+          "reward": 1.0,
+          "done": true,
+          "result": {
+            "ok": true,
+            "accepted": true,
+            "episode_done": true,
+            "public_score": 1.0,
+            "components": {
+              "hidden_correctness": 1.0,
+              "fresh_correctness": 1.0,
+              "interface_contract": 1.0,
+              "type_and_layout_fidelity": 1.0,
+              "anti_hardcoding": 1.0,
+              "safety": 1.0
+            },
+            "hidden_passed": 3,
+            "hidden_total": 3,
+            "fresh_passed": 6,
+            "fresh_total": 6,
+            "notes": "Hidden and fresh case details are not revealed to the agent."
+          }
+        }
+      ]
+    }
+  ]
+}

outputs/evals/score_summary.json

@@ -0,0 +1,131 @@
+{
+  "policies": {
+    "blank_width": {
+      "mean_public_score": 0.17666666666666667,
+      "accepted_count": 0,
+      "task_count": 6
+    },
+    "identity": {
+      "mean_public_score": 0.15,
+      "accepted_count": 0,
+      "task_count": 6
+    },
+    "oracle-model": {
+      "mean_public_score": 1.0,
+      "accepted_count": 6,
+      "task_count": 6,
+      "role": "oracle_sanity"
+    },
+    "base Qwen3-14B": {
+      "mean_public_score": 0.532,
+      "accepted_count": 2,
+      "task_count": 6,
+      "role": "base_model_before_sft"
+    },
+    "trained Qwen3-14B LoRA SFT": {
+      "mean_public_score": 0.7971166666666667,
+      "accepted_count": 4,
+      "task_count": 6,
+      "role": "trained_model_after_sft",
+      "model": "Qwen/Qwen3-14B",
+      "method": "Hugging Face TRL LoRA SFT"
+    }
+  },
+  "task_scores": {
+    "payroll_net_pay_001": {
+      "identity": 0.15,
+      "blank_width": 0.15,
+      "oracle_model": 1.0,
+      "base_qwen3_14b": 0.472
+    },
+    "customer_format_001": {
+      "identity": 0.15,
+      "blank_width": 0.15,
+      "oracle_model": 1.0,
+      "base_qwen3_14b": 0.34
+    },
+    "claims_eligibility_001": {
+      "identity": 0.15,
+      "blank_width": 0.23,
+      "oracle_model": 1.0,
+      "base_qwen3_14b": 1.0
+    },
+    "account_status_001": {
+      "identity": 0.15,
+      "blank_width": 0.23,
+      "oracle_model": 1.0,
+      "base_qwen3_14b": 1.0
+    },
+    "invoice_occurs_001": {
+      "identity": 0.15,
+      "blank_width": 0.15,
+      "oracle_model": 1.0,
+      "base_qwen3_14b": 0.15
+    },
+    "date_normalization_001": {
+      "identity": 0.15,
+      "blank_width": 0.15,
+      "oracle_model": 1.0,
+      "base_qwen3_14b": 0.23
+    }
+  },
+  "judge_table": [
+    {
+      "policy": "blank_width",
+      "role": null,
+      "mean_public_score": 0.17666666666666667,
+      "accepted_count": 0,
+      "task_count": 6
+    },
+    {
+      "policy": "identity",
+      "role": null,
+      "mean_public_score": 0.15,
+      "accepted_count": 0,
+      "task_count": 6
+    },
+    {
+      "policy": "oracle-model",
+      "role": "oracle_sanity",
+      "mean_public_score": 1.0,
+      "accepted_count": 6,
+      "task_count": 6
+    },
+    {
+      "policy": "base Qwen3-14B",
+      "role": "base_model_before_sft",
+      "mean_public_score": 0.532,
+      "accepted_count": 2,
+      "task_count": 6
+    },
+    {
+      "policy": "trained Qwen3-14B LoRA SFT",
+      "role": "trained_model_after_sft",
+      "mean_public_score": 0.7971166666666667,
+      "accepted_count": 4,
+      "task_count": 6
+    }
+  ],
+  "training_evidence": {
+    "model": "Qwen/Qwen3-14B",
+    "method": "Hugging Face TRL LoRA SFT",
+    "dataset_examples": 15,
+    "loss_first": 1.135,
+    "loss_last": 0.1924,
+    "mean_token_accuracy_first": 0.7938,
+    "mean_token_accuracy_last": 0.9483,
+    "base_mean_public_score": 0.532,
+    "trained_mean_public_score": 0.7971166666666667,
+    "base_accepted_count": 2,
+    "trained_accepted_count": 4,
+    "task_count": 6
+  },
+  "training_targets": [],
+  "evidence_notes": [
+    "missing rollout artifact: outputs/evals/azure_gpt54mini_zeroshot_rollouts.json",
+    "missing rollout artifact: outputs/evals/azure_gpt54mini_repair1_rollouts.json",
+    "historical artifact retained outside current score summary: outputs/evals/historical/azure_gpt54mini_payroll_probe.json",
+    "historical artifact retained outside current score summary: outputs/evals/historical/azure_gpt54mini_repair1_rollouts.json",
+    "historical artifact retained outside current score summary: outputs/evals/historical/azure_gpt54mini_zeroshot_rollouts.json"
+  ]
+}

outputs/training/hf_job_qwen3_14b_logs.txt

@@ -0,0 +1,797 @@
+Downloading numpy (15.9MiB)
+Downloading pillow (6.8MiB)
+Downloading nvidia-cuda-runtime (2.1MiB)
+Downloading networkx (2.0MiB)
+Downloading bitsandbytes (57.8MiB)
+Downloading cuda-bindings (6.0MiB)
+Downloading nvidia-cuda-nvrtc (86.0MiB)
+Downloading fonttools (4.8MiB)
+Downloading nvidia-cusparse (139.2MiB)
+Downloading nvidia-nvshmem-cu13 (57.6MiB)
+Downloading nvidia-nvjitlink (38.8MiB)
+Downloading nvidia-cusparselt-cu13 (162.0MiB)
+Downloading matplotlib (8.4MiB)
+Downloading torch (506.1MiB)
+Downloading pyarrow (46.6MiB)
+Downloading nvidia-cudnn-cu13 (349.1MiB)
+Downloading transformers (9.9MiB)
+Downloading hf-xet (4.0MiB)
+Downloading tokenizers (3.1MiB)
+Downloading sympy (6.0MiB)
+Downloading nvidia-cufft (204.2MiB)
+Downloading triton (179.5MiB)
+Downloading nvidia-cublas (403.5MiB)
+Downloading nvidia-cuda-cupti (10.2MiB)
+Downloading nvidia-curand (56.8MiB)
+Downloading pandas (10.4MiB)
+Downloading nvidia-nccl-cu13 (187.4MiB)
+Downloading setuptools (1.0MiB)
+Downloading nvidia-cusolver (191.6MiB)
+Downloading nvidia-cufile (1.2MiB)
+Downloading pygments (1.2MiB)
+Downloading kiwisolver (1.4MiB)
+Downloading aiohttp (1.7MiB)
+ Downloaded nvidia-cufile
+ Downloaded kiwisolver
+ Downloaded aiohttp
+ Downloaded nvidia-cuda-runtime
+ Downloaded pygments
+ Downloaded setuptools
+ Downloaded tokenizers
+ Downloaded networkx
+ Downloaded hf-xet
+ Downloaded fonttools
+ Downloaded cuda-bindings
+ Downloaded pillow
+ Downloaded matplotlib
+ Downloaded nvidia-cuda-cupti
+ Downloaded sympy
+ Downloaded numpy
+ Downloaded nvidia-nvjitlink
+ Downloaded pandas
+ Downloaded transformers
+ Downloaded nvidia-curand
+ Downloaded nvidia-nvshmem-cu13
+ Downloaded bitsandbytes
+ Downloaded nvidia-cuda-nvrtc
+ Downloaded pyarrow
+ Downloaded nvidia-cusparse
+ Downloaded nvidia-cusparselt-cu13
+ Downloaded nvidia-nccl-cu13
+ Downloaded nvidia-cusolver
+ Downloaded nvidia-cufft
+ Downloaded triton
+ Downloaded nvidia-cudnn-cu13
+ Downloaded nvidia-cublas
+ Downloaded torch
+Installed 84 packages in 450ms
+$ git clone https://github.com/ishangtxl/cobol-modernization-openenv.git /tmp/cobol-modernization-openenv
+Cloning into '/tmp/cobol-modernization-openenv'...
+$ /root/.cache/uv/builds-v0/.tmpQi62S5/bin/python -m pip install -U pip
+Requirement already satisfied: pip in /usr/local/lib/python3.12/site-packages (25.0.1)
+Collecting pip
+  Downloading pip-26.0.1-py3-none-any.whl.metadata (4.7 kB)
+Downloading pip-26.0.1-py3-none-any.whl (1.8 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 202.1 MB/s eta 0:00:00
+Installing collected packages: pip
+  Attempting uninstall: pip
+    Found existing installation: pip 25.0.1
+    Not uninstalling pip at /usr/local/lib/python3.12/site-packages, outside environment /root/.cache/uv/builds-v0/.tmpQi62S5
+    Can't uninstall 'pip'. No files were found to uninstall.
+Successfully installed pip-26.0.1
+$ /root/.cache/uv/builds-v0/.tmpQi62S5/bin/python -m pip install -r legacy_cobol_env/training/requirements-gpu.txt
+Requirement already satisfied: accelerate>=1.0.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from -r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (1.13.0)
+Requirement already satisfied: bitsandbytes>=0.43.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from -r legacy_cobol_env/training/requirements-gpu.txt (line 2)) (0.49.2)
+Requirement already satisfied: datasets>=3.0.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from -r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (4.8.4)
+Requirement already satisfied: peft>=0.13.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from -r legacy_cobol_env/training/requirements-gpu.txt (line 4)) (0.19.1)
+Requirement already satisfied: torch>=2.4.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from -r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (2.11.0)
+Requirement already satisfied: transformers>=4.45.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from -r legacy_cobol_env/training/requirements-gpu.txt (line 6)) (5.6.2)
+Requirement already satisfied: trl>=0.12.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from -r legacy_cobol_env/training/requirements-gpu.txt (line 7)) (1.2.0)
+Requirement already satisfied: numpy>=1.17 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (2.4.4)
+Requirement already satisfied: packaging>=20.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (26.2)
+Requirement already satisfied: psutil in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (7.2.2)
+Requirement already satisfied: pyyaml in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (6.0.3)
+Requirement already satisfied: huggingface_hub>=0.21.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (1.12.0)
+Requirement already satisfied: safetensors>=0.4.3 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (0.7.0)
+Requirement already satisfied: filelock in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (3.29.0)
+Requirement already satisfied: typing-extensions>=4.10.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (4.15.0)
+Requirement already satisfied: setuptools<82 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (81.0.0)
+Requirement already satisfied: sympy>=1.13.3 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (1.14.0)
+Requirement already satisfied: networkx>=2.5.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (3.6.1)
+Requirement already satisfied: jinja2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (3.1.6)
+Requirement already satisfied: fsspec>=0.8.5 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (2026.2.0)
+Requirement already satisfied: cuda-toolkit==13.0.2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.0.2)
+Requirement already satisfied: cuda-bindings<14,>=13.0.3 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.2.0)
+Requirement already satisfied: nvidia-cudnn-cu13==9.19.0.56 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (9.19.0.56)
+Requirement already satisfied: nvidia-cusparselt-cu13==0.8.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (0.8.0)
+Requirement already satisfied: nvidia-nccl-cu13==2.28.9 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (2.28.9)
+Requirement already satisfied: nvidia-nvshmem-cu13==3.4.5 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (3.4.5)
+Requirement already satisfied: triton==3.6.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (3.6.0)
+Requirement already satisfied: nvidia-cublas==13.1.0.3.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.1.0.3)
+Requirement already satisfied: nvidia-cuda-runtime==13.0.96.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.0.96)
+Requirement already satisfied: nvidia-cufft==12.0.0.61.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (12.0.0.61)
+Requirement already satisfied: nvidia-cufile==1.15.1.6.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (1.15.1.6)
+Requirement already satisfied: nvidia-cuda-cupti==13.0.85.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.0.85)
+Requirement already satisfied: nvidia-curand==10.4.0.35.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (10.4.0.35)
+Requirement already satisfied: nvidia-cusolver==12.0.4.66.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (12.0.4.66)
+Requirement already satisfied: nvidia-cusparse==12.6.3.3.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (12.6.3.3)
+Requirement already satisfied: nvidia-nvjitlink==13.0.88.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.0.88)
+Requirement already satisfied: nvidia-cuda-nvrtc==13.0.88.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.0.88)
+Requirement already satisfied: nvidia-nvtx==13.0.85.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == "Linux"->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (13.0.85)
+Requirement already satisfied: cuda-pathfinder~=1.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cuda-bindings<14,>=13.0.3->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (1.5.3)
+Requirement already satisfied: pyarrow>=21.0.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (24.0.0)
+Requirement already satisfied: dill<0.4.2,>=0.3.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (0.4.1)
+Requirement already satisfied: pandas in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (3.0.2)
+Requirement already satisfied: requests>=2.32.2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (2.33.1)
+Requirement already satisfied: httpx<1.0.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (0.28.1)
+Requirement already satisfied: tqdm>=4.66.3 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (4.67.3)
+Requirement already satisfied: xxhash in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (3.7.0)
+Requirement already satisfied: multiprocess<0.70.20 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (0.70.19)
+Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (3.13.5)
+Requirement already satisfied: anyio in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1.0.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (4.13.0)
+Requirement already satisfied: certifi in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1.0.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (2026.4.22)
+Requirement already satisfied: httpcore==1.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1.0.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (1.0.9)
+Requirement already satisfied: idna in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1.0.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (3.13)
+Requirement already satisfied: h11>=0.16 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpcore==1.*->httpx<1.0.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (0.16.0)
+Requirement already satisfied: hf-xet<2.0.0,>=1.4.3 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (1.4.3)
+Requirement already satisfied: typer in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (0.24.2)
+Requirement already satisfied: regex>=2025.10.22 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from transformers>=4.45.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 6)) (2026.4.4)
+Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from transformers>=4.45.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 6)) (0.22.2)
+Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (2.6.1)
+Requirement already satisfied: aiosignal>=1.4.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (1.4.0)
+Requirement already satisfied: attrs>=17.3.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (26.1.0)
+Requirement already satisfied: frozenlist>=1.1.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (1.8.0)
+Requirement already satisfied: multidict<7.0,>=4.5 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (6.7.1)
+Requirement already satisfied: propcache>=0.2.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (0.4.1)
+Requirement already satisfied: yarl<2.0,>=1.17.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (1.23.0)
+Requirement already satisfied: charset_normalizer<4,>=2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (3.4.7)
+Requirement already satisfied: urllib3<3,>=1.26 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (2.6.3)
+Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (1.3.0)
+Requirement already satisfied: MarkupSafe>=2.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from jinja2->torch>=2.4.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 5)) (3.0.3)
+Requirement already satisfied: python-dateutil>=2.8.2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from pandas->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (2.9.0.post0)
+Requirement already satisfied: six>=1.5 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets>=3.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 3)) (1.17.0)
+Requirement already satisfied: click>=8.2.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from typer->huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (8.3.3)
+Requirement already satisfied: shellingham>=1.3.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from typer->huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (1.5.4)
+Requirement already satisfied: rich>=12.3.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from typer->huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (15.0.0)
+Requirement already satisfied: annotated-doc>=0.0.2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from typer->huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (0.0.4)
+Requirement already satisfied: markdown-it-py>=2.2.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from rich>=12.3.0->typer->huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (4.0.0)
+Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from rich>=12.3.0->typer->huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (2.20.0)
+Requirement already satisfied: mdurl~=0.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=12.3.0->typer->huggingface_hub>=0.21.0->accelerate>=1.0.0->-r legacy_cobol_env/training/requirements-gpu.txt (line 1)) (0.1.2)
+$ /root/.cache/uv/builds-v0/.tmpQi62S5/bin/python -m pip install -e .
+Obtaining file:///tmp/cobol-modernization-openenv
+  Installing build dependencies: started
+  Installing build dependencies: finished with status 'done'
+  Checking if build backend supports build_editable: started
+  Checking if build backend supports build_editable: finished with status 'done'
+  Getting requirements to build editable: started
+  Getting requirements to build editable: finished with status 'done'
+  Preparing editable metadata (pyproject.toml): started
+  Preparing editable metadata (pyproject.toml): finished with status 'done'
+Collecting openenv-core>=0.2.3 (from openenv-legacy-cobol-submission==0.1.0)
+  Downloading openenv_core-0.2.3-py3-none-any.whl.metadata (19 kB)
+Collecting openai>=1.0 (from openenv-legacy-cobol-submission==0.1.0)
+  Downloading openai-2.32.0-py3-none-any.whl.metadata (31 kB)
+Requirement already satisfied: huggingface_hub>=1.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from openenv-legacy-cobol-submission==0.1.0) (1.12.0)
+Requirement already satisfied: filelock>=3.10.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (3.29.0)
+Requirement already satisfied: fsspec>=2023.5.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (2026.2.0)
+Requirement already satisfied: hf-xet<2.0.0,>=1.4.3 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (1.4.3)
+Requirement already satisfied: httpx<1,>=0.23.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (0.28.1)
+Requirement already satisfied: packaging>=20.9 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (26.2)
+Requirement already satisfied: pyyaml>=5.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (6.0.3)
+Requirement already satisfied: tqdm>=4.42.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (4.67.3)
+Requirement already satisfied: typer in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (0.24.2)
+Requirement already satisfied: typing-extensions>=4.1.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (4.15.0)
+Requirement already satisfied: anyio in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (4.13.0)
+Requirement already satisfied: certifi in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (2026.4.22)
+Requirement already satisfied: httpcore==1.* in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (1.0.9)
+Requirement already satisfied: idna in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (3.13)
+Requirement already satisfied: h11>=0.16 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (0.16.0)
+Collecting distro<2,>=1.7.0 (from openai>=1.0->openenv-legacy-cobol-submission==0.1.0)
+  Downloading distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)
+Collecting jiter<1,>=0.10.0 (from openai>=1.0->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jiter-0.14.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.2 kB)
+Collecting pydantic<3,>=1.9.0 (from openai>=1.0->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pydantic-2.13.3-py3-none-any.whl.metadata (108 kB)
+Collecting sniffio (from openai>=1.0->openenv-legacy-cobol-submission==0.1.0)
+  Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
+Collecting annotated-types>=0.6.0 (from pydantic<3,>=1.9.0->openai>=1.0->openenv-legacy-cobol-submission==0.1.0)
+  Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
+Collecting pydantic-core==2.46.3 (from pydantic<3,>=1.9.0->openai>=1.0->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pydantic_core-2.46.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
+Collecting typing-inspection>=0.4.2 (from pydantic<3,>=1.9.0->openai>=1.0->openenv-legacy-cobol-submission==0.1.0)
+  Downloading typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)
+Collecting fastapi>=0.104.0 (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading fastapi-0.136.1-py3-none-any.whl.metadata (28 kB)
+Collecting uvicorn>=0.24.0 (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading uvicorn-0.46.0-py3-none-any.whl.metadata (6.7 kB)
+Requirement already satisfied: requests>=2.25.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (2.33.1)
+Requirement already satisfied: rich>=13.0.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (15.0.0)
+Collecting tomli>=2.3.0 (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading tomli-2.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (10 kB)
+Collecting tomli-w>=1.2.0 (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading tomli_w-1.2.0-py3-none-any.whl.metadata (5.7 kB)
+Collecting websockets>=15.0.1 (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (6.8 kB)
+Collecting fastmcp>=3.0.0 (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading fastmcp-3.2.4-py3-none-any.whl.metadata (8.1 kB)
+Collecting gradio>=4.0.0 (from openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading gradio-6.13.0-py3-none-any.whl.metadata (17 kB)
+Collecting starlette>=0.46.0 (from fastapi>=0.104.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading starlette-1.0.0-py3-none-any.whl.metadata (6.3 kB)
+Requirement already satisfied: annotated-doc>=0.0.2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from fastapi>=0.104.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (0.0.4)
+Collecting authlib>=1.6.5 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading authlib-1.7.0-py2.py3-none-any.whl.metadata (10.0 kB)
+Collecting cyclopts>=4.0.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading cyclopts-4.11.0-py3-none-any.whl.metadata (12 kB)
+Collecting exceptiongroup>=1.2.2 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading exceptiongroup-1.3.1-py3-none-any.whl.metadata (6.7 kB)
+Collecting griffelib>=2.0.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading griffelib-2.0.2-py3-none-any.whl.metadata (1.3 kB)
+Collecting jsonref>=1.1.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jsonref-1.1.0-py3-none-any.whl.metadata (2.7 kB)
+Collecting jsonschema-path>=0.3.4 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jsonschema_path-0.4.5-py3-none-any.whl.metadata (5.9 kB)
+Collecting mcp<2.0,>=1.24.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading mcp-1.27.0-py3-none-any.whl.metadata (8.2 kB)
+Collecting openapi-pydantic>=0.5.1 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading openapi_pydantic-0.5.1-py3-none-any.whl.metadata (10 kB)
+Collecting opentelemetry-api>=1.20.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading opentelemetry_api-1.41.1-py3-none-any.whl.metadata (1.5 kB)
+Collecting platformdirs>=4.0.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading platformdirs-4.9.6-py3-none-any.whl.metadata (4.7 kB)
+Collecting py-key-value-aio<0.5.0,>=0.4.4 (from py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading py_key_value_aio-0.4.4-py3-none-any.whl.metadata (15 kB)
+Collecting pyperclip>=1.9.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pyperclip-1.11.0-py3-none-any.whl.metadata (2.4 kB)
+Collecting python-dotenv>=1.1.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading python_dotenv-1.2.2-py3-none-any.whl.metadata (27 kB)
+Collecting uncalled-for>=0.2.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading uncalled_for-0.3.1-py3-none-any.whl.metadata (2.9 kB)
+Collecting watchfiles>=1.0.0 (from fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading watchfiles-1.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
+Collecting httpx-sse>=0.4 (from mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading httpx_sse-0.4.3-py3-none-any.whl.metadata (9.7 kB)
+Collecting jsonschema>=4.20.0 (from mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jsonschema-4.26.0-py3-none-any.whl.metadata (7.6 kB)
+Collecting pydantic-settings>=2.5.2 (from mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pydantic_settings-2.14.0-py3-none-any.whl.metadata (3.4 kB)
+Collecting pyjwt>=2.10.1 (from pyjwt[crypto]>=2.10.1->mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pyjwt-2.12.1-py3-none-any.whl.metadata (4.1 kB)
+Collecting python-multipart>=0.0.9 (from mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading python_multipart-0.0.26-py3-none-any.whl.metadata (2.1 kB)
+Collecting sse-starlette>=1.6.1 (from mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading sse_starlette-3.3.4-py3-none-any.whl.metadata (14 kB)
+Collecting beartype>=0.20.0 (from py-key-value-aio<0.5.0,>=0.4.4->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading beartype-0.22.9-py3-none-any.whl.metadata (37 kB)
+Collecting aiofile>=3.5.0 (from py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading aiofile-3.9.0-py3-none-any.whl.metadata (14 kB)
+Collecting keyring>=25.6.0 (from py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading keyring-25.7.0-py3-none-any.whl.metadata (21 kB)
+Collecting cachetools>=5.0.0 (from py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading cachetools-7.0.6-py3-none-any.whl.metadata (5.9 kB)
+Collecting caio<0.10.0,>=0.9.0 (from aiofile>=3.5.0->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading caio-0.9.25-cp312-cp312-manylinux_2_34_x86_64.whl.metadata (3.3 kB)
+Collecting cryptography (from authlib>=1.6.5->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading cryptography-47.0.0-cp311-abi3-manylinux_2_34_x86_64.whl.metadata (4.5 kB)
+Collecting joserfc>=1.6.0 (from authlib>=1.6.5->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading joserfc-1.6.4-py3-none-any.whl.metadata (3.2 kB)
+Requirement already satisfied: attrs>=23.1.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from cyclopts>=4.0.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (26.1.0)
+Collecting docstring-parser<4.0,>=0.15 (from cyclopts>=4.0.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading docstring_parser-0.18.0-py3-none-any.whl.metadata (3.5 kB)
+Collecting rich-rst<2.0.0,>=1.3.1 (from cyclopts>=4.0.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading rich_rst-1.3.2-py3-none-any.whl.metadata (6.1 kB)
+Collecting docutils (from rich-rst<2.0.0,>=1.3.1->cyclopts>=4.0.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading docutils-0.22.4-py3-none-any.whl.metadata (15 kB)
+Collecting brotli>=1.1.0 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading brotli-1.2.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (6.1 kB)
+Collecting gradio-client==2.5.0 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading gradio_client-2.5.0-py3-none-any.whl.metadata (7.1 kB)
+Collecting groovy~=0.1 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading groovy-0.1.2-py3-none-any.whl.metadata (6.1 kB)
+Collecting hf-gradio<1.0,>=0.4.1 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading hf_gradio-0.4.1-py3-none-any.whl.metadata (428 bytes)
+Requirement already satisfied: jinja2<4.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (3.1.6)
+Requirement already satisfied: markupsafe<4.0,>=2.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (3.0.3)
+Requirement already satisfied: numpy<3.0,>=1.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (2.4.4)
+Collecting orjson~=3.0 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading orjson-3.11.8-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (41 kB)
+Requirement already satisfied: pandas<4.0,>=1.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (3.0.2)
+Requirement already satisfied: pillow<13.0,>=8.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (12.2.0)
+Collecting pydub<1.0 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pydub-0.25.1-py2.py3-none-any.whl.metadata (1.4 kB)
+Collecting pytz>=2017.2 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pytz-2026.1.post1-py2.py3-none-any.whl.metadata (22 kB)
+Collecting safehttpx<0.2.0,>=0.1.7 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading safehttpx-0.1.7-py3-none-any.whl.metadata (4.2 kB)
+Collecting semantic-version~=2.0 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading semantic_version-2.10.0-py2.py3-none-any.whl.metadata (9.7 kB)
+Collecting tomlkit<0.15.0,>=0.12.0 (from gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading tomlkit-0.14.0-py3-none-any.whl.metadata (2.8 kB)
+Requirement already satisfied: python-dateutil>=2.8.2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from pandas<4.0,>=1.0->gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (2.9.0.post0)
+Requirement already satisfied: click>=8.2.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from typer->huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (8.3.3)
+Requirement already satisfied: shellingham>=1.3.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from typer->huggingface_hub>=1.0->openenv-legacy-cobol-submission==0.1.0) (1.5.4)
+Collecting cffi>=2.0.0 (from cryptography->authlib>=1.6.5->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.6 kB)
+Collecting pycparser (from cffi>=2.0.0->cryptography->authlib>=1.6.5->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pycparser-3.0-py3-none-any.whl.metadata (8.2 kB)
+Collecting jsonschema-specifications>=2023.03.6 (from jsonschema>=4.20.0->mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jsonschema_specifications-2025.9.1-py3-none-any.whl.metadata (2.9 kB)
+Collecting referencing>=0.28.4 (from jsonschema>=4.20.0->mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading referencing-0.37.0-py3-none-any.whl.metadata (2.8 kB)
+Collecting rpds-py>=0.25.0 (from jsonschema>=4.20.0->mcp<2.0,>=1.24.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading rpds_py-0.30.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
+Collecting pathable<0.6.0,>=0.5.0 (from jsonschema-path>=0.3.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading pathable-0.5.0-py3-none-any.whl.metadata (5.9 kB)
+Collecting SecretStorage>=3.2 (from keyring>=25.6.0->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading secretstorage-3.5.0-py3-none-any.whl.metadata (4.0 kB)
+Collecting jeepney>=0.4.2 (from keyring>=25.6.0->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jeepney-0.9.0-py3-none-any.whl.metadata (1.2 kB)
+Collecting jaraco.classes (from keyring>=25.6.0->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB)
+Collecting jaraco.functools (from keyring>=25.6.0->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jaraco_functools-4.4.0-py3-none-any.whl.metadata (3.0 kB)
+Collecting jaraco.context (from keyring>=25.6.0->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading jaraco_context-6.1.2-py3-none-any.whl.metadata (4.2 kB)
+Collecting importlib-metadata<8.8.0,>=6.0 (from opentelemetry-api>=1.20.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading importlib_metadata-8.7.1-py3-none-any.whl.metadata (4.7 kB)
+Collecting zipp>=3.20 (from importlib-metadata<8.8.0,>=6.0->opentelemetry-api>=1.20.0->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading zipp-3.23.1-py3-none-any.whl.metadata (3.6 kB)
+Collecting email-validator>=2.0.0 (from pydantic[email]>=2.11.7->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading email_validator-2.3.0-py3-none-any.whl.metadata (26 kB)
+Collecting dnspython>=2.0.0 (from email-validator>=2.0.0->pydantic[email]>=2.11.7->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading dnspython-2.8.0-py3-none-any.whl.metadata (5.7 kB)
+Requirement already satisfied: six>=1.5 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas<4.0,>=1.0->gradio>=4.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (1.17.0)
+Requirement already satisfied: charset_normalizer<4,>=2 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from requests>=2.25.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (3.4.7)
+Requirement already satisfied: urllib3<3,>=1.26 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from requests>=2.25.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (2.6.3)
+Requirement already satisfied: markdown-it-py>=2.2.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from rich>=13.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (4.0.0)
+Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from rich>=13.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (2.20.0)
+Requirement already satisfied: mdurl~=0.1 in /root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=13.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0) (0.1.2)
+Collecting more-itertools (from jaraco.classes->keyring>=25.6.0->py-key-value-aio[filetree,keyring,memory]<0.5.0,>=0.4.4->fastmcp>=3.0.0->openenv-core>=0.2.3->openenv-legacy-cobol-submission==0.1.0)
+  Downloading more_itertools-11.0.2-py3-none-any.whl.metadata (41 kB)
+Downloading openai-2.32.0-py3-none-any.whl (1.2 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 189.5 MB/s  0:00:00
+Downloading distro-1.9.0-py3-none-any.whl (20 kB)
+Downloading jiter-0.14.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (353 kB)
+Downloading pydantic-2.13.3-py3-none-any.whl (471 kB)
+Downloading pydantic_core-2.46.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 139.5 MB/s  0:00:00
+Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
+Downloading openenv_core-0.2.3-py3-none-any.whl (174 kB)
+Downloading fastapi-0.136.1-py3-none-any.whl (117 kB)
+Downloading fastmcp-3.2.4-py3-none-any.whl (728 kB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 728.6/728.6 kB 150.8 MB/s  0:00:00
+Downloading mcp-1.27.0-py3-none-any.whl (215 kB)
+Downloading py_key_value_aio-0.4.4-py3-none-any.whl (152 kB)
+Downloading aiofile-3.9.0-py3-none-any.whl (19 kB)
+Downloading caio-0.9.25-cp312-cp312-manylinux_2_34_x86_64.whl (80 kB)
+Downloading authlib-1.7.0-py2.py3-none-any.whl (258 kB)
+Downloading beartype-0.22.9-py3-none-any.whl (1.3 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 319.7 MB/s  0:00:00
+Downloading cachetools-7.0.6-py3-none-any.whl (13 kB)
+Downloading cyclopts-4.11.0-py3-none-any.whl (208 kB)
+Downloading docstring_parser-0.18.0-py3-none-any.whl (22 kB)
+Downloading rich_rst-1.3.2-py3-none-any.whl (12 kB)
+Downloading exceptiongroup-1.3.1-py3-none-any.whl (16 kB)
+Downloading gradio-6.13.0-py3-none-any.whl (19.7 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.7/19.7 MB 279.0 MB/s  0:00:00
+Downloading gradio_client-2.5.0-py3-none-any.whl (59 kB)
+Downloading groovy-0.1.2-py3-none-any.whl (14 kB)
+Downloading hf_gradio-0.4.1-py3-none-any.whl (4.5 kB)
+Downloading orjson-3.11.8-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (133 kB)
+Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)
+Downloading python_multipart-0.0.26-py3-none-any.whl (28 kB)
+Downloading safehttpx-0.1.7-py3-none-any.whl (9.0 kB)
+Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
+Downloading starlette-1.0.0-py3-none-any.whl (72 kB)
+Downloading tomlkit-0.14.0-py3-none-any.whl (39 kB)
+Downloading brotli-1.2.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (1.4 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 224.7 MB/s  0:00:00
+Downloading griffelib-2.0.2-py3-none-any.whl (142 kB)
+Downloading httpx_sse-0.4.3-py3-none-any.whl (9.0 kB)
+Downloading joserfc-1.6.4-py3-none-any.whl (70 kB)
+Downloading cryptography-47.0.0-cp311-abi3-manylinux_2_34_x86_64.whl (4.7 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 360.4 MB/s  0:00:00
+Downloading cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (219 kB)
+Downloading jsonref-1.1.0-py3-none-any.whl (9.4 kB)
+Downloading jsonschema-4.26.0-py3-none-any.whl (90 kB)
+Downloading jsonschema_path-0.4.5-py3-none-any.whl (19 kB)
+Downloading pathable-0.5.0-py3-none-any.whl (16 kB)
+Downloading referencing-0.37.0-py3-none-any.whl (26 kB)
+Downloading jsonschema_specifications-2025.9.1-py3-none-any.whl (18 kB)
+Downloading keyring-25.7.0-py3-none-any.whl (39 kB)
+Downloading jeepney-0.9.0-py3-none-any.whl (49 kB)
+Downloading openapi_pydantic-0.5.1-py3-none-any.whl (96 kB)
+Downloading opentelemetry_api-1.41.1-py3-none-any.whl (69 kB)
+Downloading importlib_metadata-8.7.1-py3-none-any.whl (27 kB)
+Downloading platformdirs-4.9.6-py3-none-any.whl (21 kB)
+Downloading pydantic_settings-2.14.0-py3-none-any.whl (60 kB)
+Downloading email_validator-2.3.0-py3-none-any.whl (35 kB)
+Downloading dnspython-2.8.0-py3-none-any.whl (331 kB)
+Downloading pyjwt-2.12.1-py3-none-any.whl (29 kB)
+Downloading pyperclip-1.11.0-py3-none-any.whl (11 kB)
+Downloading python_dotenv-1.2.2-py3-none-any.whl (22 kB)
+Downloading pytz-2026.1.post1-py2.py3-none-any.whl (510 kB)
+Downloading rpds_py-0.30.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (394 kB)
+Downloading secretstorage-3.5.0-py3-none-any.whl (15 kB)
+Downloading sse_starlette-3.3.4-py3-none-any.whl (14 kB)
+Downloading tomli-2.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (253 kB)
+Downloading tomli_w-1.2.0-py3-none-any.whl (6.7 kB)
+Downloading typing_inspection-0.4.2-py3-none-any.whl (14 kB)
+Downloading uncalled_for-0.3.1-py3-none-any.whl (11 kB)
+Downloading uvicorn-0.46.0-py3-none-any.whl (70 kB)
+Downloading watchfiles-1.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (456 kB)
+Downloading websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (184 kB)
+Downloading zipp-3.23.1-py3-none-any.whl (10 kB)
+Downloading docutils-0.22.4-py3-none-any.whl (633 kB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 633.2/633.2 kB 180.9 MB/s  0:00:00
+Downloading jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
+Downloading jaraco_context-6.1.2-py3-none-any.whl (7.9 kB)
+Downloading jaraco_functools-4.4.0-py3-none-any.whl (10 kB)
+Downloading more_itertools-11.0.2-py3-none-any.whl (71 kB)
+Downloading pycparser-3.0-py3-none-any.whl (48 kB)
+Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)
+Building wheels for collected packages: openenv-legacy-cobol-submission
+  Building editable for openenv-legacy-cobol-submission (pyproject.toml): started
+  Building editable for openenv-legacy-cobol-submission (pyproject.toml): finished with status 'done'
+  Created wheel for openenv-legacy-cobol-submission: filename=openenv_legacy_cobol_submission-0.1.0-0.editable-py3-none-any.whl size=3473 sha256=9b4383c97de11d7e09ec00b3ef1baa48d457bd698dde2a5dadcf518844ea25b3
+  Stored in directory: /tmp/pip-ephem-wheel-cache-v0kbbpbx/wheels/7e/65/79/e562943dc6d9280fcc9daecb6a70ad61c29c4a1c92d641e1e1
+Successfully built openenv-legacy-cobol-submission
+Installing collected packages: pytz, pyperclip, pydub, brotli, zipp, websockets, uvicorn, uncalled-for, typing-inspection, tomlkit, tomli-w, tomli, sniffio, semantic-version, rpds-py, python-multipart, python-dotenv, pyjwt, pydantic-core, pycparser, platformdirs, pathable, orjson, more-itertools, jsonref, jiter, jeepney, jaraco.context, httpx-sse, groovy, griffelib, exceptiongroup, docutils, docstring-parser, dnspython, distro, caio, cachetools, beartype, annotated-types, watchfiles, starlette, referencing, pydantic, py-key-value-aio, jaraco.functools, jaraco.classes, importlib-metadata, email-validator, cffi, aiofile, sse-starlette, safehttpx, rich-rst, pydantic-settings, opentelemetry-api, openapi-pydantic, openai, jsonschema-specifications, jsonschema-path, fastapi, cryptography, SecretStorage, jsonschema, joserfc, cyclopts, mcp, keyring, gradio-client, authlib, hf-gradio, gradio, fastmcp, openenv-core, openenv-legacy-cobol-submission
+
+Successfully installed SecretStorage-3.5.0 aiofile-3.9.0 annotated-types-0.7.0 authlib-1.7.0 beartype-0.22.9 brotli-1.2.0 cachetools-7.0.6 caio-0.9.25 cffi-2.0.0 cryptography-47.0.0 cyclopts-4.11.0 distro-1.9.0 dnspython-2.8.0 docstring-parser-0.18.0 docutils-0.22.4 email-validator-2.3.0 exceptiongroup-1.3.1 fastapi-0.136.1 fastmcp-3.2.4 gradio-6.13.0 gradio-client-2.5.0 griffelib-2.0.2 groovy-0.1.2 hf-gradio-0.4.1 httpx-sse-0.4.3 importlib-metadata-8.7.1 jaraco.classes-3.4.0 jaraco.context-6.1.2 jaraco.functools-4.4.0 jeepney-0.9.0 jiter-0.14.0 joserfc-1.6.4 jsonref-1.1.0 jsonschema-4.26.0 jsonschema-path-0.4.5 jsonschema-specifications-2025.9.1 keyring-25.7.0 mcp-1.27.0 more-itertools-11.0.2 openai-2.32.0 openapi-pydantic-0.5.1 openenv-core-0.2.3 openenv-legacy-cobol-submission-0.1.0 opentelemetry-api-1.41.1 orjson-3.11.8 pathable-0.5.0 platformdirs-4.9.6 py-key-value-aio-0.4.4 pycparser-3.0 pydantic-2.13.3 pydantic-core-2.46.3 pydantic-settings-2.14.0 pydub-0.25.1 pyjwt-2.12.1 pyperclip-1.11.0 python-dotenv-1.2.2 python-multipart-0.0.26 pytz-2026.1.post1 referencing-0.37.0 rich-rst-1.3.2 rpds-py-0.30.0 safehttpx-0.1.7 semantic-version-2.10.0 sniffio-1.3.1 sse-starlette-3.3.4 starlette-1.0.0 tomli-2.4.1 tomli-w-1.2.0 tomlkit-0.14.0 typing-inspection-0.4.2 uncalled-for-0.3.1 uvicorn-0.46.0 watchfiles-1.1.1 websockets-16.0 zipp-3.23.1
+$ /root/.cache/uv/builds-v0/.tmpQi62S5/bin/python -m legacy_cobol_env.training.build_sft_dataset
+wrote 15 examples to /tmp/cobol-modernization-openenv/legacy_cobol_env/outputs/training/oracle_sft.jsonl
+$ /root/.cache/uv/builds-v0/.tmpQi62S5/bin/python -m legacy_cobol_env.eval.run_model_rollouts --provider local-transformers --max-repairs 2 --output legacy_cobol_env/outputs/evals/base_local_qwen3_14b_all_tasks.json
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+{
+  "provider": "local-transformers",
+  "task_count": 6,
+  "mean_public_score": 0.714,
+  "accepted_count": 3
+}
+$ /root/.cache/uv/builds-v0/.tmpQi62S5/bin/python -m legacy_cobol_env.training.train_sft --model-name Qwen/Qwen3-14B --max-seq-length 2048 --num-train-epochs 5 --gradient-accumulation-steps 4 --lora-rank 16 --lora-alpha 32 --output-dir legacy_cobol_env/outputs/training/sft-qwen3-14b-clean
+
+
+
+
+
+
+
+
+
+
+{
+  "sft_plan": {
+    "dataset": "legacy_cobol_env/outputs/training/oracle_sft.jsonl",
+    "output_dir": "legacy_cobol_env/outputs/training/sft-qwen3-14b-clean",
+    "model_name": "Qwen/Qwen3-14B",
+    "max_seq_length": 2048,
+    "num_train_epochs": 5.0,
+    "learning_rate": 0.0002,
+    "per_device_train_batch_size": 1,
+    "gradient_accumulation_steps": 4,
+    "lora_rank": 16,
+    "lora_alpha": 32,
+    "lora_dropout": 0.05,
+    "load_in_4bit": true,
+    "bf16": true,
+    "dataset_examples": 15,
+    "families": [
+      "account_status_level88",
+      "claims_eligibility_branching",
+      "date_normalization",
+      "decimal_copybook_payroll",
+      "fixed_width_customer",
+      "invoice_occurs_totals"
+    ],
+    "uses_lora": true,
+    "training_dependencies": [
+      "torch",
+      "transformers",
+      "datasets",
+      "peft",
+      "trl",
+      "accelerate"
+    ]
+  }
+}
+[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.
+
+
  0%|          | 0/20 [00:00<?, ?it/s]
+
  5%|▌         | 1/20 [00:06<02:07,  6.73s/it]
+
                                              
+{'loss': '1.135', 'grad_norm': '0.5547', 'learning_rate': '0.0002', 'entropy': '0.5034', 'num_tokens': '8192', 'mean_token_accuracy': '0.7938', 'epoch': '0.2667'}
+
  5%|▌         | 1/20 [00:06<02:07,  6.73s/it]
+
 10%|█         | 2/20 [00:12<01:48,  6.04s/it]
+
                                              
+{'loss': '1.044', 'grad_norm': '0.5586', 'learning_rate': '0.00019', 'entropy': '0.4721', 'num_tokens': '1.534e+04', 'mean_token_accuracy': '0.812', 'epoch': '0.5333'}
+
 10%|█         | 2/20 [00:12<01:48,  6.04s/it]
+
 15%|█▌        | 3/20 [00:17<01:38,  5.79s/it]
+{'loss': '0.8919', 'grad_norm': '0.5039', 'learning_rate': '0.00018', 'entropy': '0.5121', 'num_tokens': '2.241e+04', 'mean_token_accuracy': '0.8255', 'epoch': '0.8'}
+
                                              
+
 15%|█▌        | 3/20 [00:17<01:38,  5.79s/it]
+
 20%|██        | 4/20 [00:22<01:27,  5.45s/it]
+{'loss': '0.8053', 'grad_norm': '0.2988', 'learning_rate': '0.00017', 'entropy': '0.6193', 'num_tokens': '2.855e+04', 'mean_token_accuracy': '0.8334', 'epoch': '1'}
+
                                              
+
 20%|██        | 4/20 [00:22<01:27,  5.45s/it]
+
 25%|██▌       | 5/20 [00:29<01:27,  5.85s/it]
+
                                              
+{'loss': '0.7192', 'grad_norm': '0.2393', 'learning_rate': '0.00016', 'entropy': '0.6785', 'num_tokens': '3.637e+04', 'mean_token_accuracy': '0.8479', 'epoch': '1.267'}
+
 25%|██▌       | 5/20 [00:29<01:27,  5.85s/it]
+
 30%|███       | 6/20 [00:34<01:19,  5.70s/it]
+
                                              
+{'loss': '0.7439', 'grad_norm': '0.2422', 'learning_rate': '0.00015', 'entropy': '0.7696', 'num_tokens': '4.332e+04', 'mean_token_accuracy': '0.8394', 'epoch': '1.533'}
+
 30%|███       | 6/20 [00:34<01:19,  5.70s/it]
+
 35%|███▌      | 7/20 [00:40<01:16,  5.90s/it]
+
                                              
+{'loss': '0.6708', 'grad_norm': '0.2207', 'learning_rate': '0.00014', 'entropy': '0.7893', 'num_tokens': '5.151e+04', 'mean_token_accuracy': '0.8549', 'epoch': '1.8'}
+
 35%|███▌      | 7/20 [00:40<01:16,  5.90s/it]
+
 40%|████      | 8/20 [00:45<01:05,  5.43s/it]
+
                                              
+{'loss': '0.6371', 'grad_norm': '0.2441', 'learning_rate': '0.00013', 'entropy': '0.7936', 'num_tokens': '5.71e+04', 'mean_token_accuracy': '0.8601', 'epoch': '2'}
+
 40%|████      | 8/20 [00:45<01:05,  5.43s/it]
+
 45%|████▌     | 9/20 [00:52<01:06,  6.01s/it]
+
                                              
+{'loss': '0.5484', 'grad_norm': '0.2578', 'learning_rate': '0.00012', 'entropy': '0.727', 'num_tokens': '6.475e+04', 'mean_token_accuracy': '0.8757', 'epoch': '2.267'}
+
 45%|████▌     | 9/20 [00:52<01:06,  6.01s/it]
+
 50%|█████     | 10/20 [00:58<01:00,  6.01s/it]
+
                                               
+{'loss': '0.5467', 'grad_norm': '0.2637', 'learning_rate': '0.00011', 'entropy': '0.7256', 'num_tokens': '7.257e+04', 'mean_token_accuracy': '0.8786', 'epoch': '2.533'}
+
 50%|█████     | 10/20 [00:58<01:00,  6.01s/it]
+
 55%|█████▌    | 11/20 [01:04<00:52,  5.83s/it]
+
                                               
+{'loss': '0.4667', 'grad_norm': '0.252', 'learning_rate': '0.0001', 'entropy': '0.5945', 'num_tokens': '7.951e+04', 'mean_token_accuracy': '0.8898', 'epoch': '2.8'}
+
 55%|█████▌    | 11/20 [01:04<00:52,  5.83s/it]
+
 60%|██████    | 12/20 [01:08<00:44,  5.53s/it]
+
                                               
+{'loss': '0.4835', 'grad_norm': '0.2773', 'learning_rate': '9e-05', 'entropy': '0.6566', 'num_tokens': '8.566e+04', 'mean_token_accuracy': '0.8901', 'epoch': '3'}
+
 60%|██████    | 12/20 [01:08<00:44,  5.53s/it]
+
 65%|██████▌   | 13/20 [01:15<00:40,  5.82s/it]
+
                                               
+{'loss': '0.4375', 'grad_norm': '0.2461', 'learning_rate': '8e-05', 'entropy': '0.5715', 'num_tokens': '9.328e+04', 'mean_token_accuracy': '0.8986', 'epoch': '3.267'}
+
 65%|██████▌   | 13/20 [01:15<00:40,  5.82s/it]
+
 70%|███████   | 14/20 [01:21<00:35,  5.86s/it]
+{'loss': '0.4141', 'grad_norm': '0.3359', 'learning_rate': '7e-05', 'entropy': '0.537', 'num_tokens': '1.009e+05', 'mean_token_accuracy': '0.8993', 'epoch': '3.533'}
+
                                               
+
 70%|███████   | 14/20 [01:21<00:35,  5.86s/it]
+
 75%|███████▌  | 15/20 [01:27<00:29,  5.88s/it]
+
                                               
+{'loss': '0.3599', 'grad_norm': '0.2578', 'learning_rate': '6e-05', 'entropy': '0.4374', 'num_tokens': '1.084e+05', 'mean_token_accuracy': '0.9111', 'epoch': '3.8'}
+
 75%|███████▌  | 15/20 [01:27<00:29,  5.88s/it]
+
 80%|████████  | 16/20 [01:31<00:22,  5.50s/it]
+
                                               
+{'loss': '0.2364', 'grad_norm': '0.2754', 'learning_rate': '5e-05', 'entropy': '0.328', 'num_tokens': '1.142e+05', 'mean_token_accuracy': '0.938', 'epoch': '4'}
+
 80%|████████  | 16/20 [01:31<00:22,  5.50s/it]
+
 85%|████████▌ | 17/20 [01:38<00:17,  5.81s/it]
+
                                               
+{'loss': '0.3648', 'grad_norm': '0.2715', 'learning_rate': '4e-05', 'entropy': '0.4187', 'num_tokens': '1.217e+05', 'mean_token_accuracy': '0.9143', 'epoch': '4.267'}
+
 85%|████████▌ | 17/20 [01:38<00:17,  5.81s/it]
+
 90%|█████████ | 18/20 [01:44<00:12,  6.01s/it]
+
                                               
+{'loss': '0.2731', 'grad_norm': '0.252', 'learning_rate': '3e-05', 'entropy': '0.3601', 'num_tokens': '1.299e+05', 'mean_token_accuracy': '0.936', 'epoch': '4.533'}
+
 90%|█████████ | 18/20 [01:44<00:12,  6.01s/it]
+
 95%|█████████▌| 19/20 [01:50<00:05,  5.93s/it]
+
                                               
+{'loss': '0.2764', 'grad_norm': '0.249', 'learning_rate': '2e-05', 'entropy': '0.3304', 'num_tokens': '1.372e+05', 'mean_token_accuracy': '0.9352', 'epoch': '4.8'}
+
 95%|█████████▌| 19/20 [01:50<00:05,  5.93s/it]
+
+{'loss': '0.1924', 'grad_norm': '0.2734', 'learning_rate': '1e-05', 'entropy': '0.2629', 'num_tokens': '1.428e+05', 'mean_token_accuracy': '0.9483', 'epoch': '5'}
+
                                               
+
+
                                               
+{'train_runtime': '115.8', 'train_samples_per_second': '0.648', 'train_steps_per_second': '0.173', 'train_loss': '0.5623', 'epoch': '5'}
+
+$ /root/.cache/uv/builds-v0/.tmpQi62S5/bin/python -m legacy_cobol_env.eval.run_model_rollouts --provider local-transformers --max-repairs 2 --output legacy_cobol_env/outputs/evals/sft_local_qwen3_14b_all_tasks.json
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+
+
+
+
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+/root/.cache/uv/archive-v0/_zCg95lZEBKJ9MwE5wW11/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py:468: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+  torch._check_is_size(blocksize)
+{
+  "provider": "local-transformers",
+  "task_count": 6,
+  "mean_public_score": 0.7971166666666667,
+  "accepted_count": 4
+}
+Traceback (most recent call last):
+  File "//hf_qwen3_14b_train_job_h200.py", line 108, in <module>
+    state = json.loads((out / "trainer_state.json").read_text())
+                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/lib/python3.12/pathlib.py", line 1027, in read_text
+    with self.open(mode='r', encoding=encoding, errors=errors) as f:
+         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/lib/python3.12/pathlib.py", line 1013, in open
+    return io.open(self, mode, buffering, encoding, errors, newline)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+FileNotFoundError: [Errno 2] No such file or directory: 'legacy_cobol_env/outputs/training/sft-qwen3-14b-clean/trainer_state.json'

outputs/training/qwen3_14b_training_evidence_summary.json

@@ -0,0 +1,16 @@
+{
+  "model": "Qwen/Qwen3-14B",
+  "method": "Hugging Face TRL LoRA SFT",
+  "job_id": "69edbce1d70108f37acdfcdf",
+  "hardware": "HF Jobs l40sx1",
+  "dataset_examples": 15,
+  "loss_first": 1.135,
+  "loss_last": 0.1924,
+  "mean_token_accuracy_first": 0.7938,
+  "mean_token_accuracy_last": 0.9483,
+  "base_mean_public_score": 0.532,
+  "trained_mean_public_score": 0.7971166666666667,
+  "base_accepted_count": 2,
+  "trained_accepted_count": 4,
+  "task_count": 6
+}

outputs/training/sft_loss.csv

@@ -0,0 +1,21 @@
+epoch,loss,mean_token_accuracy
+0.2667,1.135,0.7938
+0.5333,1.044,0.812
+0.8,0.8919,0.8255
+1.0,0.8053,0.8334
+1.267,0.7192,0.8479
+1.533,0.7439,0.8394
+1.8,0.6708,0.8549
+2.0,0.6371,0.8601
+2.267,0.5484,0.8757
+2.533,0.5467,0.8786
+2.8,0.4667,0.8898
+3.0,0.4835,0.8901
+3.267,0.4375,0.8986
+3.533,0.4141,0.8993
+3.8,0.3599,0.9111
+4.0,0.2364,0.938
+4.267,0.3648,0.9143
+4.533,0.2731,0.936
+4.8,0.2764,0.9352
+5.0,0.1924,0.9483

outputs/training/sft_run_metadata.json

@@ -0,0 +1,32 @@
+{
+  "status": "completed",
+  "created_at": "2026-04-26T07:21:05.747000+00:00",
+  "model": "Qwen/Qwen3-14B",
+  "method": "Hugging Face TRL LoRA SFT",
+  "job_id": "69edbce1d70108f37acdfcdf",
+  "hardware": "HF Jobs l40sx1",
+  "dataset": "legacy_cobol_env/outputs/training/oracle_sft.jsonl",
+  "dataset_examples": 15,
+  "num_train_epochs": 5,
+  "max_seq_length": 2048,
+  "lora_rank": 16,
+  "lora_alpha": 32,
+  "loss_first": 1.135,
+  "loss_last": 0.1924,
+  "mean_token_accuracy_first": 0.7938,
+  "mean_token_accuracy_last": 0.9483,
+  "base_mean_public_score": 0.532,
+  "trained_mean_public_score": 0.7971166666666667,
+  "base_accepted_count": 2,
+  "trained_accepted_count": 4,
+  "task_count": 6,
+  "artifacts": {
+    "loss_csv": "outputs/training/sft_loss.csv",
+    "loss_plot": "plots/qwen3_14b_loss_curve.png",
+    "reward_plot": "plots/qwen3_14b_reward_before_after.png",
+    "accepted_plot": "plots/qwen3_14b_accepted_before_after.png",
+    "raw_hf_job_log": "outputs/training/hf_job_qwen3_14b_logs.txt",
+    "summary": "outputs/training/qwen3_14b_training_evidence_summary.json"
+  },
+  "note": "Training and local evaluation artifacts were extracted from the HF Jobs run. The raw job log includes a later post-run artifact upload traceback after the training and evaluation summary had been emitted."
+}

pyproject.toml

@@ -0,0 +1,34 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+[build-system]
+requires = ["setuptools>=45", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "openenv-legacy_cobol_env"
+version = "0.1.0"
+description = "Legacy COBOL migration workbench environment for OpenEnv"
+requires-python = ">=3.10"
+dependencies = [
+    "openenv-core>=0.2.3",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-cov>=4.0.0",
+]
+
+[project.scripts]
+# Server entry point - enables running via: uv run --project . server
+# or: python -m legacy_cobol_env.server.app
+server = "legacy_cobol_env.server.app:main"
+
+[tool.setuptools]
+include-package-data = true
+packages = ["legacy_cobol_env", "legacy_cobol_env.server", "legacy_cobol_env.eval", "legacy_cobol_env.training"]
+package-dir = { "legacy_cobol_env" = ".", "legacy_cobol_env.server" = "server", "legacy_cobol_env.eval" = "eval", "legacy_cobol_env.training" = "training" }

server/Dockerfile

@@ -0,0 +1,80 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Multi-stage build using openenv-base
+# This Dockerfile is flexible and works for both:
+# - In-repo environments (with local OpenEnv sources)
+# - Standalone environments (with openenv from PyPI/Git)
+# The build script (openenv build) handles context detection and sets appropriate build args.
+
+ARG BASE_IMAGE=ghcr.io/meta-pytorch/openenv-base:latest
+FROM ${BASE_IMAGE} AS builder
+
+WORKDIR /app
+
+# Ensure git is available (required for installing dependencies from VCS)
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends git && \
+    rm -rf /var/lib/apt/lists/*
+
+# Build argument to control whether we're building standalone or in-repo
+ARG BUILD_MODE=in-repo
+ARG ENV_NAME=legacy_cobol_env
+
+# Copy environment code (always at root of build context)
+COPY . /app/env
+
+# For in-repo builds, openenv is already vendored in the build context
+# For standalone builds, openenv will be installed via pyproject.toml
+WORKDIR /app/env
+
+# Ensure uv is available (for local builds where base image lacks it)
+RUN if ! command -v uv >/dev/null 2>&1; then \
+        curl -LsSf https://astral.sh/uv/install.sh | sh && \
+        mv /root/.local/bin/uv /usr/local/bin/uv && \
+        mv /root/.local/bin/uvx /usr/local/bin/uvx; \
+    fi
+    
+# Install dependencies using uv sync
+# If uv.lock exists, use it; otherwise resolve on the fly
+RUN --mount=type=cache,target=/root/.cache/uv \
+    if [ -f uv.lock ]; then \
+        uv sync --frozen --no-install-project --no-editable; \
+    else \
+        uv sync --no-install-project --no-editable; \
+    fi
+
+RUN --mount=type=cache,target=/root/.cache/uv \
+    if [ -f uv.lock ]; then \
+        uv sync --frozen --no-editable; \
+    else \
+        uv sync --no-editable; \
+    fi
+
+# Final runtime stage
+FROM ${BASE_IMAGE}
+
+WORKDIR /app
+
+# Copy the virtual environment from builder
+COPY --from=builder /app/env/.venv /app/.venv
+
+# Copy the environment code
+COPY --from=builder /app/env /app/env
+
+# Set PATH to use the virtual environment
+ENV PATH="/app/.venv/bin:$PATH"
+
+# Set PYTHONPATH so imports work correctly
+ENV PYTHONPATH="/app/env:$PYTHONPATH"
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+
+# Run the FastAPI server
+# The module path is constructed to work with the /app/env structure
+CMD ["sh", "-c", "cd /app/env && uvicorn server.app:app --host 0.0.0.0 --port 8000"]