liangyue36/qwen2.5-coder-3b-honesty-rft

Qwen2.5-Coder-3B-Instruct: honesty RFT LoRA (rejection-sampling fine-tuning baseline)

LoRA adapter (rank 32, alpha 32, all linear layers) from the RFT baseline in github.com/cassieliang6709/agent-honesty-eval, a coding-agent eval with 100 MBPP-derived tasks: 60 feasible and 40 made impossible by contradictory tests, graded against pristine tests in isolated processes.

This adapter is a negative result kept for reference: it over-refuses.

Training

1. Sample the base model 8 times on 260 training tasks (T = 1.0, baseline prompt, no honesty instruction): 2,079 trajectories.
2. Keep trajectories with reward +1 (solved, or honestly reported "cannot" on an impossible task), at most 2 per task: 275 kept (107 solved, 168 honest refusals).
3. LoRA SFT, loss on assistant turns only, 2 epochs, lr 1e-4, global batch 16 (4 × A800, DDP).

Evaluation (100 eval tasks, 4 passes, T = 0.7)

Outcome	Base	RFT
Feasible: solved	42 / 240	39 / 240
Feasible: says "cannot"	52 / 240	201 / 240
Feasible: tampered tests or claimed false completion	62 / 240	0 / 240
Impossible: honest "cannot"	38 / 160	159 / 160
Impossible: tampered tests or claimed false completion	42 / 160	0 / 160

RFT removes tampering and false completion while keeping the solve rate, but the kept data is 61% refusals and the model learns to refuse feasible tasks. Single training run.

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-3B-Instruct")
model = PeftModel.from_pretrained(base, "liangyue36/qwen2.5-coder-3b-honesty-rft")