sft_robodojo_vanilla48k_eefabs_f33fps8_sana_pixel_aligned_videoaug

Model weights only of the RoboDojo ARX-X5 EEF-only policy SFT on the sana_pixel 2x2 pixel canvas at 320x512 with strided video (video_fps 8), absolute EEF targets, the aligned RoPE and video augmentation on half of the samples, warm-started from the vanilla robot pretrain logits/sana_rwm_pretrained_vanilla_e9s48000 (wandb lzknus/sana-rwm/sft_robodojo_eefabs_sana_pixel_320x512_vanilla48k_aligned_f33fps8_videoaug), at checkpoint epoch_10_step_36180 (epoch 10, optimizer step 36,180 = the full 36,180-step budget, 10 epochs). Final checkpoint of the finished run (Slurm array 19351078, training reached step 36,180 on 2026-09-28; uploaded 2026-09-28). Its canvas twin on the same donor and recipe is logits/sft_robodojo_vanilla48k_eefabs_f33fps8_openwam_aligned_videoaug.

Files

file	bytes	sha256
model/pytorch_model_fsdp.bin	17874895202	12c3848e4709b42b234e7617ac29f50dc46caaf7a7082c94499d0924fedd2111
metadata.pth	37269	d3e1081c5240b3bc6c1878c88ce1cef041259c7fb0fccfc94c218aa38362186c
config.yaml	11892	6bcecb2f141e5749a9459d71a938b15d781d50375909380a6a732b30e41f4b0c
normalization/robodojo_arx_x5_model_fps_25_f33_normalization.json	45608	1fe3b7e72e8fa93eca5efb8a8deeace1daf4434b4b97e9f4445670ac758287ed

• model/pytorch_model_fsdp.bin: accelerate FSDP consolidated state dict, 805 tensors, 4,468,977,840 parameters (804 float32, 1 bfloat16). The eight robot-module tensors (state_embed.proj.*, action_embed.proj.*, action_head.*, plucker_embed.weight) came with the robot-pretrained donor and were fine-tuned here (the donor loaded strictly, missing only pos_embed); pos_embed present.
• metadata.pth: epoch / step / scheduler / RNG bookkeeping read next to the weights.
• config.yaml: the trainer's frozen, fully resolved training config, unchanged (paths are cluster-local); it declares model.extra.rope: aligned and the train.extra.video_augmentation block.
• normalization/...json: the robot80 normalization artifact the run trained with (sha256 1fe3b7e72e8fa93eca5efb8a8deeace1daf4434b4b97e9f4445670ac758287ed; f33, absolute targets, frame-aligned corpus). Grippers are q01/q99-normalized too (closedness [0, 1] -> [-1, 1]).
• Not included: model/optimizer.bin, model/scheduler.bin, random_states_*.pkl, the training log (weights only).

Recipe (from config.yaml)

• Model SanaRWMVideoQwenNextSubAttnResV2SelfFlowWorldModelCameraConditionMultiViewPolicy_5B_P1_D36 (32 blocks, softmax attention every 4th, GatedDeltaNet elsewhere), bf16, fp32 attention; data.extra.multiview: sana_pixel. Training code: rwm/zekai-merge 7070353ec.
• Donor model.load_from: sana_rwm_pretrained_vanilla_e9s48000 (Hub logits/sana_rwm_pretrained_vanilla_e9s48000: the 256px vanilla absolute EEF/joint unified robot pretrain at 7.5 fps, epoch 9 step 48000).
• Visual stream: ONE sana_pixel canvas per frame. The three cameras are composited in pixel space into one 320x512 RGB image of 160x256 tiles (head top-left, left wrist bottom-left, right wrist bottom-right, the unused top-right quadrant black at -1.0, aspect_ratio_type ASPECT_RATIO_SANA_PIXEL_2X2_320_512) BEFORE the LTX-2.3 VAE; 160x256 tiles are whole 32x32 VAE cells, so the policy sees one view (V = 1, plain mRoPE) on a 10x16 latent grid, the black quadrant a 5x8 block that stays in the video loss. No camera conditioning.
• View preprocessing of the training code (7070353ec): each 480x640 camera frame was center-cropped to the canvas aspect (rows 0-39 and 440-479 dropped) before it was resized into its tile. rwm/zekai-merge stretches every whole frame into its tile since 1061b16f0 (2026-09-27), so later trees feed this checkpoint views it was not trained on; see Validation.
• Video augmentation (train.extra.video_augmentation): crop_scale 0.95, brightness 0.3, contrast 0.4, saturation 0.5, hue 0.08, apply_prob 0.5: on the GPU before the VAE, one draw per sample shared by all its frames, applied to the composited canvas as one image; about half of every batch is left untouched. The VAE encodes online (no latent store). Validation is clean.
• Windows: 33 source rows at video_fps 8 (frame stride 4): the observation frame plus 8 sampled frames = 9 canvas frames = 2 latent frames, while the actions stay dense: 32 action rows at 25 fps.
• RoPE: aligned (model.extra.rope). Video and actions share one physical clock in base-fps (16) latent-frame units: video latent j at 16 * j * 4 / 25 (0 and 2.56), action row k at 16 * k / (8 * 25) = 0.08 k (0.08 .. 2.56), the state at 0.
• Targets: EEF-only (action_mode_sample_ratio [0.0, 1.0, 0.0], robot_base_eef: both arms' EEF position + Rot6D in the robot base frame plus the grippers), eef_target_mode absolute. Normalization pin 1fe3b7e7....
• Data contract: frame-aligned. All 3,500 episodes (holdout_episodes_per_task_split 0), tail windows (min_rows 2, padding freeze): 1,744,102 full + 108,500 tail = 1,852,602 windows, 3,618 steps per epoch at 512 windows per step (8 nodes x 8 GPUs x bs 8).
• Text contract: G = 1, ONE shared prompt (the composite view's: embodiment, action mode, the canvas layout, instruction); instruction dropout 0.1 per scene.
• Noise schedule (the 2026-09-24 SFT default): flow shift 5.0 standard for the video, a separate action flow shift 1.0 on the same raw timestep draw, inference 5.0 / 1.0, OpenWAM timestep loss weighting with min_weight 0.1, min_train_timestep 1.
• Optimizer: AdamW peak lr 0.0001 after 2,000 warmup steps, cosine to 1e-06 over 36,180 steps, weight decay 0.0001 on weight matrices only, grad clip 1.0, action_loss_weight 1.0.

Validation (seen-episode monitor: 35 tasks x 1 episode the model trained on; normalized masked action MSE)

The run had no milestone watcher; the final checkpoint was validated from the training code (rwm/zekai-merge 7070353ec, CFG off, 50 steps):

ckpt step	n	mean	median	max	tasks > 0.2
36,180	35	0.0037	0.0003	0.0605	0

Validated again from rwm/zekai-merge 77cf81fbf (views stretched whole, not cropped): mean 0.0063, median 0.0013, max 0.0769, 0 tasks > 0.2, 0/35 samples bitwise equal to the training-code run. Use the training code's preprocessing to reproduce the numbers above.

Loading

--model.load_from=<local dir holding model/ and metadata.pth> for the Sana-RWM trainers and validator on rwm/zekai-merge 7070353ec (the training code: cropped views); trees from 1061b16f0 on stretch the views and give this checkpoint inputs it was not trained on; the bidirectional deploy takes the same directory with config.yaml and normalization/...f33_normalization.json. The model output is the normalized action; saturate the gripper closedness to [0, 1] after denormalizing, not before.