π0.5 wrist-only: full-FT baselines on LIBERO

00Summary

A full fine-tuned π0.5 reaches 94.2% wrist-only success on LIBERO-Spatial when the third-person camera is physically removed from attention — almost the 96.6% both-camera ceiling — while the identical recipe that instead feeds a black (zeroed) frame collapses to 27.4%.

The headline finding is that the apparent difficulty of wrist-only π0.5 was largely an artifact of the masking mechanism, not a capacity ceiling. Frozen-backbone LoRA collapses to a ~5% behavior-cloning floor regardless of dropout schedule (issue #5); unfreezing the backbone (full-FT) is necessary but not sufficient — how the view is removed matters as much as whether the backbone is trainable. This report covers the five full-FT wrist-only baselines (two evaluated, three in-flight self-distillation ablations under issue #10), how each is trained and wired into the openpi pipeline, and the line of code where every mechanism lives.

01Background

Wrist-only deployment

LIBERO^[3] exposes two cameras: a fixed third-person agentview and a wrist-mounted eye_in_hand. Many real deployments lack a reliable exterior view (occlusion, mounting cost, calibration drift), so a policy that runs wrist-only is operationally valuable. The question: can a model pretrained with both cameras be adapted to ignore the exterior view without collapsing?

π0.5, in one paragraph

π0.5^[2] (a successor to π0^[1]) is a vision-language-action flow-matching model: a PaliGemma-style backbone (gemma_2b) encodes images + language, and a smaller action expert (gemma_300m) denoises an action chunk via conditional flow matching^[4]. Each camera is tokenized into image tokens that the transformer attends over jointly with the language and action tokens. The training target is the flow-matching velocity u_t = noise − actions; the loss is mean((v_θ − u_t)²).

The LoRA floor (issue #5) — why this report exists

The first wrist-only batch adapted π0.5 with LoRA^[5] (~1.5% trainable params, frozen backbone). Both dropout schedules collapsed to a ~5% floor:

• LoRA, agentview dropped 50% of the time (p=0.5): 4.4% (22/500)
• LoRA, agentview dropped always (p=1.0): 5.0% (25/500, reproduced from a 4.6% r1)

A ~1.5%-trainable adapter on an agentview-pretrained backbone cannot rewire onto a wrist-only distribution. Issue #5's TODO — "re-test the hypothesis out of the floor regime: unfrozen backbone" — is the starting point for the full-FT baselines below. Reference anchors for the wrist-only task:

• Both-camera ceiling: 96.6% — the full-obs π0.5 LoRA (PI05_LORA_r2), also the distillation teacher.
• From-scratch wrist-only (diffusion policy): 50.8% (p=1.0) / 30.0% (p=0.5) — a small from-scratch policy trivially ignores a dead channel.

02Hypothesis

Three falsifiable claims drive the full-FT batch:

1. H1 — capacity: the LoRA collapse is a frozen-backbone capacity artifact, not a wrist-only ceiling. Unfreezing the whole backbone (full-FT) should escape the ~5% floor.
2. H2 — mechanism: a from-scratch diffusion policy reaches 50.8% but full-FT π0.5 zero-mask only 27.4%. The gap is suspected to come from the zero-mask substitution — a pretrained image tower still encodes a dead black frame and the transformer still attends to it. Physically excluding the agentview tokens from attention should beat the zero-mask.
3. H3 — privileged signal: a teacher that sees both cameras can transfer its behavior to a wrist-only student at the flow-matching velocity level, lifting it toward the both-cam ceiling. Self-distillation (and a curriculum that gradually withdraws the exterior view) should beat plain behavior cloning under the same physical-removal condition.

03Approach

Shared recipe

All five baselines inherit the canonical upstream pi05_libero recipe (configs/experiment/pi05/libero_original.yaml): pi05_base loader, gemma_2b backbone + gemma_300m action expert, no LoRA (the freeze filter returns Nothing ⇒ all params trainable), model-side EMA 0.999, action_horizon=10, cosine LR with peak == decay 5e-5 (≈ constant after 10k warmup), 30k steps, seed 42. Upstream batch is 256 (multi-GPU FSDP); single-H200 runs override batch_size=32 (dry-run-confirmed: ~108 GB peak, no OOM). All run via the openpi trainer — the AiroPi 8-GPU FSDP backend has no view-masking support.

System diagram — where each mechanism is injected

flowchart LR
  DS["LeRobot LIBERO libero_spatial"]:::data
  DS --> RP["repack_transforms"]:::stage
  RP --> DTI["data_transforms.inputs after LiberoInputs"]:::stage
  DTI --> M["pi0.5 model forward : embed_prefix to make_attn_mask"]:::model
  M --> L["BC flow-matching loss : mean of (v - u_t) squared"]:::loss

  RP -. "M1 view_dropout zeros, TRAIN-only, _ViewDropoutRepack :219" .-> M
  DTI -. "M2 view_remove image_mask=False, TRAIN+EVAL, _ViewMaskOutTransform :274" .-> M
  TC["both-cam teacher self or checkpoint"]:::teacher -. "M3 distill in-loss mask, compute_distill_loss :208" .-> L

  classDef data fill:#E3DACC,stroke:#B85C3E,color:#141413;
  classDef stage fill:#F0EEE6,stroke:#87867F,color:#3D3D3A;
  classDef model fill:#FAF9F5,stroke:#D97757,color:#141413,stroke-width:2px;
  classDef loss fill:#FAF9F5,stroke:#788C5D,color:#141413,stroke-width:2px;
  classDef teacher fill:#F0EEE6,stroke:#788C5D,color:#3D3D3A;
  

Fig. 1 — Three mutually-exclusive masking seams. M1 (view_dropout) is repack-level ⇒ train-only. M2 (view_remove) is in data_transforms.inputs ⇒ applied at train and served eval. M3 (distill) leaves the data pipeline both-cam and masks inside the loss. Exclusivity is enforced loudly in run_openpi_train.py.

Mechanism diagram — how image_mask=False removes a camera

flowchart TD
  A["image_masks base_0_rgb = False (agentview)"]:::set --> B["embed_prefix pi0.py:106 : repeat mask over agentview tokens, input_mask = False"]:::stage
  B --> C["make_attn_mask pi0.py:19 : valid = input_mask outer-product"]:::stage
  C --> D["agentview rows AND cols zeroed in attention"]:::out
  D --> E["tokens excluded from attention = treated like padding"]:::win

  Z["view_dropout zeros : image_mask stays True"]:::bad -. "pixels=0 but tokens still VALID, model still attends" .-> Y["wasted capacity, OOD frame = 27.4%"]:::bad

  classDef set fill:#E3DACC,stroke:#B85C3E,color:#141413;
  classDef stage fill:#F0EEE6,stroke:#87867F,color:#3D3D3A;
  classDef out fill:#FAF9F5,stroke:#D97757,color:#141413;
  classDef win fill:#FAF9F5,stroke:#788C5D,color:#141413,stroke-width:2px;
  classDef bad fill:#FAF9F5,stroke:#B85C3E,color:#141413,stroke-dasharray:4 3;
  

Fig. 2 — Physical removal vs zero-mask. The outer product in make_attn_mask means any token with input_mask=False has its entire row and column zeroed: it can neither attend nor be attended to. This is openpi's own padding-image convention (the unused right_wrist_0_rgb slot), so num_images and the pi05_base weight contract stay intact.

make_attn_mask — the two lines that matter

def make_attn_mask(input_mask, mask_ar):           # third_party/openpi/src/openpi/models/pi0.py:19
    mask_ar = jnp.broadcast_to(mask_ar, input_mask.shape)
    cumsum = jnp.cumsum(mask_ar, axis=1)
    attn_mask = cumsum[:, None, :] <= cumsum[:, :, None]
    valid_mask = input_mask[:, None, :] * input_mask[:, :, None]   # ← False row/col => dropped
    return jnp.logical_and(attn_mask, valid_mask)

The four masking mechanisms, contrasted

Privileged distillation loss

The self-distillation variants add a velocity-matching term against a stop-gradient teacher that sees both cameras, plus a both-cam BC anchor that keeps the teacher pathway alive:

loss = BC(v_student_wrist, u_t)                          # always coefficient 1.0
     + lambda * || v_student_wrist - sg(v_teacher_bothcam) ||^2   # velocity distillation
     + anchor_frac * BC(v_teacher_bothcam, u_t)           # both-cam anchor

curriculum:  p(step) = 0.5 * (1 - cos(pi * progress)),  progress = clip(step / (0.75*N), 0, 1)

The teacher shares the student's graphdef + params (mode=self, live, ema_decay=null); the only difference between the two forwards is the input observation. This adds activations but no extra parameter memory.

04The five full-FT baselines

Each card: how it is trained (config, knobs, slurm) and how it is implemented (the line of code that does the masking).

05Results

LIBERO-Spatial, agentview removed at eval, full 500-episode suite unless noted. Bars are scaled to the 96.6% both-cam ceiling. Reference rows are not full-FT baselines — they frame the result.

06Discussion

The dominant variable in wrist-only π0.5 is not the backbone's trainability but how the exterior view is removed. Zero-masking keeps the agentview tokens "valid", so a pretrained image tower spends capacity encoding a black frame and the transformer is obligated to attend to an out-of-distribution dead view — dragging full-FT to 27.4%, below even a from-scratch diffusion policy (50.8%). Physically excluding the tokens from attention (B2) restores near-ceiling performance (94.2%), and a frozen-LoRA adapter (5%) simply lacks the capacity to compensate for either deficiency.

07Implementation details — code anchors

Every masking mechanism in this report, and the line of code where it lives.

Example: physical-removal config (B2)

_base: libero_original.yaml
wandb_run_name: pi05_wristonly_physdrop_full

# PHYSICAL camera removal (image_mask=False) - distinct from view_dropout zeros.
view_remove:
  enabled: true
  seed: 42
  train: { agentview: 1.0 }   # exclude third-person view from attention in training
  eval:  { agentview: 1.0 }   # same physical removal at served eval

Reproduce

# Train (single H200, openpi trainer)
uv run python scripts/train.py \
  --config configs/experiment/pi05/libero_wristonly_physdrop_full.yaml \
  batch_size=32

# Eval n=500, server-side physical removal
PI0_CKPT_DIR=<orbax .../29999> OPENPI_CONFIG=pi05_libero NUM_TRIALS=50 \
  VLA_ZOO_VIEW_REMOVE_EVAL=agentview \
  LOG_DIR=outputs/eval_logs/PI05_FULL_WRISTONLY_PHYSDROP_EVAL_FULL \
  bash scripts_sh/eval_pi0_trained.sh libero_spatial

08References

Papers

1. [1] Black et al. π0: A Vision-Language-Action Flow Model for General Robot Control. Physical Intelligence, 2024. arXiv:2410.24164
2. [2] Physical Intelligence. π0.5: a VLA with Open-World Generalization. 2025. arXiv:2504.16054
3. [3] Liu et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS 2023. arXiv:2306.03310
4. [4] Lipman et al. Flow Matching for Generative Modeling. ICLR 2023. arXiv:2210.02747
5. [5] Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685
6. [6] Chen et al. Learning by Cheating (privileged teacher → sensorimotor student). CoRL 2020. arXiv:1912.12294
7. [7] Rusu et al. Policy Distillation. ICLR 2016. arXiv:1511.06295

Internal

• GitHub issues #5 (wrist-only floor re-test) · #10 (curriculum + self-distillation ablation)
• Configs: configs/experiment/pi05/libero_wristonly_full.yaml, …/libero_wristonly_physdrop_full.yaml, …/distill/{curriculum_only,selfdistill_only,curric_selfdistill}.yaml
• Job ledger: configs/jobs.yaml, configs/jobs_distill.yaml
• Eval logs: outputs/eval_logs/PI05_FULL_WRISTONLY_*_EVAL_FULL/