π0.5: architecture, tensor IO & ablation knobs

00Summary

π0.5 is a dual-expert flow-matching VLA: a single 18-layer Gemma transformer where a 2B-width PaliGemma expert encodes vision + language (the "prefix") and a 300M-width action expert denoises an action chunk (the "suffix"), the two attending jointly. This card pins every module's input/output tensor shape for the pi05_libero config and maps each GitHub-issue ablation knob to the exact module and tensor it touches.

01Architecture

Dual-expert Gemma (mixture-of-transformers)

The core is a single Gemma stack of 18 layers instantiated with two per-layer weight sets (_gemma.Module(configs=[paligemma, action_expert]), pi0.py:74). Tokens are routed to an expert by which sequence they belong to, not by a gate: the VLM prefix (images + language) flows through the gemma_2b expert (width 2048), the action suffix through the gemma_300m expert (width 1024). Both experts share num_heads=8, head_dim=256, depth=18 — the matched depth + head geometry is what lets the two token streams attend to each other in every layer through one joint attention mask. Only width and mlp_dim (16384 vs 4096) differ.

SigLIP image tower

Each camera image is encoded by a shared SigLIP So400m/14 tower (_siglip.Module(variant="So400m/14", pool_type="none"), pi0.py:82): a 224×224×3 frame is cut into 14×14 patches → 16×16 = 256 patch tokens, run through the 27-layer So400m encoder (internal width 1152), and projected to the PaliGemma width (num_classes=2048). The tower is weight-tied across cameras — applied independently per view inside the for name in obs.images loop (pi0.py:113).

Image input contract — how many views?

pi0/π0.5 expects exactly three named image slots, fixed by Pi0Config.inputs_spec (pi0_config.py:61):

π0 vs π0.5 — what the pi05 flag changes

02Model diagram

flowchart TD
  IMG["images x3 : [b,224,224,3]"]:::data
  TOK["tokenized_prompt : [b,200]"]:::data
  IMG --> SIG["SigLIP So400m/14 per cam : [b,224,224,3] to [b,256,2048]"]:::vlm
  TOK --> LEMB["llm.embed : [b,200] to [b,200,2048]"]:::vlm
  SIG --> PFX["embed_prefix pi0.py:106 : prefix [b, up to 712, 2048]"]:::vlm
  LEMB --> PFX
  XT["noisy actions x_t : [b,10,32]"]:::act --> AIP["action_in_proj : [b,10,32] to [b,10,1024]"]:::act
  TSN["timestep t : [b]"]:::act --> TM["posemb_sincos + time_mlp : adarms_cond [b,1024]"]:::act
  AIP --> SFX["embed_suffix pi05 pi0.py:140 : [b,10,1024]"]:::act
  PFX --> LLM["Gemma dual-expert LLM 18 layers : prefix 2048-w expert, suffix 1024-w expert, joint attention"]:::core
  SFX --> LLM
  TM -. "adaRMS suffix only" .-> LLM
  LLM --> AOP["action_out_proj : suffix_out last-10 [b,10,1024] to [b,10,32]"]:::act
  AOP --> V["velocity v_t : [b,10,32]"]:::out

  classDef data fill:#E3DACC,stroke:#B85C3E,color:#141413;
  classDef vlm fill:#F0EEE6,stroke:#87867F,color:#3D3D3A;
  classDef act fill:#FAF9F5,stroke:#788C5D,color:#141413;
  classDef core fill:#FAF9F5,stroke:#D97757,color:#141413,stroke-width:2px;
  classDef out fill:#FAF9F5,stroke:#141413,color:#141413,stroke-width:2px;
  

Fig. 1 — Forward pass with tensor shapes (pi05_libero, batch b). Prefix length ≤ 712 = 2 active cameras × 256 + ≤ 200 language tokens. The action expert's timestep enters as adaRMS conditioning, not as a sequence token.

03Tensor IO — per module

All shapes for pi05_libero, batch b; ah=action_horizon=10, ad=action_dim=32. N = prefix_len + suffix_len.

04Flow-matching objective

Conditional flow matching^[4] over the action chunk. Convention: t=1 is noise, t=0 is the data (note: opposite of the π0 paper).

# compute_loss (pi0.py:219)
noise = N(0, I)                          # [b,10,32]
t     = Beta(1.5, 1) * 0.999 + 0.001     # [b]   (biases samples toward t->0/data)
x_t   = t*noise + (1 - t)*actions        # [b,10,32]  interpolant
u_t   = noise - actions                  # [b,10,32]  target velocity
v_t   = model._predict_velocity(obs, x_t, t)         # [b,10,32]
loss  = mean( (v_t - u_t)^2 )            # over action_dim -> [b,10]

# sample_actions (pi0.py:237): Euler integration, num_steps=10
x = noise                                # [b,10,32],  t = 1.0,  dt = -1/num_steps
# prefix KV-cache filled once (vision+language); suffix re-run each step
while t >= -dt/2:
    v_t = action_out_proj( llm([None, suffix(x,t)], kv_cache=prefix)[-10:] )
    x   = x + dt * v_t ;  t = t + dt
return x                                  # [b,10,32]  ~ clean action chunk

05Implementation details — code anchors

06Ablation knobs (GitHub issues)

Each parameter ablated across the issue tracker, mapped to the architecture: the module/tensor it touches, the range tested, the governing issue, and what it isolates. Results live in the issues themselves.

07References

1. [1] Black et al. π0: A Vision-Language-Action Flow Model for General Robot Control. Physical Intelligence, 2024. arXiv:2410.24164
2. [2] Physical Intelligence. π0.5: a VLA with Open-World Generalization. 2025. arXiv:2504.16054
3. [3] Beyer et al. PaliGemma: A versatile 3B VLM for transfer. 2024. arXiv:2407.07726 · SigLIP: Zhai et al. arXiv:2303.15343
4. [4] Lipman et al. Flow Matching for Generative Modeling. ICLR 2023. arXiv:2210.02747
5. [5] Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685
6. [6] Liu et al. LIBERO. NeurIPS 2023. arXiv:2306.03310

Internal

• Issues: #3 (head pretrain) · #4 (low-fps clamp) · #5 (wrist-only) · #6 (action_horizon) · #7 (LoRA vs full-FT) · #9 (fps30 grid) · #10 (distillation) · #12 (multi vs per-task) · #2/#8 (data audits)
• Config: configs/model/pi05.yaml · upstream third_party/openpi/src/openpi/training/config.py:751 (pi05_libero)
• Companion report: pi05-wrist-only.html (view-masking ablations)