π0.5 pre-training: landscape, throughput & π-equivalent sizing

00Summary

What would it take to pre-train π0.5 to a π0-equivalent scale on our hardware? Equivalence here = matching the number of timesteps seen (steps × global batch). π0 trained 700k steps^[1]; its batch size is not published, so the target is a range: ≈0.7–1.4 B timesteps (700k × batch 1024–2048). At our node throughput (~269 frames/s at global batch 2048), that is ~31–62 days on one 8× H200 node, falling to ~1–2 weeks on 4 nodes via data-parallel scaling.

01VLA pre-training landscape

Reported pre-training compute for the major open VLAs. The honest cross-model picture: papers report steps and batch but rarely epochs or timesteps, and several omit batch entirely — so even "timesteps seen = steps × batch" is only partially recoverable. italic = not reported.

*OpenVLA gives 27 epochs × 970k demos & batch 2048 but not a step count; the often-quoted "≈12.8k steps" wrongly treats trajectories as samples and underestimates badly — treat as unreported.^[3]

02Our case — assumptions

• Episode: 1 task = 3 s @ 30 fps → 90 frames/episode (≈ the same control rate as π0's ≤50 Hz, same order). LeRobot-format, same pipeline as the live yubi runs.
• Data size (LeRobot): measured on the live yubi corpus (…/socks_cup_penholder_box_merged: 73.9 GB / 12.02 h) — 480×640, 3 cameras, h264 → ~6.15 GB/hr native (≈ 5 MB/episode, ~52 kB/frame; video = 97% of bytes, parquet 3%), or ~2.13 GB/hr re-encoded to 224² all-intra (≈2.9× smaller). Drives the storage figures in §03.
• Model: π0.5 full fine-tune — PaliGemma gemma_2b + gemma_300m action expert, ~3.3B params (matches π0's class^[1]).
• Node: 8× H200 (141 GB each; JAX usable ≈112 GB), pure data-parallel (fsdp_devices=1).
• Batch is GLOBAL: config.batch_size spans all 8 GPUs (train.py:807 asserts divisibility by device count). per-GPU = global / 8.
• "timestep" = one observation frame consumed by a training sample; throughput is measured in frames/s = timesteps/s. Data loading is not the bottleneck (data_time_s≈0.013).
• Equivalence target: match π0's timesteps seen = steps × batch. π0 = 700k steps; batch unpublished → target ≈ 0.7–1.4 B timesteps (batch 1024–2048).

03Analysis

Throughput vs batch size

Anchored to the live yubi run: global batch 64 on 8 GPUs (8 samples/GPU) → step_time≈0.23 s compute, ≈0.48 s wall (2.1 it/s). Model: compute linear in per-GPU batch + a ~fixed 0.25 s/step host overhead.

wall_s(B)     = 0.00359·B + 0.25          # B = global batch; per-GPU = B/8
node frames/s = B / wall_s(B)             # ceiling = 1/0.00359 ≈ 278 fr/s

GPU memory vs batch size

One measured anchor: 45.7 GB at 64 samples/GPU (full-FT). Fixed term (weights + grads + AdamW + EMA, est. ≈39 GB) + ≈0.10 GB/sample activations (with remat):

Data required for a π-equivalent corpus

At π0's ~1-epoch regime the unique dataset you must collect/store ≈ the timesteps-seen target. Converting at our encoding — 3 s/episode, 30 fps → 90 frames; 6.15 GB/hr native (480×640, 3-cam h264; video = 97% of bytes) or 2.13 GB/hr re-encoded to 224² all-intra:

episodes = T / 90          hours = T / 30 / 3600          storage = hours × GB_per_hr

04Estimate — the π-equivalent target

Equivalence = process the same number of timesteps seen as π0. Because π0's batch is unpublished^[1][9], the target is a range:

T_π  = π0_steps × π0_batch = 700,000 × batch
       batch 1024  →  T_π ≈ 0.72 B timesteps      (≈ π0's 0.9 B-timestep dataset at ~0.8 epoch)
       batch 2048  →  T_π ≈ 1.43 B timesteps
time_on_node = T_π / node_frames_per_s            # at batch 2048 → 269 fr/s

05Scenario A — single 8× H200 node

Train at the efficient PT batch (global 2048, 269 fr/s). Steps follow from the timesteps target: steps = T_π / batch.

06Scenario B — multi-node (parallelism strategies)

To match timesteps faster, replicate the model across nodes and split the data — the fixed T_π is then chewed through by more GPUs. Which parallelism applies depends on whether the model fits one GPU. It does (≈40 GB ≪ 141 GB), so the choice is simple:

Time to the π-equivalent target under data-parallel scaling (per-node batch 2048; ideal-linear, then a realistic ~90% efficiency haircut for inter-node all-reduce):

(parenthetical) = with ~85–95% inter-node DP efficiency. Ideal-linear assumes all-reduce fully hidden behind the multi-second step.

07Discussion — bottlenecks & speed-ups

What is the current bottleneck?

Walking the candidate limits at the live config (global batch 64 on 8 GPUs = 8 samples/GPU):

How to speed up PT efficiency

1. Raise global batch 64 → 2048 (256/GPU): ~2× throughput (133 → 269 fr/s) by amortizing the 0.25 s/step overhead. Free, single-node, biggest immediate win.
2. Scale nodes (DDP): ~linear. Once batch is saturated this is the only wall-clock lever — 4–8 nodes → π-equivalent in 1–2 weeks (§06).
3. Cut image tokens — the dominant model FLOP. 3 cameras × 256 SigLIP tokens dominate the prefix; dropping to 2 cameras (or lower resolution) shortens the forward and raises the per-GPU ceiling. The biggest model-side lever.
4. Trim per-step overhead. Log / EMA / metric-reduce less often to shave the fixed 0.25 s — most impactful at small batch.
5. 224² all-intra data. Keeps data_time negligible and cuts storage ~2.9× (§03) — lets you add nodes/workers without I/O stalls.
6. Diminishing returns. Bigger batch beyond 2048, fused kernels, fp8 → <5% each: the wall is per-GPU model FLOPs, addressed only by more GPUs or fewer tokens.

08Method & caveats

• Throughput fit to one live anchor (8 samples/GPU, 0.23 s compute); the synthetic profiler (64/GPU) ran ~25% higher per-sample → treat throughput as ±20% and add 25–50% wall-clock margin for node-contention variance.
• Memory fixed term (~39 GB) estimated from parameter accounting, anchored to the single measured 45.7 GB @ 64/GPU point.
• π-equivalent is defined by timesteps seen, which assumes convergence parity at equal samples — large global batch (multi-node) can break that, requiring more timesteps. The estimate is a floor on data/time, not a convergence guarantee.
• π0's batch is unpublished; the 0.7–1.4 B range brackets the plausible target. Recompute with the real value if Physical Intelligence releases it.

09References

1. π0 — Black et al., "π0: A Vision-Language-Action Flow Model for General Robot Control," arXiv:2410.24164. 700k steps (Fig. 7), 903M timesteps own data (§V-A), ≤50 Hz. arxiv.org/abs/2410.24164
2. π0.5 — Physical Intelligence, "π0.5: a Vision-Language-Action Model with Open-World Generalization," arXiv:2504.16054. 280k pre-train (+80k post), ~400 h, 50 Hz. arxiv.org/abs/2504.16054 · pi05.pdf
3. OpenVLA — Kim et al., arXiv:2406.09246. 7B, 970k demos, 27 epochs, batch 2048, 64× A100, 14 d (21.5k A100·h); step count not stated. arxiv.org/abs/2406.09246
4. Octo — Octo Model Team, arXiv:2405.12213. 27/93M, 800k traj, 300k steps, batch 2048, TPU v4-128, 14 h. arxiv.org/abs/2405.12213
5. RT-2 — Brohan et al., arXiv:2307.15818. PaLM-E-12B / PaLI-X-55B; per-variant batch 512–2048, steps 80k–1M (App. E); 1–3 Hz. arxiv.org/abs/2307.15818
6. RT-1 — Brohan et al., arXiv:2212.06817 (RT-2's robot-data source: ~130k episodes, 700+ tasks, 13 robots). arxiv.org/abs/2212.06817
7. RDT-1B — Liu et al., arXiv:2410.07864 (ICLR 2025). 1.2B (DiT + SigLIP + T5-XXL), 1M+ traj / 46 sets / 21 TB, 1M iters, 48× H100, ~1 month. arxiv.org/abs/2410.07864
8. openpi — Physical Intelligence training configs (ship pi0_base as a pretrained checkpoint; only fine-tune configs are public). github.com/Physical-Intelligence/openpi
9. open-pi-zero — allenzren, community π0 reimplementation; global batch 1024, lr 5e-5 (reimplementer's choice, not official). github.com/allenzren/open-pi-zero
10. pi-zero-pytorch — lucidrains, architecture-only π0 port (no pretraining recipe). github.com/lucidrains/pi-zero-pytorch