Why GPU Wiring Dictates Model Parallelism

Training massive AI models requires keeping GPUs constantly busy, but communication bottlenecks create steep performance cliffs. Bandwidth plummets by a factor of eighteen the moment data moves beyond a single server chassis, stepping down from internal NVLink copper to external InfiniBand fabrics. This hardware reality dictates parallel training strategies, showing why tensor parallelism must stay inside the box, how hierarchical all-reduce slashes inter-node traffic, and why cross-datacenter training relies on sparse, infrequent synchronization. Key sources: • Hugging Face / Nanotron, *The Ultra-Scale Playbook: Training LLMs on GPU Clusters*, retrieved 2026-11-09 — https://nanotron-ultrascale-playbook.static.hf.space/index.html • Hugging Face, Nanotron org page listing the book, authors and affiliation, retrieved 2026-11-09 — https://huggingface.co/nanotron • Chen et al., CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training, USENIX ATC '25, submitted 2025-06-30 — https://arxiv.org/abs/2507.00217v1 Episode 6517 — myweirdprompts.com/n/6517 (permanent short link: myweirdprompts.com/118864) Listen on Android: MWP FM Player — myweirdprompts.com/apk