SkyPilot published an engineering analysis on October 29 arguing that NVIDIA's GB200 NVL72 racks break the assumptions most GPU schedulers are built on. The useful lesson is counterintuitive: on these racks, deliberately leaving part of every rack idle can give you more usable compute than trying to fill every slot.
The rack is now the server
For years the basic unit of a GPU cluster was a server with eight GPUs linked by NVLink. Anything beyond that box went over the network, InfiniBand or RoCE, and schedulers such as Slurm or Kubernetes with Kueue treated nodes as roughly interchangeable.
The NVL72 moves that boundary. It packs 72 Blackwell GPUs and Grace CPUs into one rack, organized as 18 compute trays of four GPUs each, all joined by a single NVLink switch fabric with about 130 TB/s of aggregate bandwidth. Inside the rack, a GPU talks to its peers at roughly 1.8 TB/s. Step outside, and traffic falls to scale-out links measured in hundreds of gigabits per second.
SkyPilot calls this a topology cliff. Tensor and expert parallelism sit on the critical path of every training step. When a job meant for one NVLink domain gets spread across racks, the slow link sets the pace. The report measures throughput losses of up to 43% in that situation.
Placement shows up in the bill
The report also cites NVIDIA's GR00T team, which trained GR00T N1.7 on 512 GB300 GPUs across 128 nodes. Topology-aware scheduling raised throughput by 14.4%, cutting iteration time from 1.59 to 1.39 seconds. SkyPilot extrapolates that a gain of that size on a 10,000-GPU cluster is worth roughly $400,000 a day in compute.
Treat the dollar figure as an estimate, not a law. The mechanism, though, is simple: a scheduler that sees "64 free GPUs" without knowing which rack they sit in can quietly turn a fast job into a slow one.
One failed node can strand 68 GPUs
The second problem is harder to see. If large jobs request a whole rack, all 18 nodes, then a single failed node makes the remaining 17 nodes (68 GPUs) unusable for that class of job. The hardware is healthy, but it no longer has the right shape.
Failures are not rare at scale. Under the report's reliability assumptions, an 18-node rack sees a node interruption roughly every 27 days. Across 128 racks, or 2,304 nodes, some rack loses a node about every five hours. A full-rack allocation policy keeps bleeding capacity into fragments.
Two spare nodes per rack, 27% more capacity
SkyPilot's answer is to stop sizing jobs to the full rack. The scheduler carves each rack into a fixed 16-node slice (64 GPUs) and keeps the other two nodes (8 GPUs) as in-rack hot spares. When a node fails, a spare in the same NVLink domain replaces it, so the job keeps its topology and nobody has to drain the rack or move work elsewhere.
On the 2,304-node example, the report says this yields about 2,037 usable nodes versus about 1,602 under full 18-node allocation, roughly 27% more usable capacity from identical hardware. Idling 11% of every rack wins because it removes a much larger loss from fragmentation.
Two supporting mechanisms make it work:
- Hard topology constraints. Bandwidth-hungry stages such as tensor or context parallelism must stay inside one NVLink domain; the scheduler rejects placements that cross it.
- Automatic defragmentation. When a big job is blocked because no clean rack exists, the system checkpoints and preempts small, easily restarted jobs, such as short fine-tuning runs, and moves them to free a whole rack.
The trade-offs are real. Preemption costs smaller users some work, a 64-GPU slice may not suit models designed around 72-way parallelism, and the gains depend on failure rates that vary by fleet. SkyPilot also has an obvious interest in selling scheduling software. Still, the arithmetic is transparent enough to check against your own cluster.
What Vietnamese teams should take from this
Most Vietnamese companies will not run 128 NVL72 racks. But more teams rent GPU capacity from domestic and international clouds, and the same principle applies at smaller scale.
- Ask where your GPUs are. When renting 32 or 64 GPUs on Blackwell-class hardware, ask whether they sit inside one NVLink domain. "Same region" or "same cluster" is not the same promise.
- Record topology in benchmarks. A throughput or tokens-per-second number without rack-local versus cross-rack placement can mislead you about whether a model or optimization really improved.
- Size jobs to shapes that survive failure. Designing parallelism around 64 GPUs rather than exactly 72 leaves room for spares and makes scheduling easier.
- Budget for orchestration, not just chips. Faster GPUs do not guarantee faster training if the scheduler still thinks in fungible nodes.
The broader point: as hardware gets denser, the abstraction boundary moves up from the server to the rack, and software has to follow. Teams that keep counting GPUs as interchangeable units will pay for hardware they cannot fully use.
Loading comments…