09/03/2026
✨AI infrastructure✨ is a genuinely broad term. Here’s a mind map of what it actually is layer by layer
🔹 Power & data centers:
one modern GPU rack (NVIDIA’s GB200 NVL72) draws about 120 kW, roughly 10x a traditional server rack. Power is the fixed budget, so every layer above this one is partly about getting more compute out of the same watts
🔹 Compute:
NVIDIA did $89B in data center revenue last quarter. The hyperscalers now also build their own chips (TPU, Trainium, Maia, MTIA)
🔹 Networking:
thousands of chips sync up after every training step (sharing what they each just learned) and everything stalls until the slowest link finishes
🔹 Data & storage:
GPUs regularly sit below 50 percent utilization just waiting on data. People blame the GPUs when it’s usually the storage
🔹 Orchestration:
the layer that queues your job, starts it on every GPU at once and restarts it when one dies. Slurm, Kubernetes, Ray
🔹 Training & inference:
you train a model once but you pay to run it every day after. Inference (actually running the model for users) is now about two-thirds of all AI compute (Deloitte 2026)
🔹 Software:
CUDA, ROCm, XLA, the kernel and compiler layer that decides how much performance you actually get from the hardware
🔹 Observability & governance:
the dashboards can show healthy while the model is confidently wrong
Most of these layers already power traditional software….AI mainly moves the bottleneck: from database queries and web traffic to memory bandwidth, interconnect speeds & kilowatts.
✨ Save this for the next time you need to explain what AI Infrastructure is