AI Infrastructure
#12 5 pagesAI Infrastructure
Platform engineering for AI/ML workloads — the substrate underneath models. Kubernetes stretched to its limits with GPUs, distributed compute, and high-throughput networking.
This is the natural extension of your K8s/Linux/DevOps foundation.
0/0 checks
How AI Infra Relates to What You Know
graph TD
subgraph Existing["What you already know"]
K8S["Kubernetes<br/>scheduling · networking · operators"]
LINUX["Linux<br/>cgroups · eBPF · namespaces"]
OBS["Observability<br/>Prometheus · Grafana · OTel"]
CICD["CI/CD<br/>ArgoCD · GitHub Actions"]
end
subgraph AIInfra["AI Infrastructure (Phase 1)"]
GPU["GPU Scheduling<br/>NVIDIA plugin · MIG · GPU Operator"]
RAY["Distributed Compute<br/>KubeRay · RayCluster"]
SERVE["Model Serving<br/>KServe · vLLM · InferenceService"]
NET["High-Throughput Networking<br/>Cilium · RDMA · InfiniBand concepts"]
end
subgraph MLOps["MLOps (Phase 2)"]
TRACK["Experiment Tracking<br/>MLflow · W&B"]
PIPE["Training Pipelines<br/>Kubeflow · Airflow DAGs"]
DRIFT["Data Drift Detection<br/>Evidently · Whylogs"]
end
subgraph AIOps["AIOps / LLMOps (Phase 3)"]
RAG["RAG Stack<br/>vector DBs · embeddings"]
TRACE["LLM Tracing<br/>LangSmith · Phoenix · OpenLLMetry"]
GUARD["Guardrails<br/>output validation · PII filtering"]
end
K8S --> GPU & RAY & SERVE
LINUX --> GPU & NET
OBS --> TRACK & DRIFT & TRACE
CICD --> PIPE & RAG
In the diagram, Linux feeds into both GPU Scheduling and High-Throughput Networking. What do those two have in common that traces back to Linux specifically, rather than to Kubernetes?
Both sit on kernel-level primitives K8s doesn't own. GPU scheduling depends on
cgroups and device files to isolate and expose GPUs to containers
before K8s ever sees them as a schedulable resource. High-throughput networking
(RDMA/InfiniBand) depends on kernel-bypass mechanisms that skip the normal
network stack entirely. Kubernetes just orchestrates on top of both — the
actual isolation and bypass work is Linux's job.
Files
| File | Topics |
|---|---|
| gpu-scheduling.md | NVIDIA Device Plugin, extended resources, Dynamic Resource Allocation (ResourceClaim/DeviceClass), MIG slicing, GPU Operator, DCGM metrics, gang scheduling, taints/tolerations for GPU nodes |
| kuberay.md | KubeRay operator, RayCluster CRD, head/worker nodes, autoscaling, Ray Serve, resource requests |
| model-serving.md | KServe InferenceService, vLLM continuous batching, KV cache, canary rollouts, autoscaling with Knative |
| llmops.md | RAG architecture, vector DBs (pgvector, Milvus, Pinecone), LangSmith tracing, guardrails, cost optimization |
| networking.md | NVLink, InfiniBand, RoCE, AWS EFA, NCCL AllReduce, Cilium for inference, fat-tree topology |
Learning Path
Phase 1: AI Infrastructure — start here if you have a K8s background.
gpu-scheduling.md— understand how GPUs become K8s resourceskuberay.md— distributed Python workloads on K8smodel-serving.md— expose models as APIs at scale
Phase 2: MLOps — after Phase 1, see
../mlops/.
- experiment-tracking — MLflow / Weights & Biases
- training-pipelines — Kubeflow Pipelines / Airflow
Phase 3: LLMOps — concurrent with Phase 2, not after it.
llmops.md— RAG, vector DBs, LangSmith, guardrails
Why does Phase 3 (LLMOps) run concurrently with Phase 2 (MLOps) instead of waiting for it to finish?
LLMOps depends on Phase 1 — you need a served model before you can build
RAG or guardrails around it — but it doesn't depend on Phase 2's
experiment-tracking or training-pipeline maturity. Those two tracks address
different problems (training rigor vs. serving-time behavior) and neither
blocks the other, so there's no reason to sequence them.
Quick Orientation: AI Workload Types
| Workload | K8s resource shape | Key concern |
|---|---|---|
| Model training | Long-running Job, multi-GPU, gang scheduling | GPU utilization, checkpoint, fault tolerance |
| Batch inference | Job or CronJob, GPU optional | Throughput, cost |
| Online inference | Deployment + HPA, GPU required | Latency p99, KV cache size, queue depth |
| RAG pipeline | Stateless Deployment + vector DB | Embedding latency, retrieval accuracy |
| Fine-tuning | Job, 1–8 GPUs, hours to days | Data pipeline, checkpoint storage, resume |
Online inference lists "KV cache size" as a key concern; batch inference doesn't. Why the difference?
Online inference holds many concurrent user sessions' attention KV caches in
GPU memory at once, and that memory footprint directly caps how many
requests can be served concurrently — it's a hard ceiling on
throughput. Batch inference processes requests sequentially or in large
chunks without needing to keep many long-lived per-session caches resident
simultaneously, so raw throughput and cost dominate instead.
Key Difference from Standard K8s Workloads
Same cluster, two very different sets of assumptions. Flip between them:
- Requests: CPU + memory
- Autoscaling: HPA on CPU %
- Node placement: any node
- Deploy strategy: rolling update
- Metrics: Prometheus metrics
- Image size: container image ~100MB
- Requests: CPU + memory +
nvidia.com/gpu - Autoscaling: HPA on GPU utilization or queue depth (KEDA)
- Node placement: GPU node group with taint
nvidia.com/gpu=present:NoSchedule - Deploy strategy: canary with traffic split (KServe) or blue-green
- Metrics: Prometheus + DCGM Exporter (GPU metrics) + LLM token metrics
- Image size: model image ~5–70GB (use PVC or model storage instead)
Why can't a 70GB model just be baked into the container image the way a normal app's dependencies are?
A container image that size wrecks pull time on every pod start/restart and
node scale-up, and most container registries and node disks aren't sized
for it. Serving the model from a PVC or dedicated model-storage layer
instead means the image stays small and fast to pull, and the (large,
slow-changing) weights are fetched or mounted separately — and can be
shared across pods instead of duplicated per image layer.