State of AI Infrastructure 2026: Mid-Year Reality Check

We published our annual infrastructure state report in January. It’s only been three months, but enough moved that a mid-year check is warranted. The gap between January projections and April reality tells a more honest story about this market than any retrospective could.

We work across dozens of production AI deployments — from a single A10G serving a startup’s API to multi-thousand-GPU fleets at hyperscale. What follows is what we actually see running, what surprised us, and where the second half is going. Not press releases. Ground truth.

---

Hardware: B200 Is Real, Expensive, and Supply-Constrained

When we published in January, B200 was shipping but the pricing and availability picture was foggy. It’s clearer now.

Cloud pricing has stabilized around a wide band. On-demand B200 ranges from $3.49/hr at Lambda to $14.24/hr at AWS, with reserved rates falling as low as $2.65/hr on CoreWeave. The 3-4x spread between providers is narrowing but hasn’t closed. Inworld reports cost-per-token on B200 at roughly $0.02 per million tokens versus $0.14 on H100 — a 7x improvement (Inworld AI, April 2026).

But availability is the real story. An estimated 3.6 million B200 units are in the backlog. Direct hardware purchases involve multi-month wait times; cloud rental is the only fast path into production.

B300 is arriving faster than expected. Blackwell Ultra B300 with 288GB HBM4 is now available on several neoclouds (Nebius, RunPod) at $24–$26/hr on-demand. The additional 96GB of memory matters for serving trillion-parameter models on a single NVLink domain. See our B200 vs H100 upgrade analysis for the full sizing math.

Three-generation fleets are the default. We no longer see customers on a single GPU type for production. The pattern is A100 for small-model inference and legacy workloads, H100 for 70B-class serving, and B200 allocated exclusively to the highest-throughput hot paths. Routing models to the cheapest GPU that meets the latency target is the new default placement strategy.

---

AMD Is No Longer an Afterthought

MI300X has crossed the “credible production option” threshold. Our consulting book now shows ~25% of new deployments include AMD GPUs, up from near-zero entering 2024. ROCm’s software story is effectively equivalent to CUDA for mainstream workloads — vLLM, PyTorch, and HuggingFace all run cleanly.

Where MI300X (and its successor MI325X) wins is clear: 192GB of HBM3e per GPU, pricing 15–25% below H100 on reserved terms, and strong per-token economics for large-model inference. The remaining friction is multi-node training scaling and ecosystem depth for cutting-edge features.

For most inference workloads in 2026, the choice between H100 and MI300X is now a financial optimization, not a technical gamble. More on this in our MI300X vs H100 comparison.

---

The Serving Layer: vLLM Still Leads, SGLang Goes Commercial

vLLM Remains the Default

vLLM holds ~60% market share in our deployment data. PagedAttention solved memory fragmentation, continuous batching is rock-solid, and the ecosystem breadth (model support, quantization paths, framework integration) is unmatched.

The latest releases add Gemma4 support, quantized MoE, ROCm 7.2.1, Torch 2.10, and Triton 3.6. Disaggregated prefill/decode is production-ready. vLLM’s Q2 2026 roadmap focuses on multi-model serving, advanced scheduling, and better structured-output performance. See our full vLLM deep dive.

SGLang Commercializes at $400M

The biggest serving-layer story of 2026: the SGLang team from UC Berkeley spun out as RadixArk with a reported $400M valuation, backed by Accel (TechCrunch, January 2026). RadixArk is commercializing what was already the fastest open-source inference engine.

Independent benchmarks place SGLang at ~16,200 tokens/second on H100 for Llama 3.1 8B — roughly 29% ahead of vLLM’s 12,500 tok/s on the same hardware (Prem AI blog, February 2026). That throughput gap translates to roughly $15,000 in monthly GPU savings at a million requests per day.

SGLang’s RadixAttention excels at multi-turn conversations with heavy prefix reuse — exactly the pattern that agent workloads produce. We now recommend SGLang as the first option to evaluate for agent-heavy, chat-first, and prefix-reuse workloads, with vLLM still the default for broad compatibility and simpler deployments.

The inference engine competition is the single most consequential infrastructure story in 2026. What we’re seeing is a healthy ecosystem: vLLM owns breadth, SGLang owns agent throughput, LMDeploy owns quantized-model speed, and TensorRT-LLM owns raw throughput ceiling for teams that invest the engineering hours.

---

Agent Infrastructure Is Finally Its Own Category

This was our boldest January prediction, and it’s already arrived. 51% of enterprises now run AI agents in production (Ringly, April 2026), and the infrastructure to support those agents has diverged sharply from traditional LLM serving.

The differences are concrete and measurable:

• LLM call amplification — a single user request produces 5–50 LLM calls internally. Cost per request has wide variance.
• Stateful long-running work — agents run for minutes or hours with state that must survive restarts, pauses, and worker preemption.
• Trajectory-level observability — traces branch, merge, pause for human input, and span thousands of spans per task.
• Workflow engines as core infra — Temporal, Inngest, DBOS, and LangGraph’s checkpointer are now table stakes for non-trivial agent systems.

We wrote the full treatment in Agent Infrastructure: What’s Different from LLM Serving. In that post, we showed the reference architecture with the orchestrator as the central new component. Three months later, that architecture is the blueprint most of our new clients are following.

What’s new since January: managed agent platforms (LangGraph Cloud, Bedrock Agents) have moved from beta to general availability. MCP is now supported by every major framework. And the first wave of agent FinOps tooling — tracking full-trajectory cost per user task — is shipping.

---

What’s Still Broken

Not everything is working. Three pain points come up in almost every engagement this quarter:

1. Multi-Region Agent Consistency

Running a stateless inference endpoint across 5 regions is solvable. Running an agent fleet across those same regions with consistent behavior, synchronized context stores, and identical tool registries? Harder. Context drift between regions causes agents to behave differently for identical prompts. This is the top infrastructure complaint from our enterprise clients right now.

2. Model Deprecation Whiplash

OpenAI, Anthropic, and Google deprecate older model versions aggressively. An app hard-coded to gpt-4-0613 or claude-3-sonnet-20240229 will break without a model-aliasing layer. Teams are solving this with gateway-level model routing (route claude-sonnet to whatever the current Sonnet version is) plus eval-based regression testing before each switch. Our Model Evals guide covers the testing side.

3. Agent Debugging at Scale

When an agent fails after 47 LLM calls, 23 tool invocations, and a 12-minute pause for human approval — finding the root cause is still an art. Observability tools have improved, but the fundamental complexity of debugging long-lived, branching, multi-tool workflows remains unsolved.

---

Pricing Reality: APIs Keep Getting Cheaper, Self-Hosting Gets Harder to Justify

The token pricing collapse of 2024–2025 continued through Q1 2026. The frontier model APIs that cost $10–$30 per million tokens two years ago now cost $0.25–$5. Gemini 2.5 Flash, GPT-4.1 mini, and Claude Haiku-class models all sit below $1/M input tokens.

The breakeven for self-hosting keeps moving up. What used to be justified at 10M tokens/day now requires 50–100M+ tokens/day for most open-weight LLMs. The strongest remaining self-host arguments are data privacy, fine-tune requirements, sub-100ms latency targets, and sovereignty compliance — not raw cost.

---

Three Patterns Defining H2 2026

Based on what we’re seeing in production today:

1. Disaggregated Inference Goes Mainstream

Separating prefill (compute-bound) and decode (bandwidth-bound) onto different node pools is no longer experimental. vLLM, SGLang, and TRT-LLM all ship production-grade support. The 30–50% throughput gains on long-prompt workloads are real and measurable. Teams running 70B+ models with 8K+ token contexts will see immediate wins. See our disaggregated inference explainer.

2. Multi-LoRA Per Tenant

Production systems need per-tenant fine-tunes without per-LoRa dedicated GPUs. vLLM’s multi-LoRA support has matured to the point where dozens of adapters can be loaded and dynamically swapped at serve time on a single GPU cluster. This is the architecture powering personalized agent behavior at scale.

3. Context Engineering as a Discipline

The “context stack” — the combination of vector DB, full-text search, graph store, KV cache, and working memory — is now a named architectural concern. Our context engineering deep dive from early 2026 captured this shift. What’s new: pgvectorscale is gaining real traction against dedicated vector databases, and graph stores (Neo4j, Memgraph) are being added to RAG pipelines to handle entity relationships that flat vector embeddings can’t represent.

---

The Numbers That Matter

Some ground-truth benchmarks from our deployments and verified external sources:

Metric	Q1 2025	Q2 2026	Change
H100 on-demand	$2.50–$4.00/hr	$2.00–$3.00/hr	-25%
B200 on-demand	—	$3.49–$14.24/hr	new
Cost/M tokens (70B self-host, H100 FP8)	$0.12–$0.18	$0.08–$0.14	-40%
Cost/M tokens (B200 FP4)	—	~$0.02	—
SGLang throughput (H100, Llama 3.1 8B)	~14k tok/s	~16.2k tok/s	+16%
vLLM throughput (H100, Llama 3.1 8B)	~11k tok/s	~12.5k tok/s	+14%
Enterprises running agents in production	~30%	51%	+21 pts

Sources: Inworld AI, Prem AI, Ringly, and turion.ai engagement data.

---

What This Means for You

If you’re running an H100 fleet today: Don’t rush to migrate. H100 pricing keeps dropping and the performance-to-cost ratio is excellent for 70B-class inference. Add FP8, prefix caching, and continuous batching before you consider hardware changes.

If you’re planning a new fleet in Q3: Evaluate B200, but only at reserved pricing. The on-demand premium is still too high to justify for general workloads. Put MI300X in the competitive mix — the gap between ROCm and CUDA has narrowed to the point where it’s a financial question, not a technical one.

If you’re building agent infrastructure: Start with the orchestrator. Pick Temporal, Inngest, or LangGraph with a Postgres checkpointer before you write agent logic. Add MCP-compatible tool registries from day one. Instrument with OTel + Langfuse. See our multi-agent orchestration guide for the full stack.

If you’re managing AI spend now: The agent cost-per-task is unpredictable by design. Implement per-session LLM call caps, gateway-level rate limiting per tenant, and trajectory-level cost attribution. Our AI FinOps playbook covers this in detail.

---

Further Reading

• The AI Infrastructure Stack: 2026 Edition
• The State of AI Infrastructure 2025
• Agent Infrastructure: What’s Different from LLM Serving
• vLLM: The Open-Source Inference Engine Changing LLM Serving
• NVIDIA B200 vs H100: Should You Upgrade?
• Disaggregated Inference: Prefill, Decode, and the New Serving Topology
• Multi-Agent Orchestration Infrastructure: Lessons from Production
• AI FinOps: Tracking Token Spend Across Your Org
• Model Evals in Production: Regression Testing Prompts
• Context Engineering: Storage, Retrieval, and the New Memory Stack

Building or scaling AI infrastructure in 2026? Let’s talk — we help shops from first deployment to global multi-region scale.