AI Infra Summit Keynote - Advancing Innovation in AI Infrastructure

In my keynote at the NVIDIA AI Infra Summit (published by NVIDIA), I laid out how the industry must rethink and redesign compute, memory, networking, and software to meet an era of unprecedented AI demand. I walked the audience through the multi-dimensional trade-offs of inference, the software and numeric breakthroughs that are changing the economics of AI, and a set of new hardware and rack-level innovations — including a processor dedicated to long-context workloads — that together define the next generation of AI data centers.

Below I report what I presented, highlight key facts and examples, and provide context and explanation so that readers who weren’t there can understand the strategic choices operators, hyperscalers, and developers face. I wrote this as a concise but thorough news-style account that reflects both the urgency and the optimism driving the AI infrastructure community today.

📰 Executive summary

I opened with a clear premise: inference is complex, commercially critical, and highly multidimensional. While training attracts headlines, inference is the revenue engine of AI services and demands careful balancing of model intelligence, responsiveness, throughput, cost, and energy efficiency. Hardware and software must come together — at the chip, node, rack, and data center levels — and the community must collaborate openly to accelerate progress.

To meet these needs, I described advances we’ve delivered at NVIDIA (NVLink 72, Spectrum X Ethernet, NVFP4 numeric formats, Dynamo serving software, and our best-in-class MLPerf results) and introduced new products and designs: a Rubin CPX GPU optimized for very long context processing, the Vera Rubin rack family, and a community-oriented AI Factory Gigascale Reference Design for building future data centers.

To make the business case concrete: inference performance directly maps to revenue. Using current-generation racks and optimized software, we can demonstrate multiples of throughput and massive improvements in return on investment. I summarized the math and illustrated production examples from partners and customers who are already deploying these approaches.

🧭 Why inference deserves the spotlight

AI systems are commonly discussed in two phases: training and inference. Training is hard and headline-grabbing — it’s where we build capabilities — but inference is where models become products. I emphasized a simple but powerful idea: performance in inference equals revenue. Put another way: how fast and cost-effectively you can serve tokens to users directly impacts the business that an AI factory captures.

Inference has many axes and many trade-offs. To design infrastructure for inference, you must think across these dimensions concurrently:

Model intelligence: Larger models often generate more value, but they also cost more to run.
Responsiveness: How much compute do you dedicate per query to make experiences fast and interactive?
Cost: There is a wide range of hardware choices (H100, H200, B200, GB200, GB300, and beyond) and each has different cost-performance characteristics.
Throughput versus per-user speed: Spending heavily on a single query reduces capacity for other users; you need to balance per-user latency with aggregate throughput and revenue.
Energy efficiency: Data centers are measured in megawatts and gigawatts — power and cooling choices materially affect ROI.

Those trade-offs dictate what hardware to buy and how to configure it. They also determine the software strategies you must adopt and the roadmap you should plan for — which means making bets on technologies that may take years to materialize. I described that uncertainty as one of the most difficult parts of designing AI infrastructure: projecting performance-per-dollar and power two or three years ahead while models and algorithms continue evolving rapidly.

🔎 The SMART framework: how I think about AI infrastructure

To structure these choices, I used an internal acronym we call SMART. It’s a way to focus the engineering and product decisions required to deliver an AI factory:

S — Scale and complexity: What size of infrastructure are you building? Are you optimizing a single rack or an entire megawatt-scale data center? Scaling beyond a single chassis brings new challenges for networking, management, and reliability.
M — Multi-dimensional performance: Performance isn’t a single number. You must consider accuracy, tokens per second per user, tokens per second per rack or data center, and latency targets.
A — Architecture (full-stack): This is hardware + software. Chip architects, system designers, rack integrators, and software engineers must collaborate to produce a total solution.
R — Return on investment: For inference, throughput maps to revenue. Every optimization that increases tokens per dollar increases the revenue an AI factory can generate.
T — Together (community): The AI ecosystem is uniquely open. Researchers and companies publish their techniques; open-source projects like PyTorch, Triton, and others accelerate progress. Success is a communal effort.

Putting SMART into practice means designing chips and racks that are numerically efficient, building software that extracts those efficiencies in production, and collaborating on open standards and reference designs so data centers can be deployed at scale.

💡 Key hardware and networking innovations

Hardware changes slowly and is difficult to retrofit — silicon takes years to design and manufacture. That’s why anticipating where workloads are headed is vital. In my talk I explained several foundational hardware innovations that are enabling new scale and cost structures for inference.

NVLink 72 and rack-scale disaggregation

We started by rethinking the connectivity model for multi-GPU servers. Historically, GPUs were tightly coupled in an 8-GPU server using NVLink inside the box. We “disaggregated” that tight coupling and moved NVLink out to the rack level, allowing GPUs to be connected across a rack with a fabric that provides NVLink-level performance. NVLink 72 is the term we used to describe this rack-scale NVLink connectivity.

That disaggregation allows flexible placement of compute and supports new runtime models that dynamically allocate GPU resources to a model — which is especially useful in inference where workload shapes vary dramatically.

Spectrum X: an Ethernet built for AI

Traditional Ethernet was designed for client-server patterns, not for an environment in which “every GPU wants to talk to every GPU at full performance” all the time. To meet the needs of massively parallel AI workloads, we developed a different kind of Ethernet switch — Spectrum X — and a family of network silicon optimized for the all-to-all communication patterns emerging in large distributed training and inference.

This is an important point: as compute density rises, the network is no longer a convenience — it becomes the critical fabric that determines how well the system performs at scale. Network design for AI must consider determinism, congestion avoidance, and predictable low-latency all-to-all communication in ways that conventional datacenter Ethernet was not originally intended to provide.

🔬 Numerics: trading bits for performance

One of the most exciting vectors for performance improvement is numerical representation. Machine learning is fundamentally statistical — it doesn’t always need the same exacting numeric precision we used in older scientific computing disciplines. That lets us explore lower-precision formats that substantially increase throughput and reduce memory and bandwidth requirements.

History and progress we referenced:

FP32 was the initial standard for neural network computation.
Google introduced bfloat16 about a decade ago to make training faster without dramatically changing dynamic range semantics.
IEEE and the community have extended the conversation to even lower precision formats, culminating in operational support for FP8 and now 4-bit floating point variants.

We introduced a format we call NVFP4, a 4-bit floating point variant that includes algorithmic and hardware support for microtensor scaling and constant re-biasing to keep numerical values in range without catastrophic clipping. That combination of numeric format and hardware support lets us run inference — and increasingly training — in 4-bit floating point.

To be clear: moving to lower-precision arithmetic requires both hardware support and software algorithms. It’s not just a matter of fewer bits. You need accumulation paths, scaling logic, and numerics-aware kernels to preserve accuracy. We showed that NVFP4 is usable in production inference and that research is advancing toward training in 4-bit formats as well.

The upshot is straightforward: lower precision unlocks substantial efficiency gains, and those efficiency gains directly translate to more tokens served per dollar and per watt.

🧰 Software: the other half of the equation

Hardware alone does not make an AI data center. Software is the key to extracting value from silicon and racks. I emphasized that NVIDIA adopts a "GitHub-first" approach for many of its components and that open-source ecosystems like PyTorch, Triton, and other communities are essential to innovation.

Two software innovations I highlighted are particularly important:

Dynamo: Our disaggregated serving stack that orchestrates context processing and generation across multiple GPUs and nodes. Dynamo was built to enable scale-out inference, KV cache transfers, and the operational patterns needed to turn disaggregated hardware into predictable, high-performance services.
TensorRT and other runtime optimizations: Across generations of hardware (Ampere, Hopper, Blackwell), we’ve invested heavily in runtime kernels and libraries. Together with contributions from the community, these kernels have driven dramatic performance gains without changing the underlying silicon.

Software improvements have compounded hardware gains. Since Blackwell launched, we doubled its MLPerf performance with software alone. With Hopper, we ultimately produced a fourfold performance improvement over the product’s lifetime purely through software and algorithmic enhancements. Those are not incremental tweaks — they are material shifts in how models are served and yield direct increases in revenue for operators.

🏁 MLPerf: an objective way to measure inference

Claims about inference performance must be verifiable. The MLPerf benchmark has become the industry’s standard for inference measurement. Organizations across the ecosystem — Google, Meta, NVIDIA, and many others — collaborate, agree on workloads, and peer review published submissions.

Since MLPerf Inference started in 2019, NVIDIA has been an active participant. We’ve submitted results across multiple hardware generations and we hold the per-GPU records for many of the benchmarks. In the most recent MLPerf round, we added DeepSEA and Llama 3 140B and published results on Blackwell Ultra and GB300 using NVFP4, Dynamo, and the new TensorRT optimizations, distributed across the NVLink of a full GB300 rack. We used full rack-level distribution and advanced numerical techniques to both increase throughput and preserve accuracy.

Benchmarks are not an academic exercise — they tell a real business story: how much inference capacity you can deliver for a given dollar of infrastructure and how that capacity converts into revenue.

💰 The economics: performance equals revenue

I put the business math plainly in the keynote. Performance increases translate into revenue multipliers for an AI factory. To turn the abstract into something concrete, I used an example:

A $3 million GB200 NVL72 rack can produce roughly $30 million in token revenue over its useful life under reasonable assumptions.

That’s a 10x return — but it’s not magic; it comes from optimizing every layer: better numerics, better kernels, faster interconnects, software that disaggregates workloads intelligently, and rack-level designs that increase available compute and memory bandwidth per rack.

In practice, that math dramatically changes procurement decisions: paying more for a higher-performance platform can be the most economical choice, because the revenue you get from serving tokens scales with performance. In fact, I noted that “a free GPU is not even cheap enough” if its performance is only a fraction of a better platform. The cost of the rest of the rack, servers, shell, and operations matter, but the performance-per-dollar metric is what really drives ROI for inference-heavy services.

🔄 Inference is two workloads in one: context processing and generation

Understanding inference requires breaking it into two distinct workload phases: context processing (prefill) and autoregressive generation (decode). Both use the same model, but they stress hardware and software in different ways.

Context processing:

Ingests the user query plus any system prompt, memory, or related tokens (context).
Can be parallelized: multiple input tokens can be processed simultaneously.
Is compute-rich and benefits from massive parallelism and high aggregate compute.

Autoregressive generation:

Produces the output tokens one after another; each generated token becomes the input to compute the next token.
Is latency-sensitive and depends more on memory bandwidth, NVLink/PCIe latency, and kernels optimized for sequential decoding.
Can require different parallelization and kernel strategies than prefill.

If you force both workloads onto the same static configuration, you end up with compromises. Instead, I explained how modern data centers are increasingly disaggregating inference by running prefill and generator workloads on different GPU pools optimized for their respective phases.

Disaggregation is not theoretical. It’s a production pattern. You run context processing on one set of GPUs, produce a KV cache (a compact representation of the context), and hand that cache off to a second set of GPUs optimized for fast token generation. This delivers much higher aggregate throughput and faster time-to-first-token than a single homogeneous allocation.

⚙️ Dynamo and KV cache orchestration

Dynamo is our open-source stack that coordinates this disaggregation. It is GitHub-first, meaning development and improvements are visible in public repositories and the community contributes to and benefits from the work in near real-time. Dynamo manages the orchestration of prefill and decode kernels, the efficient transfer and placement of KV caches, and the scheduling policies required to keep GPUs busy.

Empirical results speak loudly: when we ran models like Llama and DeepSeq with disaggregation using Dynamo, we observed:

First-token latency improvements on some workloads by up to 6x.
Throughput improvements of 2x–4x on many models simply by splitting prefill and generation onto different optimized pools.

These gains are not small. They multiply across fleets, resulting in materially greater revenue per rack and per data center.

🏗️ Introducing Rubin CPX: a processor optimized for long contexts

One of the major announcements I made was a new Rubin CPX GPU — a Rubin-derived processor dedicated to the unique needs of very long context processing. Here’s why this matters.

Use cases such as advanced code generation and video processing increasingly require models to ingest extremely long input sequences. Models today can accept 256K tokens and beyond, and specialized models are beginning to handle millions of tokens. Two immediate, high-value examples:

Advanced coding agents: Instead of helping with small snippets, these agents can take entire repositories (hundreds of thousands of tokens or more) and produce substantial code contributions. The amplification effect on developer productivity can be on the order of 10x for some tasks.
AI video processing and generation: Processing hours of HD footage and producing new content requires ingesting and reasoning over very large input sequences. Video generation is already a multi-billion-dollar market and is expected to balloon in the coming years.

For these workloads, context processing is the bottleneck. You need massive compute to perform attention across extremely long sequences, and the algorithms benefit from specialized attention acceleration and memory architecture.

The Rubin CPX is designed for that mission. Key attributes I announced:

It’s a Rubin-family GPU optimized specifically for massive context-length processing.
It delivers over 30 petaflops (NVFP4) of AI performance and remains CUDA-capable.
We tripled attention-processing throughput relative to the current GB300 design by adding specialized attention acceleration cores.
It is memory-optimized (suited to GDDR7), because context compute is compute-rich and less dependent on ultra-high HBM bandwidth.
It includes four video encoders and four decoders to support AI video processing pipelines.

Availability: the CPX variant will start to become available toward the end of 2026, and it’s designed to integrate tightly with rack-level products like Vera Rubin.

🗄️ Vera Rubin rack and CPX integration

Rubin is not just a chip; it’s a family of systems and racks. I described Vera Rubin — a rack architecture we introduced at GTC — and how CPX processors integrate into it.

Highlights of Vera Rubin racks:

Each compute tray contains four Rubin GPUs, two Vera CPUs, and ConnectX9 networking.
In a single rack, we can deliver over 3.6 exaflops of AI performance with the initial Vera Rubin design.
Vera Rubin provides 75 TB of fast memory and 1.4 PB of HBM4 in the base configuration, making it an extremely dense AI platform.

To support long-context scenarios, we can add CPX processors into the same rack in multiple ways:

Insert CPX modules directly into compute trays to form a Vera Rubin NVL144 CPX variant that increases compute to roughly 8 exaflops and boosts memory and fast memory capacity by multiple times.
Offer a CPX-only tray (VRCPX) with two Vera CPUs and eight CPX processors, which can be installed side-by-side with standard Vera Rubin racks to create a flexible mix of context and generation capacity.

The KV cache handoff model means these trays and racks do not need to be physically adjacent — the first-token cache can be sent over the network to token generators anywhere in the data center. That flexibility gives operators multiple deployment options and the ability to scale context and generation capacity independently.

🔗 The ecosystem: customers and partners already building

These are not hypothetical improvements. Lighthouse customers and partners are already experimenting and building with these ideas:

Base10: An inference aggregator with thousands of GPUs using Dynamo and disaggregation to deliver fast inference across multiple clouds. They were early to leverage disaggregation and saw top-tier performance across the major open models.
Cursor: A leader in intelligent code generation that benefits from larger context windows for repository-scale code assistance.
Magic.dev: Working on 100M token input models that will require enormous context-processing capacity.
Runway: Focused on cinematic AI video generation and early partners for video workloads.
Fireworks and Together AI: Providers working on the fastest model-serving architectures and advanced inference techniques.

These companies illustrate two critical points: first, the business value of long-context and video workloads is real and large; second, the ecosystem is already converging on disaggregated serving patterns and rack-level MWlink-like fabrics to make those workloads practical.

🏗️ Building the future data center: the AI Factory Gigascale Reference Design

All of these chips and racks must be assembled into data centers designed for the new power, cooling, and mechanical realities of AI. This is a generational infrastructure problem: you can’t retrofit a CPU-centric data center overnight to become an efficient AI factory.

To help solve that, I announced the AI Factory Gigascale Reference Design — an initiative to develop open, community-driven reference architectures for future AI data centers. NVIDIA does not do this alone. We’re working closely with a broad array of partners and engineering firms to create a practical blueprint:

Cadence, ETAP, GE (Verona), Jacobs: system and electrical engineering design partners
Schneider Electric, Siemens, Vertiv: partners for CDUs, power distribution, and thermal systems
Emerald AI and others: collaborators on intelligent control and optimization

The reference design addresses row length, bus bar distribution, power delivery, cooling infrastructure (CDUs and liquid cooling options), and operational tooling to provide predictable efficiency and uptime at gigascale. The goal is to produce a data center that is future-proofed for multiple generations of AI racks (Vera Rubin, Vera Rubin Ultra, and beyond to Feynman and other architectures), and to publish the first version of the reference design at our next GTC.

This is a practical, engineering-first approach: GPUs, CPUs, switches, and racks must interoperate with power and cooling systems; tools and telemetry must be in place to maintain availability and efficiency; and the industry must converge on shared best practices so deployments scale confidently across the world.

🔍 Measured results: software doubles and quadruples hardware over time

One of the most dramatic points I shared is how much software can reshape the effective performance of a given piece of hardware over its lifetime:

From the time a product ships, continuous software development and community contributions often double or quadruple its usable performance for inference.
Blackwell: since launch, we achieved a 2x MLPerf performance improvement through software and numeric improvements alone.
Hopper: over its lifetime we increased Hopper’s effective inference throughput roughly fourfold through software optimizations, new kernels, and better serving strategies.

That means purchasing decisions should consider the entire lifecycle of a platform — not just its day-one spec sheet. The combination of better numerics, improved kernels, disaggregated serving, and kernel-level tricks can significantly increase your tokens-per-dollar over time. This makes investing in platforms that provide headroom for software improvements particularly compelling.

📈 Real-world business impact: quantifying the upside

Let me restate the central business insight: inference performance is revenue. I presented a simple but powerful comparison to prove the point. If you compare a more expensive, higher-performance rack to a cheaper alternative that delivers a fraction of the throughput, the higher-performance rack will return more revenue over time. The example I used — a $3M GB200 rack generating roughly $30M in token revenue — demonstrates a 10x return on the infrastructure purchase under plausible price and usage assumptions.

When you model the cost structure of running an AI service (infrastructure capex, server shells, networking, operational expenses, power and cooling), performance-per-dollar becomes the leverage point for profitability. Inference operators who prioritize performance per watt per dollar win in the marketplace.

🔁 Community and openness: why this era is different

One theme I reiterated was the breadth and depth of community contribution. This era of AI is less siloed than prior computer revolutions. Researchers publish methods, companies release models and datasets, and open-source projects mature rapidly. Open ecosystems accelerate innovation because improvements are shared and iterate faster.

Examples include the flourishing developer and research work around PyTorch, Triton, JAX, and other frameworks; the open exchange of techniques to reduce precision or accelerate attention; and collaborative benchmarking through MLPerf. This openness means advances in numerics or serving patterns propagate quickly and deliver measurable benefits across the whole industry.

🧩 Putting it all together: an architecture view

The final image I painted was holistic: AI infrastructure is a system-of-systems. It’s not just GPUs — it’s CPUs, network switches, NICs, interconnect switch chips (MVLink / NVLink), memory choices (HBM vs GDDR), numeric formats (FP32, bfloat16, FP8, NVFP4), and software stacks that orchestrate workloads across disaggregated resources.

Key elements that must be designed to operate together:

Chip family: Rubin GPUs (Rubin, Rubin CPX), Mellanox/ConnectX NICs, Spectrum X switches, and the other network ASICs that make all-to-all communication feasible.
Rack and tray architectures: Vera Rubin trays and the option to add CPX trays for scale and flexibility.
Software stack: Dynamo, TensorRT and other runtimes, and the open-source frameworks that deliver kernels and serve models.
Facility design: power distribution, cooling, and anything that matters at megawatt scale captured in the AI Factory Gigascale Reference Design.

Success requires careful orchestration across hardware, software, and facilities engineering along with a vibrant community that continues to push algorithms and systems forward.

🔭 Looking forward: the roadmap and next steps

We are not done innovating. Rubin CPX and Vera Rubin racks are the next step in an ongoing roadmap that includes even more ambitious architectures. I mentioned that we will continue to expand the Rubin family, and hinted that future generations (like Feynman in our product roadmap) will bring further improvements. The strategy remains consistent: build a family of chips and systems that operate together, enable open software, and help customers extract the best-in-class performance-per-dollar and performance-per-watt.

Practically, my recommendations for operators and builders were:

Think about inference as a revenue-generating workload and model your ROI on tokens-per-dollar and tokens-per-watt, not just raw capex.
Adopt disaggregated serving patterns where appropriate; use KV cache handoff to maximize first-token latency and aggregate throughput.
Invest in software and keep an eye on numeric innovations like NVFP4; these can materially change your total cost of ownership.
Plan your facility roadmap with future racks and power density in mind; collaborate with ecosystem partners to adopt reference designs that scale to gigawatts.
Engage with the community — open-source contributions and joint benchmarking accelerate everyone’s progress.

📣 Closing observations and a call to action

In closing, building AI infrastructure at the scale required today is a brave, complicated endeavor. It demands collaboration across hardware designers, system architects, software engineers, facilities designers, and customers. But the opportunity is enormous: with the right stack and the right decisions, inference scales from an operational cost center into a revenue-generating machine.

I closed the keynote by underscoring two convictions: first, performance in inference equals revenue — it’s the core metric for data-center operators; second, progress is a community activity. We’ll continue to innovate across numerics, interconnects, and software, and we’ll publish reference architectures so the broader industry can build future-proof, efficient AI factories.

For those building AI services: model your economics on end-to-end performance, test disaggregated serving in production, and work with partners to design your next data center with power density and cooling headroom. For researchers and software engineers: continue to push numeric techniques and serving designs into production and publish your results so the community benefits. For hyperscalers and infrastructure providers: factor in long-term software momentum — hardware gets better with software improvements — and design for upgradeability.

We’re only at the beginning of this infrastructure evolution. The combination of new numerical formats like NVFP4, disaggregated NVLink fabrics, specialized processors like Rubin CPX for long-context workloads, and community-driven software like Dynamo will reshape what inference means for businesses and consumers over the coming years. I look forward to continuing the work with all of you and to seeing the next wave of deployments that make AI more capable, more efficient, and more broadly useful.

📚 Appendix: quick reference of technologies and partners mentioned

Hardware and chips: H100, H200, B200, GB200, GB300, Blackwell, Blackwell Ultra, Rubin (Rubin family), Rubin CPX, ConnectX9, Spectrum X.
Numerics: FP32, bfloat16, FP8, NVFP4 (4-bit floating point with microtensor re-biasing and scaling).
Software: Dynamo (open-source), TensorRT, Triton, PyTorch, JAX.
Benchmarks: MLPerf Inference (industry-standard, peer-reviewed benchmark).
Partners and customers: Base10, Cursor, Magic.dev, Runway, Fireworks, Together AI, Cadence, ETAP, GE Verona, Jacobs, Schneider Electric, Siemens, Vertiv.
Programs: AI Factory Gigascale Reference Design (community-driven reference for data center design).

📌 Final note

This article reports the substance of the keynote I gave at the NVIDIA AI Infra Summit and aims to provide practical insight into why inference matters, how to shape infrastructure choices, and which innovations are likely to pay off. If you’re building inference services or designing AI data centers, your decisions over the next 12–36 months will have long-term implications. I encourage you to test disaggregated serving patterns, plan for long context workloads, and engage with the open community — together we can accelerate the next wave of AI infrastructure.