NVIDIA’s GPU Technology Conference kicked off this week, and Jensen Huang delivered his keynote from what I can only assume is the most expensive kitchen in Silicon Valley. The leather jacket is back, the ambition is cranked to eleven, and the announcements paint a clear picture of where NVIDIA thinks computing is heading.
The headline grabber is Grace, NVIDIA’s first datacenter CPU — an ARM-based processor designed specifically for AI workloads. But there’s a lot more under the surface: the A30 and A10 GPUs for mainstream inference, NVIDIA Base Command and Fleet Command for managing AI infrastructure, and a raft of software platform updates. Let’s unpack what actually matters for developers and infrastructure teams.
Grace: NVIDIA’s ARM Bet#
The Grace CPU is named after Grace Hopper (a choice I fully endorse), and it represents NVIDIA’s first serious foray into designing datacenter CPUs. It’s ARM-based, built on ARMv9, and optimized specifically for large-scale AI training workloads where the bottleneck is moving data between CPU and GPU memory.
The key technical claim: Grace will use LPDDR5x memory with a unified memory architecture that provides 10x the bandwidth of today’s NVIDIA DGX systems for CPU-GPU data transfer. For training massive models — the kind of thing that’s becoming standard in NLP — the CPU-to-GPU memory pipeline is increasingly the constraint, not raw GPU compute.
This is a strategic move on multiple levels. First, it reduces NVIDIA’s dependency on Intel and AMD for the CPU side of their AI platforms. Second, it positions NVIDIA in the ARM server ecosystem alongside Ampere Computing, AWS Graviton, and Fujitsu’s A64FX. Third, it lets NVIDIA optimize the entire system — CPU, GPU, memory, interconnect — as a single design, much like Apple’s M1 approach but for datacenter AI.
Grace won’t ship until 2023, so this is a long-term signal rather than something to plan infrastructure around today. But it tells you where NVIDIA sees the industry going: tightly integrated, heterogeneous compute platforms purpose-built for AI workloads.
The A30 and A10: AI For the Rest of Us#
While Grace grabbed the headlines, the A30 and A10 GPUs are arguably more relevant for most organizations today. These are mainstream datacenter GPUs aimed at the inference market — the part of the AI pipeline that runs trained models in production.
The A30 offers 24GB of HBM2e memory with multi-instance GPU (MIG) support, letting you partition a single GPU into multiple isolated instances. For inference serving, this is significant: you can run multiple models or serve multiple tenants on a single GPU without interference.
The A10 targets both inference and graphics workloads, making it a versatile option for organizations that need both AI serving and virtual desktop infrastructure. At 24GB GDDR6, it’s positioned as a cost-effective step up from T4 cards.
What I find interesting about these announcements is the clear message: NVIDIA is moving beyond selling GPUs to selling AI platforms. The hardware is increasingly just the foundation for a software ecosystem that includes CUDA, TensorRT, Triton Inference Server, and now the management tools (Base Command, Fleet Command) that enterprises need to operationalize AI.
Software Platform: Triton and Beyond#
Speaking of software, the Triton Inference Server updates deserve attention. Triton is NVIDIA’s open-source inference serving framework, and it’s becoming genuinely good. Version 2.8 adds support for running on CPUs (not just GPUs), model ensembles, and improved auto-scaling.
For teams that are deploying ML models in production, the inference serving layer is often the most painful part of the stack. You’ve got your beautifully trained model, and now you need to serve it with low latency, handle batching efficiently, manage model versions, and scale appropriately. Triton handles most of this, and the fact that it’s open-source makes it a viable option even if you’re not running NVIDIA hardware for everything.
The pattern I’m seeing across the industry — from NVIDIA’s Triton to TensorFlow Serving to Seldon Core — is that ML inference serving is maturing into a proper infrastructure category. Two years ago, most teams were hand-rolling Flask APIs around their models. The tooling has gotten dramatically better.
What This Means for Developers#
If you’re a developer who doesn’t work directly with AI infrastructure, you might be wondering why you should care about GPU announcements. Here’s why: the infrastructure that NVIDIA is building isn’t just for training GPT-3 clones. It’s increasingly relevant for any application that benefits from AI-powered features.
Real-time recommendation engines, natural language processing, computer vision, anomaly detection — these capabilities are moving from “specialized AI team” territory into mainstream application development. The hardware and software platforms being announced at GTC are what make that transition possible at reasonable cost.
The move toward purpose-built AI infrastructure also has implications for cloud costs. Today, renting GPU instances on AWS, Azure, or GCP is expensive. As dedicated inference hardware like the A30 and A10 becomes widely available, and as software like Triton makes it easier to share GPUs across workloads, the cost per inference will continue to drop.
My Take#
NVIDIA’s GTC keynote was impressive but also a bit overwhelming — Jensen announced enough products and platforms to fill a week of presentations, compressed into a two-hour kitchen tour. The company’s ambition is clear: they want to own the entire AI computing stack, from silicon to software platform.
Whether that’s good for the industry depends on your perspective. NVIDIA’s CUDA ecosystem has been incredibly enabling, but it’s also created significant vendor lock-in. The Grace CPU move extends that potential lock-in from GPUs to the entire server platform.
For now, though, the practical takeaway is this: if you’re running AI inference workloads, the tooling and hardware options are better and cheaper than they were a year ago, and that trend is accelerating. If you’re not running AI workloads yet but you’re building applications that could benefit from them, the barrier to entry is dropping fast. That’s worth paying attention to, regardless of how you feel about leather jackets and kitchen keynotes.