Scaling and Optimizing Your LLM Pipeline for End-to-End Efficiency

Brandon Royal, Product Manager, AI on Google Kubernetes Engine, Google
Sam Stoelinga, Cloud Architect, Google

Are you having trouble getting language models (LLMs) to work in your organization? You're not alone. We'll look at how to deploy an open-source language model on GKE. We'll show data scientists and machine learning engineers how to use NeMo and TRT LLM with GKE's notebooks. Plus, GKE has a unique ability to help orchestrate AI workloads with efficiency and convenience. We'll also demonstrate how to train and tune a language model using NeMo and do a live technical demo of how data science teams can infer these models on GPUs with TRT LLM and GKE.

Developer Resources:

  • Training:
    • Sizing LLM Inference Systems
    • Introduction to Deploying RAG Pipelines for Production at Scale
  • Technical Blog:
    • Optimizing Inference Efficiency for LLMs at Scale with NVIDIA NIM Microservices | August 2024
    • Practical Strategies for Optimizing LLM Inference Sizing and Performance | August 2024
    • Scaling LLMs with NVIDIA Triton and NVIDIA TensorRT-LLM Using Kubernetes | October 2024
    • LLM Inference Benchmarking: Fundamental Concepts | April 2025
    • Turbocharge LLM Training Across Long-Haul Data Center Networks with NVIDIA Nemo Framework | May 2025
  • Training: Running Your Own LLM

Events & Trainings:

GTC San Jose
Date: March 2024
Topic: AI Inference
Industry: All Industries
Level: Technical – Beginner
Language: English
NVIDIA technology: Cloud / Data Center GPU
Region: USA, China, Japan, Korea, Taiwan

Join Brad Genereaux and experts from Anthropic, Google, and Microsoft for a panel discussion about the arrival of generative AI. This is a major moment in technology history. AI foundation models are now causing a similar change in healthcare.

Join us to learn how 60,000 organizations around the world are developing and deploying generative AI with NVIDIA on Microsoft Azure. Today, Microsoft Azure offers the largest and most complete model library in the market, including the latest models from OpenAI, Meta.

Explore the game-changing generative AI solutions provided by NVIDIA and Microsoft Azure. Benefit from secure cloud infrastructure and top-tier AI services to speed up the creation and deployment of AI applications.

The network defines the data center, and a high-performance network is key to successfully training and deploying AI workloads such as foundational models, LLMs, and generative AI at scale. The networking must provide high effective bandwidth, low tail-latency, and…

Learn how NVIDIA and Microsoft Azure partner to provide businesses with an end-to-end platform for building and deploying generative AI applications, including pre-trained models, workflow tools, and immediate access to multi-node GPU infrastructure, accelerating the…

AI is revolutionizing the visual effects (VFX) industry and is now ubiquitous across film, TV, animation, and graphics. However, the perception of how AI is used within the media & entertainment (M&E) industry has been murky. The reality is that studios today are already…

The AI boom has fundamentally changed the demands of data centers and cloud compute providers. Companies are now seeing a total redesign of the data center and more specialized cloud service providers. CoreWeave is vastly knowledgeable about the high-powered…

This hands-on lab guides developers through the process of creating a cloud-based IoT application using NVIDIA Omniverse Kit and Microsoft Azure. Participants will learn to develop, deploy, and integrate a containerized Omniverse Kit application with a web-based…

In the rapidly evolving world of AI, companies face challenges in incorporating AI into their applications due to the need for rapid upskilling and adapting to new models. Learn how NVIDIA NIM, provided as containerized microservices, offer cutting-edge AI models.