Introduction to Agentic Inference
Agentic inference is a type of artificial intelligence that involves using agents to make decisions and take actions. These agents can be used in a variety of applications, including natural language processing, computer vision, and robotics. However, agentic inference workloads often exhibit unique characteristics, such as write-once-read-many KV cache access patterns, which can be challenging to optimize for.
To address these challenges, we are introducing Agent-aware inference scheduling mechanism in Dynamo, enabled via agent hints such as priority and latency sensitivity. This allows us to optimize the performance and efficiency of agentic inference workloads.
Agentic inference at scale is a complex problem, but NVIDIA Dynamo is well-equipped to handle it. With its ability to handle high KV cache pressure and provide full-stack optimizations, Dynamo is the perfect tool for optimizing agentic inference workloads.
Tools like Claude Code and Codex make hundreds of API calls per session and are able to achieve a KV cache hit rate of up to 97% while using Dynamo.
Optimizing KV Cache Management
One of the key challenges in optimizing agentic inference workloads is managing the KV cache. The KV cache is a critical component of the inference pipeline, as it stores the data that is used to make predictions. However, the KV cache can become a bottleneck if it is not managed properly.
To address this challenge, we are optimizing the KV cache management in Dynamo. This includes implementing a cache reuse rate maximization strategy, which ensures that the KV blocks are kept warm and routable. We are also implementing a cross-worker sharing mechanism, which allows multiple workers to share the same KV blocks.
By optimizing the KV cache management, we can significantly improve the performance and efficiency of agentic inference workloads. This is especially important in applications where low latency and high throughput are critical, such as real-time natural language processing and computer vision.
The cache needs to understand block value, support cross-worker sharing, and respect agent lifecycle boundaries. By providing these features, we can ensure that the KV cache is optimized for agentic inference workloads.
97%
KV cache hit rate
1000+
number of API calls per session
Frontend API and Router Optimizations
In addition to optimizing the KV cache management, we are also optimizing the frontend API and router in Dynamo. This includes implementing an agent-aware inference scheduling mechanism, which allows us to optimize the performance and efficiency of agentic inference workloads.
The frontend API is responsible for receiving requests from the client and forwarding them to the inference pipeline. By optimizing the frontend API, we can reduce the latency and improve the throughput of the inference pipeline.
The router is responsible for routing the requests from the frontend API to the appropriate worker. By optimizing the router, we can ensure that the requests are routed efficiently and that the workers are utilized effectively.
By optimizing the frontend API and router, we can significantly improve the performance and efficiency of agentic inference workloads. This is especially important in applications where low latency and high throughput are critical, such as real-time natural language processing and computer vision.
Any harness can attach structured hints to a request across all three API endpoints, giving the router and runtime the context they need to make agent-aware scheduling and caching decisions.
Conclusion and Future Work
In conclusion, optimizing agentic inference workloads is a complex problem that requires a comprehensive approach. By optimizing the KV cache management, frontend API, and router in Dynamo, we can significantly improve the performance and efficiency of agentic inference workloads.
In the future, we plan to continue optimizing Dynamo for agentic inference workloads. This includes implementing new features and techniques, such as agent-aware inference scheduling and cache reuse rate maximization.
We believe that Dynamo has the potential to become a leading platform for agentic inference workloads, and we are committed to continuing to optimize and improve it. With its ability to handle high KV cache pressure and provide full-stack optimizations, Dynamo is the perfect tool for optimizing agentic inference workloads.
Agentic inference at scale is a challenging problem, but with Dynamo, we can make it more efficient and effective. By providing a comprehensive platform for agentic inference workloads, we can enable developers to build more sophisticated and efficient AI systems.
How this compares
How this compares
| Component | Open / This Approach | Proprietary Alternative |
|---|---|---|
| Model provider | Any — OpenAI, Anthropic, Ollama | Single vendor lock-in |
| Inference engine | Dynamo, TensorFlow, PyTorch | Custom-built engines |
🔑 Key Takeaway
NVIDIA Dynamo is being optimized for agentic inference workloads, addressing the write-once-read-many KV cache access patterns seen in tools like Claude Code and Codex. By maximizing cache reuse rate across all workers and keeping KV blocks warm and routable, we can significantly improve performance and efficiency.
Key Links