What is a CUDA Graph?
A CUDA Graph lets the CPU prepare a workflow of GPU operations once and launch it repeatedly with less submission overhead.
Why is that useful?
The CPU, also called the host, tells the GPU what work to run. A kernel is a function that runs on the GPU. Submitting each kernel takes some CPU and driver work, called launch overhead.
For a long-running kernel, that overhead may be small compared with the computation. For many tiny kernels repeated thousands of times, it can become a meaningful cost.
Think of giving someone a reusable recipe instead of explaining every step again each time they cook.
How does it work?
A graph records operations as nodes and their dependencies as edges. For example, a calculation must finish before another operation can use its result.
There are three steps:
- Define: Describe the operations and dependencies directly, or record them using CUDA's stream capture API.
- Instantiate: CUDA validates the graph and prepares an executable version.
- Replay: Launch that prepared workflow repeatedly with a graph launch, reducing repeated setup and submission work.
The GPU still performs the computation on each replay. New input values can produce new results; the graph reuses the execution plan.
Where does it help?
LLM decoding is one example: generating successive tokens repeats many GPU operations. Reducing CPU launch work can help when submission overhead is significant.
Graph preparation has an upfront cost, so reuse matters. Changes to shapes or operations may need a supported graph update or a new graph. A workload already dominated by long GPU computations may gain little.
CUDA Graphs make repeated work cheaper to submit. They do not automatically merge kernels into one or remove the underlying computation.
References
About the author
Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.