Ritesh Yadav
Research Notes
Inference Engineering
Notes from production inference work, serving models, especially large language models, efficiently under real traffic: latency, throughput, and cost per token.
Posts
- Prefill and Decode, how prompt processing, KV caching, and sequential generation shape inference performance.
- Prefill-Decode Disaggregation, separating workers, moving KV state, and weighing latency isolation against transfer costs in vLLM and SGLang.
- Speculative Decoding Failure Study, my SGLang experiment, the draft paths that failed, the observed NGRAM speedup, and the measurement bugs that limited the conclusions.
- Single DGX Spark Inference, memory budgeting and serving optimizations on a constrained inference node.
- LLM Serving on Kubernetes, following requests through LiteLLM, Envoy, llm-d, and vLLM, with KServe lifecycle management, GPU capacity, autoscaling, and observability.
- TurboQuant Attention in CUDA, computing raw scores directly from compressed keys by reusing query transforms.
- FlashAttention in Triton, streaming key/value tiles, understanding online softmax, and measuring a first forward kernel against PyTorch SDPA on an H100.
About the author
Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.