Ritesh Yadav
Ritesh Yadav
/inference-engineering

Inference Engineering

Notes from production inference work, serving models, especially large language models, efficiently under real traffic: latency, throughput, and cost per token.

Posts

  • Prefill and Decode, how prompt processing, KV caching, and sequential generation shape inference performance.
  • Prefill-Decode Disaggregation, separating workers, moving KV state, and weighing latency isolation against transfer costs in vLLM and SGLang.
  • Speculative Decoding Failure Study, my SGLang experiment, the draft paths that failed, the observed NGRAM speedup, and the measurement bugs that limited the conclusions.
  • Single DGX Spark Inference, memory budgeting and serving optimizations on a constrained inference node.
  • LLM Serving on Kubernetes, following requests through LiteLLM, Envoy, llm-d, and vLLM, with KServe lifecycle management, GPU capacity, autoscaling, and observability.
  • TurboQuant Attention in CUDA, computing raw scores directly from compressed keys by reusing query transforms.
  • FlashAttention in Triton, streaming key/value tiles, understanding online softmax, and measuring a first forward kernel against PyTorch SDPA on an H100.

About the author

Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.