Ritesh Yadav
Ritesh Yadav
/device-software/warp-tiling-gemm
GEMM

Warp Tiling: Taking FP32 GEMM to 40.85 TFLOPS on H100

After my vectorized kernel reached about 29 TFLOPS on H100, I kept the float4 load path and reorganized the output tile around warps. Each warp now owns a region of the output, with each thread reusing its register values across several smaller patches.

The new warp-tiling configuration reached 40.85 TFLOPS at size 8192, or 78.8% of the measured FP32 cuBLAS throughput. The comparison uses square, tile-aligned matrices with TF32 disabled. This version rejects dimensions that do not fit its tiles.

The same multiplication, another level of tiling

The operation is still C = alpha * A @ B + beta * C, where A is M × K and B is K × N. The inner dimension K is the length of each dot product.

In my 1D block-tiling kernel, each thread computed a vertical strip of outputs. 2D block tiling expanded that into a rectangular patch, allowing reuse of values from both matrices. The next step was to add a warp-sized region between the block tile and the thread's patches.

A warp contains 32 threads. My default configuration uses 128 threads, or four warps, per block to compute a 128 × 128 tile of C. Each warp owns a 64 × 64 quarter of that tile.

One block covers 128 by 128 outputs. Its four warps each own a separate 64 by 64 region.
One block covers 128 by 128 outputs. Its four warps each own a separate 64 by 64 region.

The thread index determines each warp's position:

CUDA
const uint warp_idx = threadIdx.x / WARP_SIZE;
const uint warp_col = warp_idx % (BN / WN);
const uint warp_row = warp_idx / (BN / WN);

With BN=128 and WN=64, there are two warp columns. Threads 0 to 31 form warp 0 at the top left; threads 32 to 63 form warp 1 at the top right. The remaining two warps cover the bottom half. A thread block runs on one SM, and all four warps use the block's shared-memory tiles.

The arithmetic stays on the ordinary FP32 path. The warp tile describes output ownership; it does not introduce Tensor Core instructions or warp shuffles.

An 8 by 4 patch is only part of a thread's work

For the default launch, I settled on these template parameters:

CUDA
// BM, BN, BK, WM, WN, WNITER, TM, TN, NUM_THREADS
sgemm_warptiling<128, 128, 16, 64, 64, 4, 8, 4, 128>(
    matrix_a, matrix_b, output_matrix, alpha, beta);

TM=8 and TN=4 describe one small patch. Dividing a warp's 64 × 64 output region into four 64 × 16 subtiles gives each thread one 8 × 4 patch in each subtile.

The subtile dimensions follow from the tile sizes and thread count:

CUDA
constexpr uint WMITER =
    (WM * WN) / (WARP_SIZE * TM * TN * WNITER);
constexpr uint WSUBM = WM / WMITER;
constexpr uint WSUBN = WN / WNITER;

Substituting the defaults gives WMITER=1, WSUBM=64, and WSUBN=16. Each subtile fits eight groups of thread rows and four groups of thread columns: (64 / 8) × (16 / 4) = 32 threads.

Inside the warp, the lane number chooses that thread's position:

CUDA
const uint thread_idx_in_warp = threadIdx.x % WARP_SIZE;
const uint thread_col_in_warp =
    thread_idx_in_warp % (WSUBN / TN);
const uint thread_row_in_warp =
    thread_idx_in_warp / (WSUBN / TN);

For warp 0, lane 0 computes rows 0 to 7 at columns 0 to 3, then 16 to 19, 32 to 35, and 48 to 51. Lane 1 keeps the same rows and moves four columns right in each subtile. Lane 4 starts the next eight-row group.

Each cell is an 8 by 4 output patch. Lane 0 owns the four highlighted cells across warp 0, for 128 outputs in total.
Each cell is an 8 by 4 output patch. Lane 0 owns the four highlighted cells across warp 0, for 128 outputs in total.

That gives each thread four patches, or 128 accumulators, not 32. The coverage checks out: 128 threads × 128 outputs = 16,384, exactly the block's 128 × 128 output tile. The patches are separated in matrix C; the thread keeps their partial sums together in its private accumulator array.

Reusing values across those patches

For one position along K, a thread loads eight A values and sixteen B values from shared memory into its register arrays. It multiplies every A value by every B value, updating 128 partial sums. This is an outer product, the same idea as the earlier 2D kernel, spread across four patches.

The accumulator index combines the row within a patch with its subtile column. This is the update from process_warp_tile, with line breaks adjusted:

CUDA
thread_results[
    (wsub_row_idx * TM + res_idx_m) * (WNITER * TN)
    + wsub_col_idx * TN + res_idx_n
] += register_m[wsub_row_idx * TM + res_idx_m]
   * register_n[wsub_col_idx * TN + res_idx_n];

At the source level, that is 24 shared-memory values for 128 multiply-add updates. My previous 8 × 8 thread tile used 16 values for 64 updates. Per output update, the new arrangement needs 25% fewer logical shared-memory operand reads. This count describes scalar values in the algorithm, not measured memory transactions or generated load instructions.

There is a cost: each thread now holds twice as many accumulators. The block also shrinks from eight warps to four, while BK increases from 8 to 16, growing the shared-memory staging buffers from 8 KiB to 16 KiB. These choices changed together, so my benchmark does not isolate warp mapping from tile depth or register pressure. Fewer threads per block does not establish better occupancy.

Loading and finishing a tile

The data still travels through global memory, shared memory, and then thread-local registers. Each K step stages 128 × 16 values from A and 16 × 128 from B. Together that is 4,096 FP32 values, or 16 KiB.

With 128 threads, each thread issues four float4 loads for A and four for B. Strided loading loops distribute the entire pair of tiles across the block. A lands in shared memory transposed as tile_a[k * BM + row], while B stays in tile_b[k * BN + col] order. These layouts match the addresses used in the compute loop.

The first __syncthreads() waits until the tiles are ready. The threads then compute through all 16 positions along K. A second block-wide barrier prevents a warp from overwriting shared memory while another warp is still reading it. The next iteration advances to the following 16 columns of A and 16 rows of B.

Each input has one shared-memory buffer, with loading and computation in separate phases. Double buffering and asynchronous copies remain possible next steps.

After the full dot products are accumulated, the kernel visits each thread's four patches, reads four existing C values, applies alpha and beta, and writes a float4 back. It still reads C when beta=0; the epilogue has no separate fast path for that case.

What the H100 run measures

I benchmarked the kernel on an NVIDIA H100 80GB HBM3, using CUDA 12.4 and PyTorch 2.4.0. Both inputs and outputs are FP32, with TF32 disabled. The reference is torch.matmul(..., out=...), which uses cuBLAS for these CUDA matrices.

The timing loop starts with 20 warmup iterations for both implementations, followed by repeated calls measured with CUDA events. Each entry below is the average latency from that loop, with 100 repetitions at the smallest sizes and eight at size 8192. All sizes use M=N=K. These results come from one saved run, without a distribution across repeated benchmark sessions.

SizeTime msTFLOPS% cuBLAS
5120.08173.2819.1%
10240.156513.7336.4%
20480.494534.7469.1%
40963.773336.4270.2%
819226.916440.8578.8%

Throughput is 2 × M × N × K operations divided by elapsed time. At size 8192, the cuBLAS baseline takes 21.2127 ms, so it remains faster than my kernel's 26.9164 ms. The 78.8% figure is relative to that measured baseline, not H100's advertised peak throughput.

At size 4096, my previous vectorized run delivered 28.83 TFLOPS and this run delivers 36.42, a throughput increase of about 26.3%. At 8192, it rises from 28.79 to 40.85 TFLOPS, about 41.9%. These are comparisons between saved runs of two configurations, rather than a controlled experiment that changes warp tiling alone.

I checked four correctness cases, including rectangular matrices and nonzero alpha and beta. All passed with rtol=1e-3 and atol=1e-3; the largest reported absolute error was about 9.54e-5. The benchmark JSON contains the timings, configuration, and correctness checks behind these results.

The boundaries of this version

For this version, I kept a stricter shape requirement: M and N divisible by 128, and K divisible by 16. The wrapper rejects other shapes instead of masking edge loads, giving up the tail handling in my preceding vectorized kernel. A 1000 × 1000 problem needs padding or another kernel.

The indexing assumes dense row-major, contiguous FP32 tensors on the same CUDA device, with pointers aligned for 16-byte float4 accesses. The wrapper checks several shape and dtype conditions, but does not validate all layout, alignment, or device-consistency assumptions. A larger application would need to enforce those conditions at the call site.

Next, I want to measure compiler register usage, spills, shared-memory bank conflicts, and achieved occupancy. Those measurements would help separate the benefit of additional register reuse from the cost of the larger per-thread state. Timing results alone do not answer those questions.

The useful part of this step is the ownership model: blocks own output tiles, warps own regions within those tiles, and threads accumulate several patches within their warp's region. Following one lane through those patches made the indexing much easier to work through.

Code: My warp-tiling kernel on GitHub, along with the kernel snapshot and utils.cuh include used for this post. The excerpts above focus on the default configuration; the file also instantiates two other configurations.

Further reading: Simon Boehm's CUDA matmul worklog, which the repository cites; NVIDIA's CUDA programming guide for warp execution and synchronization; and its vectorized-memory guide for alignment requirements.

About the author

Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.