Ritesh Yadav
Ritesh Yadav
/device-software/warp-tiling-multidtype-gemm
GEMM

Warp Tiling with FP16 and BF16: A CUDA GEMM Kernel

After my warp-tiling kernel reached 40.85 TFLOPS with FP32 inputs, I wanted to keep the same output ownership model and change the storage type. The new file, kernel_warptiling_multidtype.cuh, supports both FP16 and BF16 inputs and outputs while keeping the dot products in FP32.

The tile geometry is unchanged: a block computes a 128 x 128 output tile, each warp owns a 64 x 64 region, and each thread accumulates four 8 x 4 patches. The new part is the type layer around that mapping. Each thread loads two 16-bit values through half2 or nv_bfloat162, converts operands to FP32 for the multiply and add operations, and rounds back to the requested type only at the final store.

The saved H100 run reached 31.40 TFLOPS for FP16 and 30.06 TFLOPS for BF16 at size 8192. That is only 4.3% of the measured FP16 cuBLAS throughput and 3.9% of the measured BF16 cuBLAS throughput. The gap is expected for this version because it uses ordinary FP32 arithmetic after conversion, while cuBLAS selects the H100 tensor core path for low-precision matrix multiplication.

Here is the story of what changed from the FP32 kernel, how the source maps 16-bit values through the same warp tile, and what the benchmark does and does not show.

The same multiplication, another data type

The operation is still:

C = alpha * A @ B + beta * C

A is M x K, B is K x N, and C is M x N. The benchmark wrapper passes alpha and beta into the kernel, so this is not only the common alpha=1 and beta=0 case.

The new file supports two storage types:

Input and outputCompute accumulatorPair type
FP16FP32half2
BF16FP32nv_bfloat162

That distinction matters. FP16 and BF16 reduce storage and global-memory traffic, but their values are not used as the accumulator type. Every product is converted to FP32 before it updates the thread's partial sums. The final FP16 or BF16 conversion happens in the output epilogue.

The tile hierarchy stays intact

The public wrappers instantiate one configuration for both dtypes:

CUDA
// BM, BN, BK, WM, WN, WNITER, TM, TN, NUM_THREADS
sgemm_warptiling_multidtype<
    InputType, 128, 128, 16, 64, 64, 4, 8, 4, 128
>(matrix_a, matrix_b, output_matrix, alpha, beta);

The parameters mean:

  • Block tile (BM x BN): one block computes a 128 x 128 region of C.
  • K tile (BK): the block stages 16 columns of A and 16 rows of B at a time.
  • Warp tile (WM x WN): each warp owns a 64 x 64 region of the block tile.
  • Warp iterations (WNITER): four 64 x 16 subtiles cover one warp's 64 x 64 region.
  • Thread patch (TM x TN): each thread computes an 8 x 4 patch in each subtile.
  • Threads per block: 128 threads form four warps.

The coverage still works out exactly:

4 warps * 64 * 64 outputs per warp = 16,384 outputs
128 threads * 4 subtiles * 8 * 4 values = 16,384 outputs

This is the important relationship with the previous FP32 post. The dtype change does not create a new warp mapping. It changes how values enter shared memory, how operands are converted in the compute loop, and how results leave the registers.

An 8 by 4 patch is still only part of a thread's work

The block first assigns each warp a position inside the 128 x 128 output tile:

CUDA
const uint warp_idx = threadIdx.x / WARP_SIZE;
const uint warp_col = warp_idx % (BN / WN);
const uint warp_row = warp_idx / (BN / WN);

With BN=128 and WN=64, two warp columns fit across the block. Threads 0 to 31 form warp 0 at the top left. Threads 32 to 63 form warp 1 at the top right. Warps 2 and 3 cover the bottom half.

Inside the warp, the lane selects a position in each 64 x 16 subtile:

CUDA
constexpr uint WMITER =
    (WM * WN) / (WARP_SIZE * TM * TN * WNITER);
constexpr uint WSUBM = WM / WMITER;
constexpr uint WSUBN = WN / WNITER;

const uint thread_idx_in_warp = threadIdx.x % WARP_SIZE;
const uint thread_col_in_warp = thread_idx_in_warp % (WSUBN / TN);
const uint thread_row_in_warp = thread_idx_in_warp / (WSUBN / TN);

The default values give WMITER=1, WSUBM=64, and WSUBN=16. The lane coordinates therefore cover (64 / 8) x (16 / 4) = 32 thread positions in each subtile.

Each thread keeps four patches in a private FP32 array:

CUDA
float thread_results[WMITER * TM * WNITER * TN] = {0.0f};

That is 1 x 8 x 4 x 4 = 128 accumulators. The values stay with the thread through the complete K loop. The thread later visits the same patch positions when it reads C, applies alpha and beta, and writes the result.

Reusing values across those patches

For one position along K, the thread loads eight A values and sixteen B values from shared memory into small register arrays. It then forms the outer product between those arrays. This is the same register reuse that made the preceding FP32 warp-tiling kernel faster than its one-output-per-thread version.

The relevant part of process_warp_tile is:

CUDA
for (uint dot_idx = 0; dot_idx < BK; ++dot_idx) {
    for (uint wsub_row_idx = 0; wsub_row_idx < WMITER; ++wsub_row_idx) {
        for (uint i = 0; i < TM; ++i) {
            register_m[wsub_row_idx * TM + i] =
                tile_a[dot_idx * BM + warp_row * WM +
                       wsub_row_idx * WSUBM +
                       thread_row_in_warp * TM + i];
        }
    }

    for (uint wsub_col_idx = 0; wsub_col_idx < WNITER; ++wsub_col_idx) {
        for (uint i = 0; i < TN; ++i) {
            register_n[wsub_col_idx * TN + i] =
                tile_b[dot_idx * BN + warp_col * WN +
                       wsub_col_idx * WSUBN +
                       thread_col_in_warp * TN + i];
        }
    }

The source then updates every output in the thread's four patches:

CUDA
    for (uint wsub_row_idx = 0; wsub_row_idx < WMITER; ++wsub_row_idx) {
        for (uint wsub_col_idx = 0; wsub_col_idx < WNITER; ++wsub_col_idx) {
            for (uint res_idx_m = 0; res_idx_m < TM; ++res_idx_m) {
                for (uint res_idx_n = 0; res_idx_n < TN; ++res_idx_n) {
                    const int result_idx =
                        (wsub_row_idx * TM + res_idx_m) * (WNITER * TN) +
                        wsub_col_idx * TN + res_idx_n;
                    thread_results[result_idx] +=
                        to_float(register_m[wsub_row_idx * TM + res_idx_m]) *
                        to_float(register_n[wsub_col_idx * TN + res_idx_n]);
                }
            }
        }
    }
}

At the source level, 24 shared-memory values produce 128 multiply-add updates for each K position. This is the same outer-product idea as the FP32 kernel. The difference is that the values in register_m and register_n are stored in the selected 16-bit input type, then converted to FP32 at the multiply.

Loading two values into shared memory

The type-specific helpers keep the vector path explicit. The FP16 overloads use half2, while the BF16 overloads use nv_bfloat162:

CUDA
__device__ __forceinline__ float to_float(half value) {
    return __half2float(value);
}

__device__ __forceinline__ float to_float(nv_bfloat16 value) {
    return __bfloat162float(value);
}

__device__ __forceinline__ half2 pack_vector(half first, half second) {
    return __halves2half2(first, second);
}

__device__ __forceinline__ nv_bfloat162 pack_vector(
    nv_bfloat16 first, nv_bfloat16 second) {
    nv_bfloat162 value;
    value.x = first;
    value.y = second;
    return value;
}

The cooperative loader selects the pair type at compile time:

CUDA
using VecT = typename std::conditional<
    std::is_same<InputType, half>::value, half2, nv_bfloat162>::type;

Matrix A is transposed as it enters shared memory. The global load is a pair of contiguous values, but the shared-memory layout makes the later K-step access convenient:

CUDA
const VecT *src = reinterpret_cast<const VecT *>(
    &matrix_a[(inner_row_a + offset) * num_cols_a + inner_col_a * 2]);

InputType first;
InputType second;
unpack_vector(*src, first, second);
tile_a[(inner_col_a * 2 + 0) * BM + inner_row_a + offset] = first;
tile_a[(inner_col_a * 2 + 1) * BM + inner_row_a + offset] = second;

B stays row-major in shared memory, so the vector can be copied directly:

CUDA
const VecT *src = reinterpret_cast<const VecT *>(
    &matrix_b[(inner_row_b + offset) * num_cols_b + inner_col_b * 2]);
VecT *dst = reinterpret_cast<VecT *>(
    &tile_b[(inner_row_b + offset) * BN + inner_col_b * 2]);
*dst = *src;

Each K step stages 128 x 16 values from A and 16 x 128 values from B. The values are half the width of FP32 values, but the tile still contains the same number of matrix elements. The pairwise path reduces the number of scalar load and store operations needed for those 16-bit elements.

The barriers remain essential. The first __syncthreads() prevents the compute loop from reading an incomplete tile. The second prevents the next K step from overwriting shared memory while another warp is still using the current tile.

The output epilogue rounds once

The accumulator stays in FP32 until the final write. For each pair of output values, the epilogue reads the existing C values, applies alpha and beta, converts both results to InputType, and stores the matching pair type:

CUDA
InputType first = matrix_c_interim[output_idx + 0];
InputType second = matrix_c_interim[output_idx + 1];

first = from_float(alpha * thread_results[result_idx + 0] +
                       beta * to_float(first), InputType{});
second = from_float(alpha * thread_results[result_idx + 1] +
                        beta * to_float(second), InputType{});

VecT *dst = reinterpret_cast<VecT *>(&matrix_c_interim[output_idx]);
*dst = pack_vector(first, second);

This is the complete precision policy of the kernel:

Pipeline stageTypeWhat happens
Global memory inputsFP16 or BF16A and B are loaded from contiguous CUDA tensors.
Shared memory tilesFP16 or BF16The cooperative loader stages the selected storage type.
Register operandsFP16 or BF16, converted to FP32Each operand is converted before multiplication and addition.
Thread accumulatorsFP32Partial sums remain in FP32 through the K loop.
Global memory outputFP16 or BF16The FP32 epilogue result is rounded to the requested output type.

It is a useful distinction from a Tensor Core GEMM. The file supports low- precision storage and FP32 accumulation, but it does not issue tensor core matrix multiply instructions.

The dispatch wrapper defines the shape contract

The FP16 entry point selects half; the BF16 entry point selects nv_bfloat16. Both call the same templated kernel:

CUDA
void sgemm_warptiling_fp16(
    const torch::Tensor &matrix_a,
    const torch::Tensor &matrix_b,
    torch::Tensor &output_matrix,
    float alpha,
    float beta) {
    TORCH_CHECK(matrix_a.dtype() == torch::kFloat16 &&
                    matrix_b.dtype() == torch::kFloat16,
                "A and B must both be float16");
    sgemm_warptiling_multidtype<half, 128, 128, 16, 64, 64, 4, 8, 4, 128>(
        matrix_a, matrix_b, output_matrix, alpha, beta);
}

Before launching, the templated wrapper checks CUDA placement, contiguity, two-dimensional shapes, matching dimensions, and output dtype. It also keeps the fast path tile aligned:

CUDA
TORCH_CHECK(num_rows_a % BM == 0,
            "Matrix A rows must be a multiple of ", BM);
TORCH_CHECK(num_cols_a % BK == 0,
            "Matrix A columns must be a multiple of ", BK);
TORCH_CHECK(num_cols_b % BN == 0,
            "Matrix B columns must be a multiple of ", BN);

For this instantiation, M and N must be divisible by 128, and K must be divisible by 16. The loader does not zero-pad a tail. A 1000 x 1000 matrix therefore needs padding or a different kernel path.

What the H100 run measures

I benchmarked the kernel on an NVIDIA H100 80GB HBM3, using CUDA 12.4 and PyTorch 2.4.0. The extension is compiled with -O3 for 9.0a, and both torch.backends.cuda.matmul.allow_tf32 and torch.backends.cudnn.allow_tf32 are disabled.

The harness creates FP16 and BF16 inputs, warms both the custom kernel and torch.matmul for 20 launches, then measures CUDA event time. It uses 100 iterations for sizes up to 1024, 50 iterations at 2048, 20 at 4096, and 8 at 8192. All benchmark shapes use M=N=K and satisfy the tile constraints.

The throughput calculation is:

FLOPs = 2 * size^3
TFLOPS = FLOPs / average_time_ms / 1e9

FP16 on H100

Matrix Size (M = N = K)Custom Kernel TimeCustom TFLOPScuBLAS TimecuBLAS TFLOPS% of cuBLAS
5120.0973 ms2.760.0144 ms18.6314.8%
10240.1864 ms11.520.0098 ms220.095.2%
20480.6010 ms28.590.0290 ms593.314.8%
40964.5582 ms30.150.2250 ms610.774.9%
819235.0151 ms31.401.5194 ms723.664.3%

BF16 on H100

Matrix Size (M = N = K)Custom Kernel TimeCustom TFLOPScuBLAS TimecuBLAS TFLOPS% of cuBLAS
5120.1068 ms2.510.0103 ms26.029.7%
10240.2049 ms10.480.0221 ms97.0810.8%
20480.6493 ms26.460.0294 ms583.504.5%
40964.7040 ms29.220.1971 ms697.304.2%
819236.5767 ms30.061.4208 ms773.863.9%

The tables come from one saved run, not a distribution across repeated H100 sessions. The reference uses torch.matmul with the same input dtype. At size 8192, the custom kernel takes 35.0151 ms for FP16 and 36.5767 ms for BF16. The matching cuBLAS calls take 1.5194 ms and 1.4208 ms.

The low percentage is not a contradiction in the arithmetic. The custom file converts 16-bit values and performs the dot product through ordinary FP32 operations. The cuBLAS reference can use H100 tensor core instructions for FP16 and BF16. This benchmark measures how far the current warp-tiled CUDA C++ path goes; it does not measure the low-precision performance ceiling of the GPU.

Correctness checks

The harness tests four aligned shapes for each dtype:

CaseShapealphabeta
Single block128 x 128 x 1281.00.0
Square with epilogue256 x 256 x 2561.50.5
Rectangular aligned512 x 768 x 2561.00.0
Multi-block aligned1024 x 640 x 5120.750.0

The reference computes torch.matmul(a.float(), b.float()) and applies the epilogue in FP32 before casting to the tested dtype. All eight checks pass with rtol=2e-2 and atol=2e-2.

DtypeLargest absolute errorResult
FP160.03125All four passed
BF160.25All four passed

The BF16 error is larger in absolute terms because BF16 has fewer mantissa bits than FP16. The tolerance is applied to the rounded output, so the result should be read as a dtype-aware correctness check rather than an FP32 exactness claim.

The boundaries of this version

This kernel is deliberately narrower than cuBLAS. It assumes dense, row-major, contiguous CUDA tensors with compatible alignment for pairwise 16-bit loads and stores. It does not handle arbitrary strides, transposed inputs, or edge tiles.

The benchmark also does not claim Tensor Core performance. The accumulator is FP32, but the multiply and add operations in process_warp_tile are ordinary CUDA arithmetic after conversion. The next implementation step would need a different instruction path, likely through WMMA, MMA, or a higher-level kernel library.

The large per-thread accumulator array is another tradeoff. Each thread owns 128 FP32 values before the epilogue. That creates reuse, but it can increase register pressure and reduce the number of resident blocks. I would measure compiler register usage, local-memory spills, achieved occupancy, and math pipeline utilization before changing the tile sizes.

The useful result of this version is the separation of concerns. The FP32 kernel's warp ownership and register tiling remain intact. The new type layer controls pair loads, conversions, and final stores. That gives the next optimization a clear baseline: improve the instruction path without changing the output mapping at the same time.

Code: My multi-dtype warp-tiling kernel on GitHub, along with the kernel snapshot and utils.cuh include used for this post.

Benchmark: The FP16 and BF16 H100 harness contains the compilation, correctness, timing, and JSON serialization logic. The saved benchmark JSON contains the device details and raw rows behind the tables.

Further reading: Simon Boehm's CUDA matmul worklog, NVIDIA's CUDA Programming Guide for execution and data types, and NVIDIA's CUDA C++ Best Practices Guide for memory and performance guidance.

About the author

Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.