Warp Tiling with FP16 and BF16: A CUDA GEMM Kernel
After my warp-tiling kernel reached
40.85 TFLOPS with FP32 inputs, I wanted to keep the same output ownership
model and change the storage type. The new file,
kernel_warptiling_multidtype.cuh, supports both FP16 and BF16 inputs and
outputs while keeping the dot products in FP32.
The tile geometry is unchanged: a block computes a 128 x 128 output tile,
each warp owns a 64 x 64 region, and each thread accumulates four 8 x 4
patches. The new part is the type layer around that mapping. Each thread loads
two 16-bit values through half2 or nv_bfloat162, converts operands to
FP32 for the multiply and add operations, and rounds back to the requested
type only at the final store.
The saved H100 run reached 31.40 TFLOPS for FP16 and 30.06 TFLOPS for BF16 at size 8192. That is only 4.3% of the measured FP16 cuBLAS throughput and 3.9% of the measured BF16 cuBLAS throughput. The gap is expected for this version because it uses ordinary FP32 arithmetic after conversion, while cuBLAS selects the H100 tensor core path for low-precision matrix multiplication.
Here is the story of what changed from the FP32 kernel, how the source maps 16-bit values through the same warp tile, and what the benchmark does and does not show.
The same multiplication, another data type
The operation is still:
C = alpha * A @ B + beta * C
A is M x K, B is K x N, and C is M x N. The benchmark wrapper
passes alpha and beta into the kernel, so this is not only the common
alpha=1 and beta=0 case.
The new file supports two storage types:
| Input and output | Compute accumulator | Pair type |
|---|---|---|
| FP16 | FP32 | half2 |
| BF16 | FP32 | nv_bfloat162 |
That distinction matters. FP16 and BF16 reduce storage and global-memory traffic, but their values are not used as the accumulator type. Every product is converted to FP32 before it updates the thread's partial sums. The final FP16 or BF16 conversion happens in the output epilogue.
The tile hierarchy stays intact
The public wrappers instantiate one configuration for both dtypes:
1 2 3 4
// BM, BN, BK, WM, WN, WNITER, TM, TN, NUM_THREADS
sgemm_warptiling_multidtype<
InputType, 128, 128, 16, 64, 64, 4, 8, 4, 128
>(matrix_a, matrix_b, output_matrix, alpha, beta);
The parameters mean:
- Block tile (
BM x BN): one block computes a128 x 128region ofC. - K tile (
BK): the block stages 16 columns of A and 16 rows of B at a time. - Warp tile (
WM x WN): each warp owns a64 x 64region of the block tile. - Warp iterations (
WNITER): four64 x 16subtiles cover one warp's64 x 64region. - Thread patch (
TM x TN): each thread computes an8 x 4patch in each subtile. - Threads per block: 128 threads form four warps.
The coverage still works out exactly:
4 warps * 64 * 64 outputs per warp = 16,384 outputs
128 threads * 4 subtiles * 8 * 4 values = 16,384 outputs
This is the important relationship with the previous FP32 post. The dtype change does not create a new warp mapping. It changes how values enter shared memory, how operands are converted in the compute loop, and how results leave the registers.
An 8 by 4 patch is still only part of a thread's work
The block first assigns each warp a position inside the 128 x 128 output
tile:
1 2 3
const uint warp_idx = threadIdx.x / WARP_SIZE;
const uint warp_col = warp_idx % (BN / WN);
const uint warp_row = warp_idx / (BN / WN);
With BN=128 and WN=64, two warp columns fit across the block. Threads 0 to
31 form warp 0 at the top left. Threads 32 to 63 form warp 1 at the top right.
Warps 2 and 3 cover the bottom half.
Inside the warp, the lane selects a position in each 64 x 16 subtile:
1 2 3 4 5 6 7 8
constexpr uint WMITER =
(WM * WN) / (WARP_SIZE * TM * TN * WNITER);
constexpr uint WSUBM = WM / WMITER;
constexpr uint WSUBN = WN / WNITER;
const uint thread_idx_in_warp = threadIdx.x % WARP_SIZE;
const uint thread_col_in_warp = thread_idx_in_warp % (WSUBN / TN);
const uint thread_row_in_warp = thread_idx_in_warp / (WSUBN / TN);
The default values give WMITER=1, WSUBM=64, and WSUBN=16. The lane
coordinates therefore cover (64 / 8) x (16 / 4) = 32 thread positions in
each subtile.
Each thread keeps four patches in a private FP32 array:
1
float thread_results[WMITER * TM * WNITER * TN] = {0.0f};
That is 1 x 8 x 4 x 4 = 128 accumulators. The values stay with the thread
through the complete K loop. The thread later visits the same patch positions
when it reads C, applies alpha and beta, and writes the result.
Reusing values across those patches
For one position along K, the thread loads eight A values and sixteen B values from shared memory into small register arrays. It then forms the outer product between those arrays. This is the same register reuse that made the preceding FP32 warp-tiling kernel faster than its one-output-per-thread version.
The relevant part of process_warp_tile is:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
for (uint dot_idx = 0; dot_idx < BK; ++dot_idx) {
for (uint wsub_row_idx = 0; wsub_row_idx < WMITER; ++wsub_row_idx) {
for (uint i = 0; i < TM; ++i) {
register_m[wsub_row_idx * TM + i] =
tile_a[dot_idx * BM + warp_row * WM +
wsub_row_idx * WSUBM +
thread_row_in_warp * TM + i];
}
}
for (uint wsub_col_idx = 0; wsub_col_idx < WNITER; ++wsub_col_idx) {
for (uint i = 0; i < TN; ++i) {
register_n[wsub_col_idx * TN + i] =
tile_b[dot_idx * BN + warp_col * WN +
wsub_col_idx * WSUBN +
thread_col_in_warp * TN + i];
}
}
The source then updates every output in the thread's four patches:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
for (uint wsub_row_idx = 0; wsub_row_idx < WMITER; ++wsub_row_idx) {
for (uint wsub_col_idx = 0; wsub_col_idx < WNITER; ++wsub_col_idx) {
for (uint res_idx_m = 0; res_idx_m < TM; ++res_idx_m) {
for (uint res_idx_n = 0; res_idx_n < TN; ++res_idx_n) {
const int result_idx =
(wsub_row_idx * TM + res_idx_m) * (WNITER * TN) +
wsub_col_idx * TN + res_idx_n;
thread_results[result_idx] +=
to_float(register_m[wsub_row_idx * TM + res_idx_m]) *
to_float(register_n[wsub_col_idx * TN + res_idx_n]);
}
}
}
}
}
At the source level, 24 shared-memory values produce 128 multiply-add
updates for each K position. This is the same outer-product idea as the FP32
kernel. The difference is that the values in register_m and register_n
are stored in the selected 16-bit input type, then converted to FP32 at the
multiply.
Loading two values into shared memory
The type-specific helpers keep the vector path explicit. The FP16 overloads
use half2, while the BF16 overloads use nv_bfloat162:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
__device__ __forceinline__ float to_float(half value) {
return __half2float(value);
}
__device__ __forceinline__ float to_float(nv_bfloat16 value) {
return __bfloat162float(value);
}
__device__ __forceinline__ half2 pack_vector(half first, half second) {
return __halves2half2(first, second);
}
__device__ __forceinline__ nv_bfloat162 pack_vector(
nv_bfloat16 first, nv_bfloat16 second) {
nv_bfloat162 value;
value.x = first;
value.y = second;
return value;
}
The cooperative loader selects the pair type at compile time:
1 2
using VecT = typename std::conditional<
std::is_same<InputType, half>::value, half2, nv_bfloat162>::type;
Matrix A is transposed as it enters shared memory. The global load is a pair of contiguous values, but the shared-memory layout makes the later K-step access convenient:
1 2 3 4 5 6 7 8
const VecT *src = reinterpret_cast<const VecT *>(
&matrix_a[(inner_row_a + offset) * num_cols_a + inner_col_a * 2]);
InputType first;
InputType second;
unpack_vector(*src, first, second);
tile_a[(inner_col_a * 2 + 0) * BM + inner_row_a + offset] = first;
tile_a[(inner_col_a * 2 + 1) * BM + inner_row_a + offset] = second;
B stays row-major in shared memory, so the vector can be copied directly:
1 2 3 4 5
const VecT *src = reinterpret_cast<const VecT *>(
&matrix_b[(inner_row_b + offset) * num_cols_b + inner_col_b * 2]);
VecT *dst = reinterpret_cast<VecT *>(
&tile_b[(inner_row_b + offset) * BN + inner_col_b * 2]);
*dst = *src;
Each K step stages 128 x 16 values from A and 16 x 128 values from B. The
values are half the width of FP32 values, but the tile still contains the same
number of matrix elements. The pairwise path reduces the number of scalar
load and store operations needed for those 16-bit elements.
The barriers remain essential. The first __syncthreads() prevents the compute
loop from reading an incomplete tile. The second prevents the next K step from
overwriting shared memory while another warp is still using the current tile.
The output epilogue rounds once
The accumulator stays in FP32 until the final write. For each pair of output
values, the epilogue reads the existing C values, applies alpha and beta,
converts both results to InputType, and stores the matching pair type:
1 2 3 4 5 6 7 8 9 10
InputType first = matrix_c_interim[output_idx + 0];
InputType second = matrix_c_interim[output_idx + 1];
first = from_float(alpha * thread_results[result_idx + 0] +
beta * to_float(first), InputType{});
second = from_float(alpha * thread_results[result_idx + 1] +
beta * to_float(second), InputType{});
VecT *dst = reinterpret_cast<VecT *>(&matrix_c_interim[output_idx]);
*dst = pack_vector(first, second);
This is the complete precision policy of the kernel:
| Pipeline stage | Type | What happens |
|---|---|---|
| Global memory inputs | FP16 or BF16 | A and B are loaded from contiguous CUDA tensors. |
| Shared memory tiles | FP16 or BF16 | The cooperative loader stages the selected storage type. |
| Register operands | FP16 or BF16, converted to FP32 | Each operand is converted before multiplication and addition. |
| Thread accumulators | FP32 | Partial sums remain in FP32 through the K loop. |
| Global memory output | FP16 or BF16 | The FP32 epilogue result is rounded to the requested output type. |
It is a useful distinction from a Tensor Core GEMM. The file supports low- precision storage and FP32 accumulation, but it does not issue tensor core matrix multiply instructions.
The dispatch wrapper defines the shape contract
The FP16 entry point selects half; the BF16 entry point selects
nv_bfloat16. Both call the same templated kernel:
1 2 3 4 5 6 7 8 9 10 11 12
void sgemm_warptiling_fp16(
const torch::Tensor &matrix_a,
const torch::Tensor &matrix_b,
torch::Tensor &output_matrix,
float alpha,
float beta) {
TORCH_CHECK(matrix_a.dtype() == torch::kFloat16 &&
matrix_b.dtype() == torch::kFloat16,
"A and B must both be float16");
sgemm_warptiling_multidtype<half, 128, 128, 16, 64, 64, 4, 8, 4, 128>(
matrix_a, matrix_b, output_matrix, alpha, beta);
}
Before launching, the templated wrapper checks CUDA placement, contiguity, two-dimensional shapes, matching dimensions, and output dtype. It also keeps the fast path tile aligned:
1 2 3 4 5 6
TORCH_CHECK(num_rows_a % BM == 0,
"Matrix A rows must be a multiple of ", BM);
TORCH_CHECK(num_cols_a % BK == 0,
"Matrix A columns must be a multiple of ", BK);
TORCH_CHECK(num_cols_b % BN == 0,
"Matrix B columns must be a multiple of ", BN);
For this instantiation, M and N must be divisible by 128, and K must be
divisible by 16. The loader does not zero-pad a tail. A 1000 x 1000 matrix
therefore needs padding or a different kernel path.
What the H100 run measures
I benchmarked the kernel on an NVIDIA H100 80GB HBM3, using CUDA 12.4 and
PyTorch 2.4.0. The extension is compiled with -O3 for 9.0a, and both
torch.backends.cuda.matmul.allow_tf32 and
torch.backends.cudnn.allow_tf32 are disabled.
The harness creates FP16 and BF16 inputs, warms both the custom kernel and
torch.matmul for 20 launches, then measures CUDA event time. It uses 100
iterations for sizes up to 1024, 50 iterations at 2048, 20 at 4096, and 8 at
8192. All benchmark shapes use M=N=K and satisfy the tile constraints.
The throughput calculation is:
FLOPs = 2 * size^3
TFLOPS = FLOPs / average_time_ms / 1e9
FP16 on H100
| Matrix Size (M = N = K) | Custom Kernel Time | Custom TFLOPS | cuBLAS Time | cuBLAS TFLOPS | % of cuBLAS |
|---|---|---|---|---|---|
| 512 | 0.0973 ms | 2.76 | 0.0144 ms | 18.63 | 14.8% |
| 1024 | 0.1864 ms | 11.52 | 0.0098 ms | 220.09 | 5.2% |
| 2048 | 0.6010 ms | 28.59 | 0.0290 ms | 593.31 | 4.8% |
| 4096 | 4.5582 ms | 30.15 | 0.2250 ms | 610.77 | 4.9% |
| 8192 | 35.0151 ms | 31.40 | 1.5194 ms | 723.66 | 4.3% |
BF16 on H100
| Matrix Size (M = N = K) | Custom Kernel Time | Custom TFLOPS | cuBLAS Time | cuBLAS TFLOPS | % of cuBLAS |
|---|---|---|---|---|---|
| 512 | 0.1068 ms | 2.51 | 0.0103 ms | 26.02 | 9.7% |
| 1024 | 0.2049 ms | 10.48 | 0.0221 ms | 97.08 | 10.8% |
| 2048 | 0.6493 ms | 26.46 | 0.0294 ms | 583.50 | 4.5% |
| 4096 | 4.7040 ms | 29.22 | 0.1971 ms | 697.30 | 4.2% |
| 8192 | 36.5767 ms | 30.06 | 1.4208 ms | 773.86 | 3.9% |
The tables come from one saved run, not a distribution across repeated H100
sessions. The reference uses torch.matmul with the same input dtype. At
size 8192, the custom kernel takes 35.0151 ms for FP16 and 36.5767 ms for
BF16. The matching cuBLAS calls take 1.5194 ms and 1.4208 ms.
The low percentage is not a contradiction in the arithmetic. The custom file converts 16-bit values and performs the dot product through ordinary FP32 operations. The cuBLAS reference can use H100 tensor core instructions for FP16 and BF16. This benchmark measures how far the current warp-tiled CUDA C++ path goes; it does not measure the low-precision performance ceiling of the GPU.
Correctness checks
The harness tests four aligned shapes for each dtype:
| Case | Shape | alpha | beta |
|---|---|---|---|
| Single block | 128 x 128 x 128 | 1.0 | 0.0 |
| Square with epilogue | 256 x 256 x 256 | 1.5 | 0.5 |
| Rectangular aligned | 512 x 768 x 256 | 1.0 | 0.0 |
| Multi-block aligned | 1024 x 640 x 512 | 0.75 | 0.0 |
The reference computes torch.matmul(a.float(), b.float()) and applies the
epilogue in FP32 before casting to the tested dtype. All eight checks pass with
rtol=2e-2 and atol=2e-2.
| Dtype | Largest absolute error | Result |
|---|---|---|
| FP16 | 0.03125 | All four passed |
| BF16 | 0.25 | All four passed |
The BF16 error is larger in absolute terms because BF16 has fewer mantissa bits than FP16. The tolerance is applied to the rounded output, so the result should be read as a dtype-aware correctness check rather than an FP32 exactness claim.
The boundaries of this version
This kernel is deliberately narrower than cuBLAS. It assumes dense, row-major, contiguous CUDA tensors with compatible alignment for pairwise 16-bit loads and stores. It does not handle arbitrary strides, transposed inputs, or edge tiles.
The benchmark also does not claim Tensor Core performance. The accumulator is
FP32, but the multiply and add operations in process_warp_tile are ordinary
CUDA arithmetic after conversion. The next implementation step would need a
different instruction path, likely through WMMA, MMA, or a higher-level kernel
library.
The large per-thread accumulator array is another tradeoff. Each thread owns 128 FP32 values before the epilogue. That creates reuse, but it can increase register pressure and reduce the number of resident blocks. I would measure compiler register usage, local-memory spills, achieved occupancy, and math pipeline utilization before changing the tile sizes.
The useful result of this version is the separation of concerns. The FP32 kernel's warp ownership and register tiling remain intact. The new type layer controls pair loads, conversions, and final stores. That gives the next optimization a clear baseline: improve the instruction path without changing the output mapping at the same time.
Code: My multi-dtype warp-tiling kernel on GitHub, along with the kernel snapshot and utils.cuh include used for this post.
Benchmark: The FP16 and BF16 H100 harness contains the compilation, correctness, timing, and JSON serialization logic. The saved benchmark JSON contains the device details and raw rows behind the tables.
Further reading: Simon Boehm's CUDA matmul worklog, NVIDIA's CUDA Programming Guide for execution and data types, and NVIDIA's CUDA C++ Best Practices Guide for memory and performance guidance.
About the author
Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.