CUDA Types and Intrinsics for ML Kernels
When I look at a CUDA kernel, I usually start with a very ordinary question: where does the value change shape? It may start as a half precision weight in global memory, move through shared memory, become an FP32 accumulator, and end as a half precision output. The multiplication itself is often the easy part. The decisions around that multiplication are where most of the bugs live.
One value might begin as an FP8 activation in global memory, be unpacked into a pair or a small vector, converted to FP16 or BF16, accumulated in FP32, and finally written back in a lower precision format. A different kernel might keep everything in FP32 but use a warp shuffle to reduce a partial sum. Another one might use an atomic instead of a barrier because its producers are not in the same block.
CUDA gives each boundary a small API. The names are compact, but they are not
interchangeable. __half2 is a storage and arithmetic type. float4 is a four
field structure that can describe a memory movement. A bit reinterpretation is
not a numeric conversion. __syncwarp() is not a replacement for
__syncthreads().
This article is a field guide to the built-in types, conversions, and device intrinsics that appear most often in CUDA C++ kernels for machine learning and large language models. It follows the headers and groups in the CUDA Math API, but keeps the main path practical: what the group means, where it belongs, what to show in code, and what to check before trusting it.
The exact names change with the CUDA Toolkit and target architecture. I use the
CUDA Math API Reference Manual
and the CUDA C++ Programming Guide
to check a particular header, compiler, and -arch value.
The four questions I ask first
Before choosing a CUDA type or intrinsic, I try to answer four questions:
- How is the value stored? Global memory bandwidth, shared memory capacity, and communication format often favor FP16, BF16, FP8, or a packed integer representation.
- Where is the arithmetic performed? A low precision input does not imply a low precision accumulator. Many useful kernels load in a compact format, convert to FP32, and accumulate in FP32.
- Who needs to see the result? A value shared by lanes in one warp needs a different primitive from a value shared by all threads in a block or by unrelated blocks.
- Which hardware can execute it? Some names are available in the header but require a minimum compute capability, a newer Toolkit, or a particular instruction path.
The rest of this guide is easier to use if these questions stay visible. A function name is only useful when its storage format, scope, precision, and hardware contract are also understood.
One small example
Here is the simplest mixed precision story to keep in mind while reading the rest of the article. The values are stored as FP16, promoted to FP32 for the calculation, and converted back only when the result is stored.
| Step | Small CUDA example | What it means in plain language |
|---|---|---|
| Store a compact value | __half a = __float2half(1.5f); | Keep the input in a smaller 16 bit format. |
| Promote before arithmetic | float a32 = __half2float(a); | Convert the value to FP32 before doing sensitive math. |
| Compute in FP32 | float result = a32 * 2.0f; | The multiplication uses the wider accumulator type. |
| Convert for output | __half output = __float2half(result); | Return to FP16 only at the storage boundary. |
| Read the result later | float readable = __half2float(output); | Promote it again when a later operation needs FP32. |
For this example, the stored value represents 1.5, the FP32 calculation
produces 3.0, and the final conversion stores that result as FP16. The same
pattern appears in larger kernels: compact inputs save memory bandwidth, while
the wider arithmetic protects the accumulation from unnecessary rounding.
Headers and scope
The header is part of the contract. It tells the compiler which type family and intrinsic declarations are in scope, but it does not guarantee that every function can execute on every GPU.
| Header or interface | Main contents | Typical reason to include it |
|---|---|---|
vector_types.h | float2, float4, integer vectors, dim3 | Pack fields or move adjacent values together. |
cuda_fp16.h | __half, __half2, half conversions and math | FP16 storage, arithmetic, and packing. |
cuda_bf16.h | __nv_bfloat16, __nv_bfloat162 | BF16 storage and conversion. |
cuda_fp8.h | FP8 formats, packed forms, conversion helpers | Quantized storage and scaled low precision paths. |
mma.h | WMMA fragments and matrix operations | A structured C++ interface to tensor cores. |
| Cooperative Groups | cg:: group handles and collectives | Explicit group scope and synchronization. |
| C++ CUDA standard library | cuda::atomic, cuda::pipeline, cuda::barrier | Modern typed synchronization and memory operations. |
Some declarations are available in both host and device code. Others are device only. The CUDA Math API marks those details per function. A common mistake is to include a header successfully, compile a template successfully, and only later discover that the selected function needs a newer architecture. Compile a small target-specific test when the feature is new or architecture dependent.
1. Built-in vector types
The built-in vector types come from vector_types.h. In ML kernels, the most
useful families are float2, float4, int2, int4, uint2, uint4, and
dim3. CUDA also provides wider integer and floating point families, but they
are less common in the memory movement and launch geometry of neural network
kernels.
They are small structs with named fields:
1 2 3 4 5 6
float2 position = make_float2(x, y);
float4 color = make_float4(r, g, b, alpha);
uint4 packed = make_uint4(a, b, c, d);
float x_value = position.x;
float alpha_value = color.w;
| Type | Fields | Useful for | Important boundary |
|---|---|---|---|
float2 | .x, .y | Two adjacent FP32 values per load or store. | It does not automatically invoke a vector arithmetic instruction. |
float4 | .x, .y, .z, .w | 16 byte memory movement and simple packing. | The pointer must meet the alignment and bounds requirements. |
int2, int4 | Integer fields | Coordinates, packed indices, and integer data movement. | Field grouping does not change integer overflow rules. |
dim3 | .x, .y, .z | Grid, block, and thread dimensions. | It describes launch geometry, not a data vector. |
The constructors make intent readable:
1 2 3 4
float4 a = make_float4(1.0f, 2.0f, 3.0f, 4.0f);
int2 tile = make_int2(blockIdx.x, blockIdx.y);
dim3 block(128, 1, 1);
dim3 grid((n + block.x - 1) / block.x, 1, 1);
What a vector type does and does not promise
In a kernel, float4 is often used to express a 16 byte load:
1 2 3 4 5
const float4 values = reinterpret_cast<const float4 *>(input)[index];
output[index * 4 + 0] = values.x;
output[index * 4 + 1] = values.y;
output[index * 4 + 2] = values.z;
output[index * 4 + 3] = values.w;
This can reduce the number of source-level load operations and can give the
compiler a better description of the intended memory transaction. It does not
mean that the GPU performs four independent arithmetic operations as one
float4 operation. The fields still behave like scalar values unless the
compiler lowers the surrounding code to a suitable instruction sequence.
Before using a vector pointer, check three things:
- The address is suitably aligned for the vector width.
- The logical element count includes every value in the vector.
- The tail path does not read or write beyond the tensor boundary.
For a shape that is not divisible by four, use a scalar tail or a masked path. The fastest interior load is not worth an out of bounds access at the edge.
2. FP16: compact storage and pairwise arithmetic
Include cuda_fp16.h for the FP16 family. The two types that appear most often
in kernels are __half and __half2.
__half stores one half precision value. __half2 stores two half values and
has pairwise operations. The second type is especially useful when a kernel
already has adjacent FP16 elements and wants to load or process them together.
| Type or family | Meaning | Good use |
|---|---|---|
__half | One FP16 value | Compact activations, weights, and intermediate storage. |
__half2 | Two FP16 values | Pairwise arithmetic, vectorized loads, and packed output stores. |
Conversion and rounding
Numeric conversion changes the value's representation. The suffix describes the rounding direction when the function provides one.
| Conversion family | Examples | Use |
|---|---|---|
| Float to half | __float2half, __float2half_rn | Round an FP32 value into FP16. |
| Directed rounding | _rn, _rz, _rd, _ru | Select nearest even, toward zero, down, or up. |
| Half to float | __half2float | Promote an FP16 operand before wider arithmetic. |
| Two floats to pair | __floats2half2_rn, __float22half2_rn | Pack two FP32 values into one __half2. |
| Integer to half | __int2half_*, __half2int_* | Convert integer values with an explicit rounding mode. |
The integer conversion family also covers short, ushort, uint, long long, and unsigned long long variants. The exact overload set belongs to the
installed cuda_fp16.h, so the header should be treated as the final authority
when a kernel depends on a particular width.
For a mixed precision dot product, the pattern is usually more important than the individual name:
1 2 3 4 5 6
__half a_half = input_a[index];
__half b_half = input_b[index];
float a = __half2float(a_half);
float b = __half2float(b_half);
accumulator = fmaf(a, b, accumulator);
The inputs remain compact in memory, but the accumulator does not lose a bit of precision at every multiply and add just because the inputs were FP16.
Packing and unpacking
The pair helpers are useful when a kernel owns two adjacent values:
| Operation | Intrinsics | Where it helps |
|---|---|---|
| Pack two halves | __halves2half2 | Build a pair before a vector store. |
| Duplicate a half | __half2half2 | Broadcast one value to both pair lanes. |
| Select low or high values | __lows2half2, __highs2half2 | Combine selected lanes from two pairs. |
| Extract low or high | __low2half, __high2half | Read one value from a __half2. |
| Extract a pair | __low2half2, __high2half2 | Select a pair from packed data. |
| Convert pair to floats | __half22float2, __low2float, __high2float | Promote pair values before arithmetic. |
Arithmetic and comparisons
The scalar arithmetic family includes __hadd, __hsub, __hmul, __hdiv,
__hfma, __hneg, and __habs. Saturating forms include __hadd_sat,
__hsub_sat, and __hmul_sat. __hfma_relu combines a fused multiply add
with a ReLU style clamp where supported by the header.
The pairwise family follows the same naming pattern:
| Operation | Scalar | Packed pair |
|---|---|---|
| Add | __hadd | __hadd2 |
| Subtract | __hsub | __hsub2 |
| Multiply | __hmul | __hmul2 |
| Divide | __hdiv | __h2div |
| Fused multiply add | __hfma | __hfma2 |
| Absolute value | __habs | __habs2 |
| Saturating add | __hadd_sat | __hadd2_sat |
| Saturating subtract | __hsub_sat | Check the installed header for packed support |
| Saturating multiply | __hmul_sat | __hmul2_sat |
| Saturating fused multiply add | __hfma_sat | __hfma2_sat |
| Fused multiply add with ReLU | __hfma_relu | __hfma2_relu |
Comparisons and half precision math also exist, but they are not the first
things to reach for in an ML kernel. For reductions, normalization, softmax,
and most fused operations, promote values to FP32, use ordinary fmaf, expf,
or rsqrtf as appropriate, and convert back at the output boundary. Use the
half arithmetic helpers when the kernel has a measured reason to keep the
operation in FP16.
3. BF16: more exponent range, fewer mantissa bits
Include cuda_bf16.h for __nv_bfloat16 and __nv_bfloat162. The API mirrors
the FP16 API closely, which makes it possible to write one templated kernel
with type specific conversion helpers.
| Type or family | Meaning | Good use |
|---|---|---|
__nv_bfloat16 | One BF16 value | Training and inference data with FP32-like exponent range. |
__nv_bfloat162 | Two BF16 values | Pairwise loads, arithmetic, and stores. |
| Raw BF16 structs | Exact bit representation | Serialization and low level packing. |
nv_bfloat16 aliases | Common CUDA spelling | C++ templates and overloads. |
BF16 has the exponent range of FP32 but fewer fraction bits than FP16. That often makes BF16 more forgiving when values vary over a large range, while its coarser mantissa can produce a larger absolute error for some operations. The right choice depends on the model, the reduction length, and the validation tolerance.
The conversion pattern is the same as the FP16 pattern:
1 2 3 4
__nv_bfloat16 value = input[index];
float promoted = __bfloat162float(value);
float result = fmaf(promoted, scale, accumulator);
output[index] = __float2bfloat16(result);
The conversion family includes __float2bfloat16, __bfloat162float,
__floats2bfloat162_rn, and __bfloat1622float2. The installed
cuda_bf16.h also provides packing and lane extraction helpers.
The arithmetic names mirror the half family: __hadd, __hmul, __hfma,
__hadd2, __hmul2, __hfma2, and their related comparison and math
functions are overloaded for BF16 where the Toolkit supports them.
If a library wants BF16 to remain a storage type, it can disable implicit BF16 conversions and operators with the macros provided by the header. That is a library policy choice, not something most standalone kernels need to add.
4. FP8: format selection and conversion policy
FP8 is where a type name stops being enough to describe the numerical contract. The two common formats have different allocations of exponent and fraction bits:
| Format | CUDA type | Typical role |
|---|---|---|
| E4M3 | __nv_fp8_e4m3 | More precision for values in a narrower range. |
| E5M2 | __nv_fp8_e5m2 | More exponent range with less fraction precision. |
| E8M0 | __nv_fp8_e8m0 | Scale representation in newer CUDA low precision paths. |
| Two element pack | __nv_fp8x2_* | Pairwise storage and conversion. |
| Four element pack | __nv_fp8x4_* | Compact vectorized storage and movement. |
For conversion, the names most likely to appear in an ML kernel are
__nv_cvt_float_to_fp8, __nv_cvt_float2_to_fp8x2, and the corresponding
FP16 or BF16 conversion helpers. The conversion policy also decides whether
overflow saturates, commonly through __NV_SATFINITE or __NV_NOSAT.
FP8 is most useful when the algorithm has a scale policy. A common pattern is to keep the scale in FP32, convert a scaled value into FP8 for storage or matrix multiplication, and unscale into a wider accumulator:
1 2 3 4 5
float scaled = value * scale;
// The exact constructor or conversion helper depends on the selected FP8 API.
__nv_fp8_e4m3 packed_value = convert_to_fp8(scaled);
float restored = convert_to_float(packed_value) / scale;
The placeholder names in that example are intentional. The exact conversion
signature and rounding options vary with the CUDA Toolkit. Use the installed
cuda_fp8.h and the FP8 section of the CUDA Math API
instead of copying a signature from a different Toolkit release.
Do not treat FP8 as a drop in replacement for FP16 in every scalar expression. It is usually part of a larger scaled matrix multiplication or quantization design. Check range, scale granularity, accumulation type, and error against a reference implementation.
5. FP32 arithmetic and fast math
The standard float type is the workhorse for accumulators, normalization
statistics, scaling factors, and reference calculations in ML kernels. Its
intrinsic families add explicit rounding, device-only fast approximations,
and type conversion. The focus here stays on the single precision path that
surrounds low precision ML arithmetic.
The names that show up most often in ML kernels are fmaf for accumulation,
expf and logf for activation or normalization math, and rsqrtf for
normalization. When a kernel needs explicit rounding, the _rn, _rz, _ru,
and _rd suffixes mean nearest, toward zero, upward, and downward rounding.
The fast intrinsic forms, such as __expf, __logf, and __frsqrt_rn, can be
useful in a hot path, but they should be validated against the model's error
budget. Faster math is not automatically better math if it changes convergence
or the tail of a distribution.
6. Integer intrinsics used in ML kernels
Integer intrinsics show up in indexing, compression, quantization, hash tables, bitmaps, and low precision dot products. They often save a surprising amount of control flow in kernels that do not look like numerical linear algebra.
| Family | Representative intrinsics | What it does |
|---|---|---|
| Count and find bits | __clz, __clzll, __ffs, __ffsll | Count leading zeros or find the first set bit. |
| Population count | __popc, __popcll | Count set bits in a mask. |
| Integer dot products | __dp4a, __dp2a_lo, __dp2a_hi | Pack small integer products into dot products. |
| SIMD in a word | __vadd2, __vsub2, __vabsdiff2, __vavrg2, __vcmpeq2 | Operate on packed subwords. |
A bitmask example
Suppose each bit represents a valid element in a compressed row. __popc gives
the number of valid entries without a loop over every bit:
1 2 3 4 5 6
unsigned valid_mask = row_masks[row_index];
int valid_count = __popc(valid_mask);
if (valid_count != 0) {
process_compacted_row(valid_mask, valid_count);
}
The __dp4a family is useful when four small integer products can be packed
into one instruction path. It appears in int8 inference and image processing,
but the accumulator type and signedness must be checked carefully. A packed
representation is only useful if the producer and consumer agree on bit width,
signedness, and lane order.
7. Warp level primitives
Warp primitives let lanes communicate or vote without writing every temporary value to shared memory. They are powerful because the communication scope is small. They are dangerous when the participation mask or control flow is unclear.
| Category | Intrinsics or API | Use |
|---|---|---|
| Shuffle | __shfl_sync, __shfl_up_sync, __shfl_down_sync, __shfl_xor_sync | Move register values between lanes. |
| Vote | __all_sync, __any_sync, __ballot_sync, __activemask | Ask a question about active lane predicates. |
| Warp reduction | __reduce_add_sync, __reduce_min_sync, __reduce_max_sync | Hardware supported reductions on supported architectures. |
| Synchronization | __syncwarp | Establish a synchronization point for participating lanes. |
| Cooperative Groups | cg::this_thread_block(), cg::tiled_partition<N> | Give a block or subwarp an explicit group handle. |
A shuffle reduction
For a full participating warp, a reduction can keep values in registers:
1 2 3 4 5
unsigned mask = __activemask();
for (int offset = warpSize / 2; offset > 0; offset /= 2) {
value += __shfl_down_sync(mask, value, offset);
}
The important part is not the loop. It is the mask. The mask identifies the lanes that participate and are expected to reach the operation. If the code is inside divergent control flow, capture and pass a mask that matches the actual collective contract.
__activemask() reports active lanes at a point in execution. It does not by
itself make an algorithm safe, and it does not replace a synchronization
protocol. When the intended group is known, a deliberate mask or a Cooperative
Groups handle is often clearer.
Cooperative Groups
Cooperative Groups are useful when a kernel has more than one kind of group.
cg::this_thread_block() names the block, while
cg::tiled_partition<32>() creates fixed size subgroups. For a small kernel,
the raw shuffle and barrier intrinsics are often easier to read. The group API
becomes worthwhile when the same collective is shared by several helpers.
NVIDIA's Cooperative Groups documentation is especially useful when a hand-written warp primitive starts accumulating implicit assumptions.
8. Synchronization, fences, and atomics
These three categories solve different problems. A barrier coordinates arrival. A fence orders a thread's memory operations. An atomic combines an update with a visibility and conflict control mechanism at a defined scope.
| Scope or operation | Functions | Use it when |
|---|---|---|
| Block barrier | __syncthreads | Every thread in one block must reach a phase boundary. |
| Thread fence | __threadfence_block | Ordering is needed for the current block scope. |
| Device fence | __threadfence | Ordering must be visible across the device. |
| Atomic update | atomicAdd | Multiple agents update one location safely. |
| Atomic selection | atomicMax, atomicCAS | Implement claims, locks, or max selection. |
| Scoped atomic | atomicAdd_block, atomicAdd_system, and related scoped forms | Select a narrower or wider visibility scope. |
| C++ atomic | cuda::atomic, cuda::atomic_ref | Express typed memory ordering in modern C++. |
__syncthreads() is the common shared memory pattern:
1 2 3 4 5 6
__shared__ float tile[256];
tile[threadIdx.x] = input[block_offset + threadIdx.x];
__syncthreads();
float reused = tile[(threadIdx.x + 1) & 255];
Every thread in the block must reach the barrier in a compatible way. Putting a block barrier in a branch that only some threads take can deadlock the block or produce invalid behavior.
A fence does not make other threads wait. It establishes ordering for memory operations. An atomic does not automatically make an unrelated algorithm correct either. The update, scope, order, and lifetime of the data must all match the producer and consumer protocol.
9. Asynchronous copies, barriers, and TMA
Once a tiled kernel works, the next bottleneck is often the phase boundary: load a tile, wait, compute, wait, load the next tile. CUDA exposes several ways to overlap movement and computation.
| Feature | Representative API | Use |
|---|---|---|
| Pipeline copy | __pipeline_memcpy_async | Start an asynchronous global to shared transfer. |
| C++ async copy | cuda::memcpy_async | Use typed C++ pipeline operations. |
| C++ pipeline | cuda::pipeline | Track producer and consumer stages. |
| C++ barrier | cuda::barrier | Coordinate asynchronous phases. |
| Tensor memory accelerator | cp.async.bulk.tensor, cuTensorMapEncodeTiled, cuda::device::experimental::cp_async_bulk_* | Move multidimensional tensors with a tensor map. |
The programming pattern is double buffering. While one shared memory buffer is being consumed, the next buffer is being filled. The synchronization object must prove that the next buffer is ready before computation reads it, and that the previous buffer is no longer in use before it is reused.
TMA is not just a faster spelling of a normal load. It needs a tensor map,
appropriate architecture support, and a synchronization protocol. Low level
address helpers such as __cvta_generic_to_shared belong inside that same
specialized layer. TMA is most valuable when a kernel moves regular
multidimensional tiles repeatedly, not when one scalar needs to be read once.
10. Tensor core access
Tensor cores change the unit of arithmetic. Instead of asking ordinary CUDA cores to perform one scalar multiply and add at a time, a matrix multiply API describes fragments and lets the hardware execute a specialized matrix operation.
| Interface | Representative names | Best understood as |
|---|---|---|
| WMMA | nvcuda::wmma::fragment, load_matrix_sync, mma_sync, store_matrix_sync | A structured C++ interface to warp level matrix operations. |
| Inline PTX | mma.sync.aligned, ldmatrix, stmatrix | Direct control over lower level matrix instructions. |
| Libraries | CUTLASS and CuTe | Tuned abstractions around the instruction families. |
Hopper and Blackwell add newer instruction families such as WGMMA and
tcgen05. They are important when targeting those GPUs directly, but they are
not a good first interface for learning GEMM. Start with WMMA or CUTLASS, then
move lower only when profiling shows that the abstraction is the bottleneck.
A minimal WMMA shaped example looks like this:
1 2 3 4 5 6 7 8 9 10 11 12
#include <mma.h>
using namespace nvcuda;
wmma::fragment<wmma::matrix_a, 16, 16, 16, half, wmma::row_major> a_frag;
wmma::fragment<wmma::matrix_b, 16, 16, 16, half, wmma::col_major> b_frag;
wmma::fragment<wmma::accumulator, 16, 16, 16, float> c_frag;
wmma::fill_fragment(c_frag, 0.0f);
wmma::load_matrix_sync(a_frag, a_tile, 16);
wmma::load_matrix_sync(b_frag, b_tile, 16);
wmma::mma_sync(c_frag, a_frag, b_frag, c_frag);
wmma::store_matrix_sync(c_tile, c_frag, 16, wmma::mem_row_major);
The example hides the hard parts: layout, alignment, fragment ownership, shared memory coordination, instruction shape, and architecture support. It is useful as a mental model, not as a complete production kernel.
Inline PTX is appropriate when the higher level interface does not expose an
instruction or layout you need. It is also a maintenance commitment. PTX is
versioned, and the generated SASS depends on the target architecture. Hopper
features such as WGMMA and TMA, and Blackwell features such as tcgen05, need a
carefully selected compile target and a validation path on the intended GPU.
CUTLASS and CuTe are often the right first choice for a real GEMM. They encode many details that are easy to get wrong by hand: layouts, stage counts, asynchronous movement, accumulator types, and architecture specific instruction selection.
11. Qualifiers, built-ins, and device utilities
The final group is less glamorous, but it is present in almost every kernel. Qualifiers tell CUDA where code or storage may live. Built-ins identify the calling thread's position. Utility functions expose timing, debugging, cache behavior, and special execution controls.
Qualifiers and attributes
| Qualifier | Meaning | Typical use |
|---|---|---|
__global__ | Kernel callable from the host | Define a launchable GPU entry point. |
__device__ | Function callable from device code | Share helper logic inside kernels. |
__shared__ | Per block shared storage | Stage tiles and communicate within a block. |
__restrict__ | Pointer aliasing promise | Help the compiler optimize non-overlapping pointers. |
__forceinline__ | Strong inline request | Remove helper call overhead in hot device code. |
__launch_bounds__ | Launch and register guidance | Constrain a kernel's launch configuration. |
__maxnreg__ | Register count limit | Explore register pressure and occupancy tradeoffs. |
__align__ | Alignment declaration | Make a storage or type alignment explicit. |
__restrict__ is a promise, not a hint that the compiler is free to ignore.
If two pointers overlap when the function says they do not, the compiler may
legitimately produce a wrong result from the program's point of view.
Built-in coordinates
| Built-in | Meaning |
|---|---|
threadIdx | Current thread's three dimensional index in the block. |
blockIdx | Current block's three dimensional index in the grid. |
blockDim | Dimensions of the launched block. |
gridDim | Dimensions of the launched grid. |
warpSize | Hardware warp width exposed to the kernel. |
The usual global index calculation is still worth writing explicitly:
1 2 3 4 5 6 7 8 9
__global__ void add(const float *__restrict__ a,
const float *__restrict__ b,
float *__restrict__ c,
int n) {
const int index = blockIdx.x * blockDim.x + threadIdx.x;
if (index < n) {
c[index] = a[index] + b[index];
}
}
For debugging, printf and assert are useful during small test launches, but
they are not part of a production kernel's performance path. Cache hints,
prefetch, dynamic parallelism, texture, surface, and device timing APIs are
specialized tools. Reach for them after the memory layout and synchronization
of the main kernel are already correct.
A practical selection table
When I start a new kernel, this is the short list I use before reaching for a more specialized intrinsic:
| Kernel problem | Start with | Then verify |
|---|---|---|
| Move adjacent scalar values | float2, float4, or a matching pair type | Alignment, tail handling, and generated loads. |
| Store activations compactly | __half, __nv_bfloat16, or a supported FP8 type | Range, error, and accumulator precision. |
| Pair low precision values | __half2 or __nv_bfloat162 | Pair layout and conversion points. |
| Convert a value numerically | Explicit conversion intrinsic | Rounding mode and special values. |
| Share a register inside a warp | __shfl_sync family | Correct participation mask. |
| Vote on a lane predicate | __all_sync, __any_sync, __ballot_sync | Divergence and mask lifetime. |
| Coordinate one block | __syncthreads or cg::this_thread_block | Every thread reaches the phase boundary. |
| Publish data across scopes | Fence or scoped atomic | Memory order and consumer scope. |
| Overlap tile movement and math | cuda::memcpy_async, pipeline, barrier, or TMA | Stage lifetime and architecture support. |
| Use matrix hardware | WMMA, MMA, WGMMA, or CUTLASS and CuTe | Shape, layout, target architecture, and accumulator type. |
Common wrong turns
The following mistakes are easy to make because the code still looks plausible.
| Wrong turn | Why it is dangerous | Better replacement |
|---|---|---|
Treat float4 as automatic vector arithmetic | It is primarily a small struct and memory layout tool. | Inspect the generated code and use the arithmetic API you actually need. |
| Accumulate a long dot product in FP16 or BF16 by accident | Rounding error compounds through the reduction. | Promote operands and keep the accumulator in FP32 when accuracy requires it. |
| Use bit reinterpretation as numeric conversion | The bits stay the same, so the numeric value changes unpredictably. | Use a numeric conversion intrinsic. |
Use __syncthreads inside a partial branch | Threads that do not reach it can deadlock the block. | Structure the branch uniformly or use a correct group protocol. |
Use __syncwarp to coordinate a block | A warp barrier does not cover other warps. | Use __syncthreads or a block Cooperative Group. |
Use __activemask as a permanent group identity | Active lanes can change after divergence. | Establish the intended mask or use a named group. |
| Assume every low precision name is portable | Availability depends on Toolkit and architecture. | Check headers, compute capability, and generated code. |
| Hand write tensor core PTX too early | Layout and synchronization bugs are difficult to diagnose. | Start with WMMA or CUTLASS and move lower only with evidence. |
| Trust one fast math result without an error test | Approximate functions can change convergence or tails. | Compare against a precise reference over the real input range. |
A small mixed precision example
The following example combines several ideas without pretending to be a full GEMM. Values are stored as half precision, two values are loaded together, the computation is promoted to FP32, and the result is written back after a simple scale:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
#include <cuda_fp16.h>
__global__ void scale_half2(const __half *input,
__half *output,
int count) {
const int pair_index = blockIdx.x * blockDim.x + threadIdx.x;
const int value_index = pair_index * 2;
if (value_index + 1 >= count) {
return;
}
const __half2 packed = reinterpret_cast<const __half2 *>(input)[pair_index];
const float2 values = __half22float2(packed);
const float2 scaled = make_float2(values.x * 0.5f, values.y * 0.5f);
reinterpret_cast<__half2 *>(output)[pair_index] =
__floats2half2_rn(scaled.x, scaled.y);
}
This is not automatically the fastest implementation for every shape. It is a good example of a visible precision boundary. The load is compact, the math is wide, and the store rounds once. A production kernel would also validate alignment, handle the tail carefully, and benchmark the generated code.
Version and architecture checklist
Before publishing a kernel that uses one of the newer groups, I record:
| Check | Why it matters |
|---|---|
CUDA Toolkit and -arch | Headers and hardware instructions change over time. |
| Alignment and layout | Packed types and vector loads require correct addresses and shapes. |
| Numerical reference | Low precision and fast math need a tolerance and error report. |
| Synchronization scope | Warp, block, and device visibility are different contracts. |
For an architecture-specific feature, keep a small compile and correctness test next to the kernel. It is much easier to diagnose a missing overload or unsupported instruction in a ten line test than inside a large fused model.
Closing perspective
The CUDA intrinsic catalog looks like a long list when viewed in isolation. It becomes more manageable when each name is attached to a boundary:
- A vector type describes how values are grouped in memory.
- A conversion describes how the number changes representation.
- A math intrinsic describes a precision and accuracy tradeoff.
- A warp primitive describes a lane communication scope.
- A barrier or fence describes when work and memory become visible.
- An atomic describes a conflicting update at a chosen scope.
- An asynchronous copy describes a staged movement contract.
- A tensor core interface describes a matrix instruction and its layout.
- A qualifier describes where code or storage is allowed to live.
The best kernels are rarely the ones with the most intrinsic names. They are the ones where every name answers a real question about storage, arithmetic, communication, or hardware. Start with the clearest type and scope, measure the bottleneck, and only then move toward the lower level interface.
References
- CUDA Math API Reference Manual, the reference for CUDA vector, floating point, integer, and type conversion APIs.
- CUDA C++ Programming Guide, the programming model, synchronization, memory, and architecture guide.
- CUDA floating point functionality, including standard math, intrinsic functions, and precision behavior.
- Half Precision Intrinsics, including
__half,__half2, conversion, arithmetic, and math functions. - Bfloat16 Precision Intrinsics, including
__nv_bfloat16and__nv_bfloat162. - Cooperative Groups, for explicit group handles, collectives, and synchronization.
About the author
Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.