Ritesh Yadav
Ritesh Yadav
/device-software/cuda-built-in-types-and-intrinsics
CUDA

CUDA Types and Intrinsics for ML Kernels

When I look at a CUDA kernel, I usually start with a very ordinary question: where does the value change shape? It may start as a half precision weight in global memory, move through shared memory, become an FP32 accumulator, and end as a half precision output. The multiplication itself is often the easy part. The decisions around that multiplication are where most of the bugs live.

One value might begin as an FP8 activation in global memory, be unpacked into a pair or a small vector, converted to FP16 or BF16, accumulated in FP32, and finally written back in a lower precision format. A different kernel might keep everything in FP32 but use a warp shuffle to reduce a partial sum. Another one might use an atomic instead of a barrier because its producers are not in the same block.

CUDA gives each boundary a small API. The names are compact, but they are not interchangeable. __half2 is a storage and arithmetic type. float4 is a four field structure that can describe a memory movement. A bit reinterpretation is not a numeric conversion. __syncwarp() is not a replacement for __syncthreads().

This article is a field guide to the built-in types, conversions, and device intrinsics that appear most often in CUDA C++ kernels for machine learning and large language models. It follows the headers and groups in the CUDA Math API, but keeps the main path practical: what the group means, where it belongs, what to show in code, and what to check before trusting it.

The exact names change with the CUDA Toolkit and target architecture. I use the CUDA Math API Reference Manual and the CUDA C++ Programming Guide to check a particular header, compiler, and -arch value.

The four questions I ask first

Before choosing a CUDA type or intrinsic, I try to answer four questions:

  1. How is the value stored? Global memory bandwidth, shared memory capacity, and communication format often favor FP16, BF16, FP8, or a packed integer representation.
  2. Where is the arithmetic performed? A low precision input does not imply a low precision accumulator. Many useful kernels load in a compact format, convert to FP32, and accumulate in FP32.
  3. Who needs to see the result? A value shared by lanes in one warp needs a different primitive from a value shared by all threads in a block or by unrelated blocks.
  4. Which hardware can execute it? Some names are available in the header but require a minimum compute capability, a newer Toolkit, or a particular instruction path.

The rest of this guide is easier to use if these questions stay visible. A function name is only useful when its storage format, scope, precision, and hardware contract are also understood.

One small example

Here is the simplest mixed precision story to keep in mind while reading the rest of the article. The values are stored as FP16, promoted to FP32 for the calculation, and converted back only when the result is stored.

StepSmall CUDA exampleWhat it means in plain language
Store a compact value__half a = __float2half(1.5f);Keep the input in a smaller 16 bit format.
Promote before arithmeticfloat a32 = __half2float(a);Convert the value to FP32 before doing sensitive math.
Compute in FP32float result = a32 * 2.0f;The multiplication uses the wider accumulator type.
Convert for output__half output = __float2half(result);Return to FP16 only at the storage boundary.
Read the result laterfloat readable = __half2float(output);Promote it again when a later operation needs FP32.

For this example, the stored value represents 1.5, the FP32 calculation produces 3.0, and the final conversion stores that result as FP16. The same pattern appears in larger kernels: compact inputs save memory bandwidth, while the wider arithmetic protects the accumulation from unnecessary rounding.

Headers and scope

The header is part of the contract. It tells the compiler which type family and intrinsic declarations are in scope, but it does not guarantee that every function can execute on every GPU.

Header or interfaceMain contentsTypical reason to include it
vector_types.hfloat2, float4, integer vectors, dim3Pack fields or move adjacent values together.
cuda_fp16.h__half, __half2, half conversions and mathFP16 storage, arithmetic, and packing.
cuda_bf16.h__nv_bfloat16, __nv_bfloat162BF16 storage and conversion.
cuda_fp8.hFP8 formats, packed forms, conversion helpersQuantized storage and scaled low precision paths.
mma.hWMMA fragments and matrix operationsA structured C++ interface to tensor cores.
Cooperative Groupscg:: group handles and collectivesExplicit group scope and synchronization.
C++ CUDA standard librarycuda::atomic, cuda::pipeline, cuda::barrierModern typed synchronization and memory operations.

Some declarations are available in both host and device code. Others are device only. The CUDA Math API marks those details per function. A common mistake is to include a header successfully, compile a template successfully, and only later discover that the selected function needs a newer architecture. Compile a small target-specific test when the feature is new or architecture dependent.

1. Built-in vector types

The built-in vector types come from vector_types.h. In ML kernels, the most useful families are float2, float4, int2, int4, uint2, uint4, and dim3. CUDA also provides wider integer and floating point families, but they are less common in the memory movement and launch geometry of neural network kernels.

They are small structs with named fields:

CUDA
float2 position = make_float2(x, y);
float4 color = make_float4(r, g, b, alpha);
uint4 packed = make_uint4(a, b, c, d);

float x_value = position.x;
float alpha_value = color.w;
TypeFieldsUseful forImportant boundary
float2.x, .yTwo adjacent FP32 values per load or store.It does not automatically invoke a vector arithmetic instruction.
float4.x, .y, .z, .w16 byte memory movement and simple packing.The pointer must meet the alignment and bounds requirements.
int2, int4Integer fieldsCoordinates, packed indices, and integer data movement.Field grouping does not change integer overflow rules.
dim3.x, .y, .zGrid, block, and thread dimensions.It describes launch geometry, not a data vector.

The constructors make intent readable:

CUDA
float4 a = make_float4(1.0f, 2.0f, 3.0f, 4.0f);
int2 tile = make_int2(blockIdx.x, blockIdx.y);
dim3 block(128, 1, 1);
dim3 grid((n + block.x - 1) / block.x, 1, 1);

What a vector type does and does not promise

In a kernel, float4 is often used to express a 16 byte load:

CUDA
const float4 values = reinterpret_cast<const float4 *>(input)[index];
output[index * 4 + 0] = values.x;
output[index * 4 + 1] = values.y;
output[index * 4 + 2] = values.z;
output[index * 4 + 3] = values.w;

This can reduce the number of source-level load operations and can give the compiler a better description of the intended memory transaction. It does not mean that the GPU performs four independent arithmetic operations as one float4 operation. The fields still behave like scalar values unless the compiler lowers the surrounding code to a suitable instruction sequence.

Before using a vector pointer, check three things:

  • The address is suitably aligned for the vector width.
  • The logical element count includes every value in the vector.
  • The tail path does not read or write beyond the tensor boundary.

For a shape that is not divisible by four, use a scalar tail or a masked path. The fastest interior load is not worth an out of bounds access at the edge.

2. FP16: compact storage and pairwise arithmetic

Include cuda_fp16.h for the FP16 family. The two types that appear most often in kernels are __half and __half2.

__half stores one half precision value. __half2 stores two half values and has pairwise operations. The second type is especially useful when a kernel already has adjacent FP16 elements and wants to load or process them together.

Type or familyMeaningGood use
__halfOne FP16 valueCompact activations, weights, and intermediate storage.
__half2Two FP16 valuesPairwise arithmetic, vectorized loads, and packed output stores.

Conversion and rounding

Numeric conversion changes the value's representation. The suffix describes the rounding direction when the function provides one.

Conversion familyExamplesUse
Float to half__float2half, __float2half_rnRound an FP32 value into FP16.
Directed rounding_rn, _rz, _rd, _ruSelect nearest even, toward zero, down, or up.
Half to float__half2floatPromote an FP16 operand before wider arithmetic.
Two floats to pair__floats2half2_rn, __float22half2_rnPack two FP32 values into one __half2.
Integer to half__int2half_*, __half2int_*Convert integer values with an explicit rounding mode.

The integer conversion family also covers short, ushort, uint, long long, and unsigned long long variants. The exact overload set belongs to the installed cuda_fp16.h, so the header should be treated as the final authority when a kernel depends on a particular width.

For a mixed precision dot product, the pattern is usually more important than the individual name:

CUDA
__half a_half = input_a[index];
__half b_half = input_b[index];

float a = __half2float(a_half);
float b = __half2float(b_half);
accumulator = fmaf(a, b, accumulator);

The inputs remain compact in memory, but the accumulator does not lose a bit of precision at every multiply and add just because the inputs were FP16.

Packing and unpacking

The pair helpers are useful when a kernel owns two adjacent values:

OperationIntrinsicsWhere it helps
Pack two halves__halves2half2Build a pair before a vector store.
Duplicate a half__half2half2Broadcast one value to both pair lanes.
Select low or high values__lows2half2, __highs2half2Combine selected lanes from two pairs.
Extract low or high__low2half, __high2halfRead one value from a __half2.
Extract a pair__low2half2, __high2half2Select a pair from packed data.
Convert pair to floats__half22float2, __low2float, __high2floatPromote pair values before arithmetic.

Arithmetic and comparisons

The scalar arithmetic family includes __hadd, __hsub, __hmul, __hdiv, __hfma, __hneg, and __habs. Saturating forms include __hadd_sat, __hsub_sat, and __hmul_sat. __hfma_relu combines a fused multiply add with a ReLU style clamp where supported by the header.

The pairwise family follows the same naming pattern:

OperationScalarPacked pair
Add__hadd__hadd2
Subtract__hsub__hsub2
Multiply__hmul__hmul2
Divide__hdiv__h2div
Fused multiply add__hfma__hfma2
Absolute value__habs__habs2
Saturating add__hadd_sat__hadd2_sat
Saturating subtract__hsub_satCheck the installed header for packed support
Saturating multiply__hmul_sat__hmul2_sat
Saturating fused multiply add__hfma_sat__hfma2_sat
Fused multiply add with ReLU__hfma_relu__hfma2_relu

Comparisons and half precision math also exist, but they are not the first things to reach for in an ML kernel. For reductions, normalization, softmax, and most fused operations, promote values to FP32, use ordinary fmaf, expf, or rsqrtf as appropriate, and convert back at the output boundary. Use the half arithmetic helpers when the kernel has a measured reason to keep the operation in FP16.

3. BF16: more exponent range, fewer mantissa bits

Include cuda_bf16.h for __nv_bfloat16 and __nv_bfloat162. The API mirrors the FP16 API closely, which makes it possible to write one templated kernel with type specific conversion helpers.

Type or familyMeaningGood use
__nv_bfloat16One BF16 valueTraining and inference data with FP32-like exponent range.
__nv_bfloat162Two BF16 valuesPairwise loads, arithmetic, and stores.
Raw BF16 structsExact bit representationSerialization and low level packing.
nv_bfloat16 aliasesCommon CUDA spellingC++ templates and overloads.

BF16 has the exponent range of FP32 but fewer fraction bits than FP16. That often makes BF16 more forgiving when values vary over a large range, while its coarser mantissa can produce a larger absolute error for some operations. The right choice depends on the model, the reduction length, and the validation tolerance.

The conversion pattern is the same as the FP16 pattern:

CUDA
__nv_bfloat16 value = input[index];
float promoted = __bfloat162float(value);
float result = fmaf(promoted, scale, accumulator);
output[index] = __float2bfloat16(result);

The conversion family includes __float2bfloat16, __bfloat162float, __floats2bfloat162_rn, and __bfloat1622float2. The installed cuda_bf16.h also provides packing and lane extraction helpers.

The arithmetic names mirror the half family: __hadd, __hmul, __hfma, __hadd2, __hmul2, __hfma2, and their related comparison and math functions are overloaded for BF16 where the Toolkit supports them.

If a library wants BF16 to remain a storage type, it can disable implicit BF16 conversions and operators with the macros provided by the header. That is a library policy choice, not something most standalone kernels need to add.

4. FP8: format selection and conversion policy

FP8 is where a type name stops being enough to describe the numerical contract. The two common formats have different allocations of exponent and fraction bits:

FormatCUDA typeTypical role
E4M3__nv_fp8_e4m3More precision for values in a narrower range.
E5M2__nv_fp8_e5m2More exponent range with less fraction precision.
E8M0__nv_fp8_e8m0Scale representation in newer CUDA low precision paths.
Two element pack__nv_fp8x2_*Pairwise storage and conversion.
Four element pack__nv_fp8x4_*Compact vectorized storage and movement.

For conversion, the names most likely to appear in an ML kernel are __nv_cvt_float_to_fp8, __nv_cvt_float2_to_fp8x2, and the corresponding FP16 or BF16 conversion helpers. The conversion policy also decides whether overflow saturates, commonly through __NV_SATFINITE or __NV_NOSAT.

FP8 is most useful when the algorithm has a scale policy. A common pattern is to keep the scale in FP32, convert a scaled value into FP8 for storage or matrix multiplication, and unscale into a wider accumulator:

CUDA
float scaled = value * scale;
// The exact constructor or conversion helper depends on the selected FP8 API.
__nv_fp8_e4m3 packed_value = convert_to_fp8(scaled);

float restored = convert_to_float(packed_value) / scale;

The placeholder names in that example are intentional. The exact conversion signature and rounding options vary with the CUDA Toolkit. Use the installed cuda_fp8.h and the FP8 section of the CUDA Math API instead of copying a signature from a different Toolkit release.

Do not treat FP8 as a drop in replacement for FP16 in every scalar expression. It is usually part of a larger scaled matrix multiplication or quantization design. Check range, scale granularity, accumulation type, and error against a reference implementation.

5. FP32 arithmetic and fast math

The standard float type is the workhorse for accumulators, normalization statistics, scaling factors, and reference calculations in ML kernels. Its intrinsic families add explicit rounding, device-only fast approximations, and type conversion. The focus here stays on the single precision path that surrounds low precision ML arithmetic.

The names that show up most often in ML kernels are fmaf for accumulation, expf and logf for activation or normalization math, and rsqrtf for normalization. When a kernel needs explicit rounding, the _rn, _rz, _ru, and _rd suffixes mean nearest, toward zero, upward, and downward rounding.

The fast intrinsic forms, such as __expf, __logf, and __frsqrt_rn, can be useful in a hot path, but they should be validated against the model's error budget. Faster math is not automatically better math if it changes convergence or the tail of a distribution.

6. Integer intrinsics used in ML kernels

Integer intrinsics show up in indexing, compression, quantization, hash tables, bitmaps, and low precision dot products. They often save a surprising amount of control flow in kernels that do not look like numerical linear algebra.

FamilyRepresentative intrinsicsWhat it does
Count and find bits__clz, __clzll, __ffs, __ffsllCount leading zeros or find the first set bit.
Population count__popc, __popcllCount set bits in a mask.
Integer dot products__dp4a, __dp2a_lo, __dp2a_hiPack small integer products into dot products.
SIMD in a word__vadd2, __vsub2, __vabsdiff2, __vavrg2, __vcmpeq2Operate on packed subwords.

A bitmask example

Suppose each bit represents a valid element in a compressed row. __popc gives the number of valid entries without a loop over every bit:

CUDA
unsigned valid_mask = row_masks[row_index];
int valid_count = __popc(valid_mask);

if (valid_count != 0) {
    process_compacted_row(valid_mask, valid_count);
}

The __dp4a family is useful when four small integer products can be packed into one instruction path. It appears in int8 inference and image processing, but the accumulator type and signedness must be checked carefully. A packed representation is only useful if the producer and consumer agree on bit width, signedness, and lane order.

7. Warp level primitives

Warp primitives let lanes communicate or vote without writing every temporary value to shared memory. They are powerful because the communication scope is small. They are dangerous when the participation mask or control flow is unclear.

CategoryIntrinsics or APIUse
Shuffle__shfl_sync, __shfl_up_sync, __shfl_down_sync, __shfl_xor_syncMove register values between lanes.
Vote__all_sync, __any_sync, __ballot_sync, __activemaskAsk a question about active lane predicates.
Warp reduction__reduce_add_sync, __reduce_min_sync, __reduce_max_syncHardware supported reductions on supported architectures.
Synchronization__syncwarpEstablish a synchronization point for participating lanes.
Cooperative Groupscg::this_thread_block(), cg::tiled_partition<N>Give a block or subwarp an explicit group handle.

A shuffle reduction

For a full participating warp, a reduction can keep values in registers:

CUDA
unsigned mask = __activemask();

for (int offset = warpSize / 2; offset > 0; offset /= 2) {
    value += __shfl_down_sync(mask, value, offset);
}

The important part is not the loop. It is the mask. The mask identifies the lanes that participate and are expected to reach the operation. If the code is inside divergent control flow, capture and pass a mask that matches the actual collective contract.

__activemask() reports active lanes at a point in execution. It does not by itself make an algorithm safe, and it does not replace a synchronization protocol. When the intended group is known, a deliberate mask or a Cooperative Groups handle is often clearer.

Cooperative Groups

Cooperative Groups are useful when a kernel has more than one kind of group. cg::this_thread_block() names the block, while cg::tiled_partition<32>() creates fixed size subgroups. For a small kernel, the raw shuffle and barrier intrinsics are often easier to read. The group API becomes worthwhile when the same collective is shared by several helpers.

NVIDIA's Cooperative Groups documentation is especially useful when a hand-written warp primitive starts accumulating implicit assumptions.

8. Synchronization, fences, and atomics

These three categories solve different problems. A barrier coordinates arrival. A fence orders a thread's memory operations. An atomic combines an update with a visibility and conflict control mechanism at a defined scope.

Scope or operationFunctionsUse it when
Block barrier__syncthreadsEvery thread in one block must reach a phase boundary.
Thread fence__threadfence_blockOrdering is needed for the current block scope.
Device fence__threadfenceOrdering must be visible across the device.
Atomic updateatomicAddMultiple agents update one location safely.
Atomic selectionatomicMax, atomicCASImplement claims, locks, or max selection.
Scoped atomicatomicAdd_block, atomicAdd_system, and related scoped formsSelect a narrower or wider visibility scope.
C++ atomiccuda::atomic, cuda::atomic_refExpress typed memory ordering in modern C++.

__syncthreads() is the common shared memory pattern:

CUDA
__shared__ float tile[256];

tile[threadIdx.x] = input[block_offset + threadIdx.x];
__syncthreads();

float reused = tile[(threadIdx.x + 1) & 255];

Every thread in the block must reach the barrier in a compatible way. Putting a block barrier in a branch that only some threads take can deadlock the block or produce invalid behavior.

A fence does not make other threads wait. It establishes ordering for memory operations. An atomic does not automatically make an unrelated algorithm correct either. The update, scope, order, and lifetime of the data must all match the producer and consumer protocol.

9. Asynchronous copies, barriers, and TMA

Once a tiled kernel works, the next bottleneck is often the phase boundary: load a tile, wait, compute, wait, load the next tile. CUDA exposes several ways to overlap movement and computation.

FeatureRepresentative APIUse
Pipeline copy__pipeline_memcpy_asyncStart an asynchronous global to shared transfer.
C++ async copycuda::memcpy_asyncUse typed C++ pipeline operations.
C++ pipelinecuda::pipelineTrack producer and consumer stages.
C++ barriercuda::barrierCoordinate asynchronous phases.
Tensor memory acceleratorcp.async.bulk.tensor, cuTensorMapEncodeTiled, cuda::device::experimental::cp_async_bulk_*Move multidimensional tensors with a tensor map.

The programming pattern is double buffering. While one shared memory buffer is being consumed, the next buffer is being filled. The synchronization object must prove that the next buffer is ready before computation reads it, and that the previous buffer is no longer in use before it is reused.

TMA is not just a faster spelling of a normal load. It needs a tensor map, appropriate architecture support, and a synchronization protocol. Low level address helpers such as __cvta_generic_to_shared belong inside that same specialized layer. TMA is most valuable when a kernel moves regular multidimensional tiles repeatedly, not when one scalar needs to be read once.

10. Tensor core access

Tensor cores change the unit of arithmetic. Instead of asking ordinary CUDA cores to perform one scalar multiply and add at a time, a matrix multiply API describes fragments and lets the hardware execute a specialized matrix operation.

InterfaceRepresentative namesBest understood as
WMMAnvcuda::wmma::fragment, load_matrix_sync, mma_sync, store_matrix_syncA structured C++ interface to warp level matrix operations.
Inline PTXmma.sync.aligned, ldmatrix, stmatrixDirect control over lower level matrix instructions.
LibrariesCUTLASS and CuTeTuned abstractions around the instruction families.

Hopper and Blackwell add newer instruction families such as WGMMA and tcgen05. They are important when targeting those GPUs directly, but they are not a good first interface for learning GEMM. Start with WMMA or CUTLASS, then move lower only when profiling shows that the abstraction is the bottleneck.

A minimal WMMA shaped example looks like this:

CUDA
#include <mma.h>
using namespace nvcuda;

wmma::fragment<wmma::matrix_a, 16, 16, 16, half, wmma::row_major> a_frag;
wmma::fragment<wmma::matrix_b, 16, 16, 16, half, wmma::col_major> b_frag;
wmma::fragment<wmma::accumulator, 16, 16, 16, float> c_frag;

wmma::fill_fragment(c_frag, 0.0f);
wmma::load_matrix_sync(a_frag, a_tile, 16);
wmma::load_matrix_sync(b_frag, b_tile, 16);
wmma::mma_sync(c_frag, a_frag, b_frag, c_frag);
wmma::store_matrix_sync(c_tile, c_frag, 16, wmma::mem_row_major);

The example hides the hard parts: layout, alignment, fragment ownership, shared memory coordination, instruction shape, and architecture support. It is useful as a mental model, not as a complete production kernel.

Inline PTX is appropriate when the higher level interface does not expose an instruction or layout you need. It is also a maintenance commitment. PTX is versioned, and the generated SASS depends on the target architecture. Hopper features such as WGMMA and TMA, and Blackwell features such as tcgen05, need a carefully selected compile target and a validation path on the intended GPU.

CUTLASS and CuTe are often the right first choice for a real GEMM. They encode many details that are easy to get wrong by hand: layouts, stage counts, asynchronous movement, accumulator types, and architecture specific instruction selection.

11. Qualifiers, built-ins, and device utilities

The final group is less glamorous, but it is present in almost every kernel. Qualifiers tell CUDA where code or storage may live. Built-ins identify the calling thread's position. Utility functions expose timing, debugging, cache behavior, and special execution controls.

Qualifiers and attributes

QualifierMeaningTypical use
__global__Kernel callable from the hostDefine a launchable GPU entry point.
__device__Function callable from device codeShare helper logic inside kernels.
__shared__Per block shared storageStage tiles and communicate within a block.
__restrict__Pointer aliasing promiseHelp the compiler optimize non-overlapping pointers.
__forceinline__Strong inline requestRemove helper call overhead in hot device code.
__launch_bounds__Launch and register guidanceConstrain a kernel's launch configuration.
__maxnreg__Register count limitExplore register pressure and occupancy tradeoffs.
__align__Alignment declarationMake a storage or type alignment explicit.

__restrict__ is a promise, not a hint that the compiler is free to ignore. If two pointers overlap when the function says they do not, the compiler may legitimately produce a wrong result from the program's point of view.

Built-in coordinates

Built-inMeaning
threadIdxCurrent thread's three dimensional index in the block.
blockIdxCurrent block's three dimensional index in the grid.
blockDimDimensions of the launched block.
gridDimDimensions of the launched grid.
warpSizeHardware warp width exposed to the kernel.

The usual global index calculation is still worth writing explicitly:

CUDA
__global__ void add(const float *__restrict__ a,
                    const float *__restrict__ b,
                    float *__restrict__ c,
                    int n) {
    const int index = blockIdx.x * blockDim.x + threadIdx.x;
    if (index < n) {
        c[index] = a[index] + b[index];
    }
}

For debugging, printf and assert are useful during small test launches, but they are not part of a production kernel's performance path. Cache hints, prefetch, dynamic parallelism, texture, surface, and device timing APIs are specialized tools. Reach for them after the memory layout and synchronization of the main kernel are already correct.

A practical selection table

When I start a new kernel, this is the short list I use before reaching for a more specialized intrinsic:

Kernel problemStart withThen verify
Move adjacent scalar valuesfloat2, float4, or a matching pair typeAlignment, tail handling, and generated loads.
Store activations compactly__half, __nv_bfloat16, or a supported FP8 typeRange, error, and accumulator precision.
Pair low precision values__half2 or __nv_bfloat162Pair layout and conversion points.
Convert a value numericallyExplicit conversion intrinsicRounding mode and special values.
Share a register inside a warp__shfl_sync familyCorrect participation mask.
Vote on a lane predicate__all_sync, __any_sync, __ballot_syncDivergence and mask lifetime.
Coordinate one block__syncthreads or cg::this_thread_blockEvery thread reaches the phase boundary.
Publish data across scopesFence or scoped atomicMemory order and consumer scope.
Overlap tile movement and mathcuda::memcpy_async, pipeline, barrier, or TMAStage lifetime and architecture support.
Use matrix hardwareWMMA, MMA, WGMMA, or CUTLASS and CuTeShape, layout, target architecture, and accumulator type.

Common wrong turns

The following mistakes are easy to make because the code still looks plausible.

Wrong turnWhy it is dangerousBetter replacement
Treat float4 as automatic vector arithmeticIt is primarily a small struct and memory layout tool.Inspect the generated code and use the arithmetic API you actually need.
Accumulate a long dot product in FP16 or BF16 by accidentRounding error compounds through the reduction.Promote operands and keep the accumulator in FP32 when accuracy requires it.
Use bit reinterpretation as numeric conversionThe bits stay the same, so the numeric value changes unpredictably.Use a numeric conversion intrinsic.
Use __syncthreads inside a partial branchThreads that do not reach it can deadlock the block.Structure the branch uniformly or use a correct group protocol.
Use __syncwarp to coordinate a blockA warp barrier does not cover other warps.Use __syncthreads or a block Cooperative Group.
Use __activemask as a permanent group identityActive lanes can change after divergence.Establish the intended mask or use a named group.
Assume every low precision name is portableAvailability depends on Toolkit and architecture.Check headers, compute capability, and generated code.
Hand write tensor core PTX too earlyLayout and synchronization bugs are difficult to diagnose.Start with WMMA or CUTLASS and move lower only with evidence.
Trust one fast math result without an error testApproximate functions can change convergence or tails.Compare against a precise reference over the real input range.

A small mixed precision example

The following example combines several ideas without pretending to be a full GEMM. Values are stored as half precision, two values are loaded together, the computation is promoted to FP32, and the result is written back after a simple scale:

CUDA
#include <cuda_fp16.h>

__global__ void scale_half2(const __half *input,
                            __half *output,
                            int count) {
    const int pair_index = blockIdx.x * blockDim.x + threadIdx.x;
    const int value_index = pair_index * 2;
    if (value_index + 1 >= count) {
        return;
    }

    const __half2 packed = reinterpret_cast<const __half2 *>(input)[pair_index];
    const float2 values = __half22float2(packed);
    const float2 scaled = make_float2(values.x * 0.5f, values.y * 0.5f);

    reinterpret_cast<__half2 *>(output)[pair_index] =
        __floats2half2_rn(scaled.x, scaled.y);
}

This is not automatically the fastest implementation for every shape. It is a good example of a visible precision boundary. The load is compact, the math is wide, and the store rounds once. A production kernel would also validate alignment, handle the tail carefully, and benchmark the generated code.

Version and architecture checklist

Before publishing a kernel that uses one of the newer groups, I record:

CheckWhy it matters
CUDA Toolkit and -archHeaders and hardware instructions change over time.
Alignment and layoutPacked types and vector loads require correct addresses and shapes.
Numerical referenceLow precision and fast math need a tolerance and error report.
Synchronization scopeWarp, block, and device visibility are different contracts.

For an architecture-specific feature, keep a small compile and correctness test next to the kernel. It is much easier to diagnose a missing overload or unsupported instruction in a ten line test than inside a large fused model.

Closing perspective

The CUDA intrinsic catalog looks like a long list when viewed in isolation. It becomes more manageable when each name is attached to a boundary:

  • A vector type describes how values are grouped in memory.
  • A conversion describes how the number changes representation.
  • A math intrinsic describes a precision and accuracy tradeoff.
  • A warp primitive describes a lane communication scope.
  • A barrier or fence describes when work and memory become visible.
  • An atomic describes a conflicting update at a chosen scope.
  • An asynchronous copy describes a staged movement contract.
  • A tensor core interface describes a matrix instruction and its layout.
  • A qualifier describes where code or storage is allowed to live.

The best kernels are rarely the ones with the most intrinsic names. They are the ones where every name answers a real question about storage, arithmetic, communication, or hardware. Start with the clearest type and scope, measure the bottleneck, and only then move toward the lower level interface.

References

About the author

Ritesh Yadav works as an AI/ML Engineer. He writes independent research notes on ML performance, infrastructure, and systems, covering CUDA, low-latency inference, generative AI, distributed training, Kubernetes, and LLMOps.