How Floating-Point Determinism Affects LLM Reproducibility

Running the same prompt through the same LLM does not always give the same output, even with temperature 0 and a fixed seed. One reason is floating-point arithmetic. A GPU stores numbers with limited precision, so each addition and multiplication rounds its result. When a long sum is added up in a different order, the rounded result can come out slightly different. The order can change when the batch size changes, when the server picks a different kernel or when the model runs on different GPUs.

Usually the difference is too small to change which token the model picks. It only changes the output when two candidate tokens have almost the same score. Then a tiny numerical change can flip the choice. Because each new token is fed back into the model as input for the next one, the rest of the answer can change with it. A seed and temperature 0 control sampling. To reproduce an output exactly, you also have to control how the numbers are computed.

Rounding depends on the order of additions

Because each step rounds, floating-point addition is not associative. (a + b) + c can give a different result than a + (b + c), even though the two are equal in ordinary math. NVIDIA's floating-point guide shows this with three single-precision values. Python shows the same effect in double precision:

>>> (0.1 + 0.2) + 0.3
0.6000000000000001
>>> 0.1 + (0.2 + 0.3)
0.6

Floating-point math is still deterministic. The same sum, added in the same order, gives the same bits every time. On a GPU, large sums run as a parallel reduction, which adds many values through a tree of partial sums. Two valid trees can give different results with the same number format and the same rounding mode.

LLM inference is full of these sums. Each value in the output of a matrix multiplication is a dot product, and GPU kernels split that work into tiles. A different tile size changes the order in which the partial sums are combined. So does Split-K, which divides each long dot product across several cores. In attention, each score is a dot product, the softmax divides by a sum of exponentials over all the scores, and the output is a weighted sum across tokens. RMSNorm and LayerNorm sum values or squares across the hidden dimension. The output layer adds up products to get one logit, a score, for each token in the vocabulary. When a model runs on several GPUs, an all-reduce combines their partial results, and the GPU count and connection layout decide the shape of that reduction.

What changes the order on an inference server

The order comes from the kernels and from how the server runs each request. Servers with continuous batching group incoming requests to keep the GPU busy, so the batch a request lands in depends on the traffic at that moment. The batch shape can decide which kernel runs and how the work is split. Chunked prefill, which processes a long prompt in several pieces, can change the path too, and so can stream scheduling and available workspace. A different GPU model or GPU count, a new library version or another tensor-parallel layout changes it as well.

PyTorch's documentation on numerical accuracy warns about this batch effect. A batched computation is not guaranteed to match, bit for bit, the same computation done one element at a time. An operation on a slice of a tensor is not guaranteed to match the same slice of the full result either.

Thinking Machines Lab argues that this is the main reason inference endpoints are nondeterministic. In their analysis, a typical LLM forward pass has no atomic adds (GPU operations whose order depends on which thread finishes first), so the server gives the same output when it processes the same set of requests again. For a single user, though, the other requests are not fixed. The server's load decides the batch size, and kernels that are not batch-invariant can return slightly different numbers for the same request at a different batch size. A request that gives the same output every time you test it alone can still change in production, where it shares batches with other traffic.

How one flipped token changes the answer

At temperature 0 the model uses greedy decoding and picks the token with the highest logit. If a rounding difference swaps the top two logits, it picks a different token, and neither temperature 0 nor a fixed seed prevents that. The seed only fixes the pseudorandom numbers used for sampling. It does not control which kernels run or the order in which values are added.

From the first different token on, the model continues from a different prefix. That one token can change the reasoning steps, the length of the response, a tool call, generated code or the final answer.

In the Thinking Machines report, the authors sampled 1,000 completions of "Tell me about Richard Feynman" at temperature 0 with Qwen/Qwen3-235B-A22B-Instruct-2507 and got 80 unique completions. The most common one appeared 78 times. All 1,000 were identical for the first 102 tokens. At token 103, 992 completions continued with "Queens, New York" and 8 with "New York City". With their batch-invariant kernels turned on, all 1,000 completions were the same. This is one model, one prompt and one serving setup, and the report is a company blog post that has not been peer reviewed.

The NeurIPS 2025 study found the effect strongest in reasoning models. The gap between their top two candidate tokens is often very small, and an early flip can lead to a different chain of thought and a different final answer.

FP32, FP16 and BF16

Precision sets how large each rounding error is. The order of operations is a separate problem, and one deployment can have both.

FP32 stores 23 significand bits, FP16 stores 10 and BF16 stores 7. More significand bits mean finer spacing between the values a format can represent, so each rounding step moves a number less. BF16 has the same 8 exponent bits as FP32, so it covers about the same range of magnitudes, but its values are spaced much more coarsely. FP16 has only 5 exponent bits, which makes overflow and underflow more likely than in BF16.

The NeurIPS study ran four models in 12 setups, all with greedy decoding. The setups combined two GPU types (NVIDIA L40S and A100), two GPU counts (2 and 4) and three batch sizes (8, 16 and 32). FP32 gave nearly identical results across the setups, FP16 varied moderately and BF16 varied the most. For DeepSeek-R1-Distill-Qwen-7B in BF16, the authors saw up to 9% variation in accuracy and differences of up to 9,000 tokens in response length. The study measured differences between setups. It does not show that repeated calls in one unchanged setup give different answers. Even FP32 does not guarantee identical results across library versions, GPU architectures, kernels or schedules.

The authors write that FP32 doubles memory use and inference time compared to BF16. Mixed precision saves memory. A kernel can keep weights and activations in FP16 or BF16 and accumulate the products in FP32, and steps such as normalization or the final logits can run in higher precision. The authors' own proposal, LayerCast, stores the weights in BF16 and converts each one to FP32 just before its matrix multiplication, so all computation runs in FP32. In their tests it was about as stable as full FP32 and used 34% less memory.

Which kind of reproducibility do you need?

"Reproducible" can mean four different things, and each needs different controls:

Matching text does not prove that the tensors matched, and a request that repeats when it runs alone can still change in another batch. An aggregate score reproduced on another GPU says nothing about whether each token matched. Batch invariance covers batch size and request order, and the NeurIPS authors point out that it does not cover a change in tensor-parallel size or GPU type. Bitwise equality is the hardest target to meet, so require it only when approximate equality is not enough.

Deterministic settings and their limits

PyTorch's reproducibility documentation says that identical results are not guaranteed across PyTorch releases, individual commits or platforms, or between CPU and GPU runs, even with identical seeds. torch.use_deterministic_algorithms(True) makes supported operations use deterministic algorithms and raises an error when an operation has no deterministic version. The documentation also warns that deterministic operations are often slower.

TensorFlow's op determinism mode, tf.config.experimental.enable_op_determinism(), targets ops whose threads add floating-point numbers in a nondeterministic order, which happens mostly on GPUs. The documentation warns that the program may run slower and that some ops raise an error because they have no deterministic version yet. It also says determinism requires the same hardware and software in each run.

cuBLAS returns bitwise identical results on every run only with the same toolkit version, on GPUs with the same architecture and the same number of SMs (streaming multiprocessors). The guarantee no longer holds when several CUDA streams are active. The documentation lists fixes for that case, such as giving each stream its own workspace or setting CUBLAS_WORKSPACE_CONFIG to :16:8 or :4096:8. The first value may limit performance, and the second uses about 24 MiB more GPU memory. cuDNN gives the same bits across runs for most routines on the same GPU architecture, and none of its routines guarantee that across architectures.

As of September 2026, vLLM's batch invariance mode is in beta. You turn it on with the environment variable VLLM_BATCH_INVARIANT=1. It needs an NVIDIA GPU with compute capability 8.0 or higher (Ampere or newer) or an Intel XPU with Triton, and the documentation lists the models it has been tested on. The mode uses deterministic kernels and turns off some optimizations, such as custom all-reduce in tensor-parallel mode, and vLLM warns that it can reduce performance.

What determinism costs

Thinking Machines tested their batch-invariant kernels, which they had not optimized much, on one GPU running Qwen3-8B with 1,000 requests of 90 to 110 output tokens each. Default vLLM finished in 26 seconds. The deterministic version took 55 seconds, and 42 seconds with an improved attention kernel.

A 2024 study from the SC24 workshops measured parallel sums on GPUs. It is a microbenchmark and does not measure LLM serving. The gap between the fastest nondeterministic sum and a deterministic tree reduction was under 0.2% on an NVIDIA V100 and up to 7.8% on an NVIDIA GH200. On an AMD MI250X, a deterministic method was the fastest of all.

The cost depends on the hardware, the model and the workload, so it has to be measured on the setup you plan to run. Test with your own model, precision, GPU count and backends, with realistic prompt and output lengths, and under concurrent load as well as with single requests. If exact regeneration costs too much, use a weaker target: compare logits within a tolerance, check the margin between the top two tokens, judge answers by meaning or cache approved outputs.

How to test and record the setup

A final answer can match while the logits underneath differ. It can also change because of one near-tie while everything else is almost identical. So compare more than the final text. Record the logits or token probabilities, the position of the first token that differs and the margin between the top two candidates at that position. The first divergent token shows where a small numerical difference turned into a different choice. Run the same prompts alone, under concurrent load and in different batch mixes, then compare the results.

Save the full setup next to the evaluation results:

• model revision, weight hash, tokenizer, prompt template, decoding settings and seed
• precision of weights, activations, accumulators, KV cache and communication buffers
• GPU model, architecture, count and topology
• driver, CUDA, cuBLAS and cuDNN versions
• framework and inference engine versions
• tensor, pipeline, data and expert parallelism settings
• attention, matrix multiplication, all-reduce and quantization backends
• batch limits, scheduler policy, request order, concurrency and chunked prefill settings
• deterministic flags, workspace settings and any fast paths you turned off

For strict audit replay, store the approved responses and their token sequences instead of relying on regeneration. After a driver, library or hardware change, the same request may no longer produce the same text.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com