If LLMs Are Pure Functions, Where Does the Non-Determinism Come From?

TL;DR

An LLM is deterministic in theory: the same input, weights, and exact computation should produce the same logits.

But deployed LLM systems can still vary because of sampling and tiny numerical differences from floating-point arithmetic, parallel execution, batching, hardware kernels, precision settings, and compiler optimizations.

Even at temperature zero, near-tied token scores can flip, and one small change can cascade through the rest of the generated answer.

So, the model is deterministic in principle, but the full text-generation system is not always deterministic in practice.

Technically, during inference, neural networks, including Large Language Models, can be understood as pure functions.

A pure function is a function where the same input produces the same output, as long as nothing else changes. A neural network takes an input, passes it through a fixed set of weights and mathematical operations, and produces an output.

Given:

• the same weights,
• the same input tokens,
• the same architecture,
• and the same exact numerical computation,

the model should produce the same output logits.

In that narrow sense, the forward pass of an LLM is just a mathematical function.

So why do people say LLMs are non-deterministic?

Why can the same prompt produce different answers?

The answer is that the non-determinism usually does not come from the abstract mathematical model. It comes from the process of turning that model into generated text on real hardware.

More precisely, the main sources are:

• sampling randomness,
• and numerical implementation variation.

The second category includes floating-point approximation, parallelized execution, batching, hardware kernels, memory layout, distributed inference, precision settings, and compiler optimizations.

To understand the difference, we need to separate the LLM as a mathematical object from the LLM as a deployed inference system.

The LLM as a Pure Function

A transformer model maps an input sequence to a probability distribution over the next token.

Conceptually:

input tokens → model → logits → probabilities over next token

The model does not directly output a sentence. It outputs scores, called logits, for possible next tokens.

If the input tokens and model weights are exactly the same, then in ideal mathematics the logits should also be exactly the same.

So the model forward pass is deterministic in principle.

The model gives you a distribution. Something else must decide how to choose from that distribution.

That “something else” is the decoding process.

The First Source: Sampling

The most obvious source of non-determinism is sampling.

After the model produces probabilities for the next token, the decoder has to select one token.

For factual completions, one token may dominate.

For example:

The capital of France is → Paris

But for open-ended prompts, many continuations may be valid.

For example:

Intelligence is...

The model may assign reasonable probability to many different continuations:

• the ability to solve problems
• the capacity to adapt to new situations
• not a single ability, but a collection of skills

If the decoder samples from the distribution, the next token is intentionally chosen with randomness.

This is where settings like temperature, top-p, and top-k matter.

Temperature controls how strongly the decoder prefers the highest-probability tokens. Higher temperature allows more variation. Lower temperature makes the output more conservative.

Top-p and top-k restrict the set of tokens from which the model can sample, but they do not remove randomness completely unless decoding becomes greedy.

So if sampling is enabled, the system is no longer behaving like a pure deterministic function from prompt to answer.

It is more like:

tokens + random seed → sampled next token

With a different random draw, you can get a different token.

And because generation is autoregressive, one different token can change everything that follows.

The Second Source: Numerical Implementation Variation

Even if you remove sampling, another issue remains: the mathematical model still has to be executed on real hardware.

The transformer may be a clean mathematical function on paper, but in practice it is evaluated using finite-precision arithmetic: float32, float16, bfloat16, quantized integers, or other optimized formats.

Real computers do not perform exact real-number arithmetic. They use approximate numerical representations.

This does not mean that floating-point arithmetic is random.

If the exact same operations are performed in the exact same order, on the same hardware, using the same precision and numerical settings, the same rounding errors should usually repeat.

The problem is that deployed LLM systems do not always evaluate the computation through exactly the same numerical path.

Floating-Point Arithmetic Is Approximate

Floating-point arithmetic is approximate because real numbers have to be represented using a limited number of bits.

Tiny rounding differences can appear during matrix multiplications, attention calculations, normalization, and reductions.

Usually these differences are extremely small. Most of the time, they do not matter.

But during decoding, the model eventually has to rank tokens.

If the highest-probability token is far ahead, tiny differences will not change the result.

But if two tokens are nearly tied, a tiny numerical difference can flip their order.

For example:

Token A score: 12.000001
Token B score: 12.000000

A tiny numerical difference could turn this into:

Token A score: 12.000000
Token B score: 12.000001

Now greedy decoding may choose a different token.

So floating-point arithmetic gives us the underlying sensitivity: tiny numerical differences can sometimes become visible if they affect token ranking.

But floating-point arithmetic alone is not the whole story. Approximation becomes a reproducibility problem when the computation is not executed through the exact same path every time.

Parallelized Execution Can Change the Numerical Path

Modern LLM inference is massively parallel.

The computation is not performed as a simple, single-threaded sequence of operations. It is distributed across many cores on graphics processors or specialized accelerators.

This matters because floating-point arithmetic is not perfectly associative.

In real-number arithmetic:

(a + b) + c = a + (b + c)

But in floating-point arithmetic, this equality is not guaranteed exactly.

Because each operation rounds the result, changing the order in which values are added, multiplied, grouped, or reduced can slightly change the final number.

Parallelized execution matters because parallel hardware may change the order in which floating-point operations are grouped and reduced.

Parallel systems often perform large reductions and matrix operations in different orders depending on:

• hardware scheduling,
• kernel implementation,
• batching,
• memory layout,
• distributed inference,
• compiler optimizations.

So the same mathematical expression may be evaluated through slightly different computational paths.

Again, the differences are usually tiny.

But if the next-token probabilities are close, tiny differences can become visible differences in the generated text.

This is why “same prompt, same model” does not always mean “bit-for-bit identical output” in deployed systems.

Why Temperature Zero Is Not a Perfect Guarantee

People often say:

Set temperature to zero if you want deterministic output.

This is mostly true, but not perfectly true.

Temperature zero usually means greedy decoding: always choose the highest-probability token.

That removes sampling randomness.

But it does not remove all possible non-determinism in deployed systems.

If the logits are exactly reproducible and the token ranking is stable, temperature zero should return the same output.

But if implementation-level numerical differences cause tiny logit changes, and two candidate tokens are almost tied, greedy decoding can still diverge.

So temperature zero gives you practical determinism most of the time.

But strict reproducibility requires more:

• same model weights
• same input tokens
• same decoding parameters
• same numerical precision
• same hardware
• same kernels
• same batching behavior
• same execution order
• same tie-breaking rules

That is a much stronger requirement than just setting temperature to zero.

Autoregression Amplifies Small Differences

An LLM generates one token at a time.

Each generated token is appended to the context before the next token is produced.

So the process looks like this:

prompt → token 1 → token 2 → token 3 → token 4...

If token 1 changes, token 2 is now generated from a different context.

Then token 3 changes.

Then token 4 changes.

So a tiny difference at one step can cascade into a noticeably different final answer.

The original cause may have been microscopic: a sampling choice, a tiny numerical difference, or a near-tie between two token scores.

But the final effect can look like a completely different response.

This is why LLMs can feel more unstable than the underlying computation really is.

Conclusion

The LLM as a mathematical function is deterministic. The LLM as a text-generating system is not always deterministic, because sampling and implementation-level numerical variation sit between the pure function and the final text.

Sampling is the obvious source of randomness. If the decoder samples from a probability distribution, different random draws can produce different tokens.

Numerical implementation variation is more subtle. Floating-point arithmetic is not randomness by itself. If the same operations happen in the same order under the same conditions, the same rounding errors should usually repeat. But deployed inference systems can involve different hardware paths, kernels, batching patterns, precision settings, memory layouts, compiler optimizations, and parallel execution orders. Since floating-point arithmetic is approximate and order-sensitive, those implementation differences can create tiny logit differences.

That is where the non-determinism comes from.

This is especially important in Finance AI, where a fluent answer is not enough; the answer must be reproducible, auditable, and traceable back to the exact data and calculation used.

So in financial systems, the LLM should not be treated as the source of truth, but as a reasoning and language layer around deterministic computation, validation, and provenance.