Why the Same AI Prompt Gives Different Answers

An AI step in your automation can return different output for the exact same input, and setting temperature to 0 does not fully fix it. On a hosted API, the model runs your request in a batch with other people's requests, and the size of that batch shifts with server load. That changes the low-level arithmetic just enough to flip a token now and then, so identical inputs quietly drift apart. The durable fix is not chasing a determinism the API cannot promise. It is building the automation so it does not break when two runs disagree.

Most explanations stop at "set temperature to 0" or "it's floating-point math." Both miss what actually happens, and both leave you debugging a workflow that passed every test and then behaved differently in production. Here is the real cause, and the handful of build decisions that make it a non-issue.

Temperature 0 controls the choice, not the machine

Temperature is the knob for how much randomness the model uses when it picks each next token. At temperature 0 the model stops sampling and takes the single highest-probability token every time. That part is genuinely deterministic. Given the exact same probabilities, greedy selection returns the same token.

The catch is that temperature 0 only fixes the selection rule. It does nothing to guarantee that the probabilities feeding that rule are identical from one run to the next. Anthropic says this directly in its own glossary: at temperature 0 results are near-deterministic, and identical inputs may still produce different outputs. OpenAI documents the same reality around its seed parameter, which makes a best effort to sample deterministically but explicitly states that determinism is not guaranteed, and points you at a system_fingerprint value to watch for backend changes. Two vendors, one message. The randomness you are chasing does not live in the sampler.

The real reason: your request rides in a batch you do not control

For years the standard answer was floating-point math plus GPU concurrency. Parallel hardware adds numbers in a nondeterministic order, floating-point addition is not associative, and those tiny rounding differences pile up. That story is not wrong, but it is not the actual culprit, and a September 2025 analysis from Thinking Machines Lab pinned down the one that is.

Their finding: the individual GPU kernels are run-to-run deterministic. Run the same matrix multiplication twice on the same hardware and you get bitwise identical results. What those kernels lack is batch invariance. The numerical result of an operation like normalization, matrix multiply, or attention changes when the batch size changes, because the order in which the kernel reduces its values depends on how many requests are packed together.

Now connect that to how a real endpoint works. A hosted inference service does not run your request alone. It packs your request into a batch with whatever other traffic is arriving at that instant, and the batch size fluctuates with server load, which you have zero visibility into. Different batch size, slightly different arithmetic, occasionally a different token, and from that one token the outputs diverge. In the authors' words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load, and therefore the batch size, varies. You changed nothing between the two runs. Someone else's traffic did.

That is the killer detail most guides never reach. The nondeterminism is not really in the model or in your parameters. It is in the fact that you are sharing a machine, and your neighbors are noisy.

What breaks when your automation assumes the output is stable

The problem is not the variation itself. A slightly reworded summary rarely matters. The damage comes from build decisions that silently assume two runs will match. These pass in testing, because in a quiet test run they usually do match, and then fail intermittently once real traffic makes the batch size jump around.

The assumption baked in	What it quietly breaks	How it shows up in production
Regression tests assert the exact output string	Tests flake for no code reason	A green suite goes red on reruns with no change, so people stop trusting it
Idempotency or dedup key is a hash of the model's output	The same input creates a "new" record	Duplicate records appear whenever the output shifts by a character
A cache is keyed on output equality	Cache never hits, or serves stale mismatches	AI cost stays high because the cache silently misses
A branch matches the model's free-text phrasing	The wrong path fires	"If reply contains 'approved'" misses when the model writes "Approved."
Acceptance is "it worked in ten runs"	Rare divergence ships unnoticed	The eleventh run in production takes a path nobody tested

None of these are exotic. They are the default way people wire an AI step when they picture the model as a pure function. It is not one. Treating it like one is the actual bug.

How much reproducibility can you actually buy

You have three real levers, and it helps to know exactly what each one gives you before you lean on it.

Lever	What you get	The ceiling
Temperature 0	Greedy selection, sharply reduced variance, near-deterministic	Still batch-sensitive, so not reproducible
Seed plus watching system_fingerprint	Best-effort identical output while the fingerprint holds	Not guaranteed, and breaks silently when the backend config changes
Self-host with batch-invariant kernels	True bitwise reproducibility	Requires running the model yourself, not available on hosted OpenAI or Anthropic endpoints

For most operators the honest read is that the top of that table, real reproducibility, is not on the menu unless you are hosting your own model and have adopted the batch-invariant kernels Thinking Machines Lab published. On a shared hosted API you get near-determinism at best, and you should treat it as such.

One thing this is not: a model version changing under you. When a floating alias like gpt-4o repoints to a new snapshot, that is a genuine version change, and the fix there is to pin the model version and re-test before you upgrade. This article is the other axis. Same model, same input, different output, because of how the request was batched. Both are real, and they call for different responses, so it is worth knowing which one you are looking at.

Design the automation to tolerate variation

Once you accept that identical output is not guaranteed, the fixes are straightforward and mostly about not depending on equality where you never needed it.

Validate the shape and the values, not the exact string. This is the same discipline that ends broken JSON: use structured outputs so the fields are guaranteed, then check that the values are sane. A record with amount greater than zero and a real date passes whether the model phrased its reasoning one way or another. You almost never actually need the output to be byte-identical. You need the extracted values to be right.

Key idempotency and deduplication on the input, not the output. If you need to know whether you already processed an item, hash the source payload or use a stable business ID, never a hash of what the model returned. The input is stable. The output is not.

Branch on parsed fields, not free text. Ask the model for an enum or a boolean field and route on that, so your workflow reads decision == "approve" from a constrained field instead of scanning prose for the word "approved." Constrained fields do not wobble the way sentences do.

Keep temperature at 0 and constrain the output anyway. Low temperature plus a tight schema still meaningfully cuts variance, which is worth having. It just is not a substitute for the design choices above.

And when a step genuinely must produce the same result every single time, do not ask the model to. Use a rule. Deterministic logic is the correct tool for anything that has to be exactly reproducible, and the model is the right tool for the fuzzy judgment around it. If you are wiring these together, a workflow automation system is where that split between rule and model gets drawn cleanly. We run typed extraction this way in our own permit data pipeline at ma-permits.geninfos.com, where the model reads messy source text but a rule, not the model, decides what counts as a duplicate.

What to do next

Open any AI step whose output feeds a comparison, a cache key, a dedup check, or a text-matching branch, and change what it depends on. Validate values instead of exact strings, key dedup on the input, and branch on a constrained field. Leave temperature at 0 for the variance reduction, but stop treating it as a guarantee it was never designed to give.

If you have an automation that behaves differently on reruns and you cannot tell whether it is batching variation or a model that changed underneath you, tell us what it does and we will help you find which axis you are on and where to add the guard.