Newer AI Models Missed More Fraud, Not Less

A newer AI model can miss more payment fraud while looking more accurate on paper. Coinbase’s benchmark shows why model upgrades need a fraud-specific test before they reach a live payment flow.

Coinbase’s results should change how payment companies think about AI upgrades. In a historical replay of its Onramp payment screening process, newer versions of three major model families caught less fraud, captured less fraud value, and delivered lower overall detection scores than the versions they replaced. The policy stayed fixed. The transaction history stayed fixed. The model changed, and performance moved in the wrong direction.

That finding supports a straightforward rule: a newer general-purpose model carries no automatic fraud-detection advantage. Payment providers should treat every upgrade as a new component that requires testing against their own fraud patterns, decision thresholds, latency limits, and operating costs.

Coinbase reported the results on Oct. 7 after replaying 16,140 historical transactions involving 7,293 users. The test included 813 confirmed fraudulent transactions. The data covered nine weeks before Coinbase deployed its risk agent, with matured fraud cases retained and legitimate traffic sampled for comparison.

Each candidate model reviewed recent transaction behavior under the same guidance. Coinbase also used the same policy to turn model risk classifications into final decisions. That setup matters because it isolated the model’s behavior within an existing screening system. The company did not redesign the entire fraud stack for one model and leave the others with older rules.

Coinbase compared three pairs:

Every newer model produced lower recall, lower F1, and lower dollar-weighted recall in the replay. Both Opus and Sonnet also produced lower precision.

This kind of fixed replay gives payment teams a useful warning. A model can perform well on broad language or reasoning benchmarks and still behave differently when it reviews payment histories, account activity, and fraud signals. The relevant question is the one inside the payment workflow: how many fraudulent transactions does the candidate catch, how much fraud value does it stop, and what legitimate activity does it disrupt?

Coinbase’s Onramp fraud-screening replay found that newer versions of three major AI model families performed worse under the same policy and historical transaction cohort: they posted lower recall, F1, and dollar-weighted recall, with some also losing precision. The results show why payment providers should treat every model replacement as a fraud-risk change, not a routine upgrade. Teams should validate candidates on mature local data, evaluate fraud coverage and fraud value alongside customer friction, latency, reliability, and cost, then use protected rollout controls before granting a new model production authority.

Fraud screening has an uneven cost structure. Missing a high-value fraudulent payment can matter far more than missing a small transaction. A single summary score can hide that difference, so Coinbase examined several measures.

Recall measures the share of confirmed fraud cases a model catches. Dollar-weighted recall measures the share of total fraudulent value it catches. F1 combines precision and recall into one score. Precision measures the share of transactions flagged as fraudulent that were actually fraudulent.

The newer Sonnet model showed the largest deterioration. Its recall dropped 22.2 percentage points, while dollar-weighted recall fell 22.9 points. That means the model missed a larger share of fraud cases and allowed a larger share of the fraudulent dollar value to pass through the replay.

Opus posted a smaller recall decline of 0.8 points, yet its overall results still moved downward. Small changes can matter in a payment system processing large volumes, particularly when the missed cases cluster around expensive transactions or repeat attack patterns.

GPT supplied the clearest example of why one better metric cannot settle an upgrade decision. Its precision increased by 11.5 percentage points. At the same time, recall fell by 20.7 points and dollar-weighted recall fell by 21.8 points. The model’s fraud flags were more accurate among the cases it selected, while many more fraudulent cases and dollars escaped detection.

A payment provider that watches precision alone could approve that change. Customers and finance teams would experience the other side of the result: more fraud reaching the payment flow.

The results challenge a common assumption in model operations. Developers often expect a newer model to preserve an existing task’s performance while adding better reasoning, broader knowledge, or stronger instruction following. That expectation works poorly when a model’s behavior depends on a narrow prompt, a specific transaction format, and a fixed threshold policy.

A model upgrade can change several things at once:

Those changes can alter the distribution of fraud flags even when the prompt and policy remain unchanged. A newer model may become more selective. That selectivity can lift precision while reducing recall. Another model may respond differently to sparse histories, unusual merchants, new devices, or transaction sequences that resemble legitimate behavior.

The model does not need to be generally worse to be worse for this task. It only needs to shift its errors toward the cases a payment provider most needs to catch.

That is why general model benchmarks have limited value for payment screening. A strong score on reasoning, coding, or text understanding says little about whether the model catches a particular provider’s fraud patterns under production conditions. Fraud detection depends on local data, account behavior, chargeback timing, policy choices, and the economic value of errors.

The important lesson is not that general-purpose models are unsuitable for fraud screening; it is that version labels are not evidence. Fraud detection is an asymmetric, local decision problem shaped by a provider’s transaction mix, attack patterns, thresholds, and loss exposure. A model that seems more capable overall can become more selective in exactly the wrong way, improving the appearance of its flags while letting costly fraud pass. The disciplined response is to make replay evaluation, error analysis, and controlled rollout gates standard parts of model operations.

Coinbase previously tested a different change: adding selective LLM review to an existing combination of models and rules. That agent-enabled flow recorded 30% fewer fraudulent transactions and 22% less fraud value in the company’s online experiment.

That result should be kept separate from the model-version replay. The online test evaluated an added review flow. The historical replay compared newer and older versions inside a fixed setup. One result concerns system design; the other concerns model replacement.

This distinction is easy to lose when teams describe an AI system as a single product. A payment screener usually contains several moving parts:

An improvement in one layer cannot validate a change in another layer. The fact that selective LLM review helped in one Coinbase experiment does not mean every newer LLM will improve the existing reviewer. Likewise, a custom model’s strong benchmark result does not prove that it will produce the same latency or fraud coverage in every production path.

Coinbase’s Oct. 8 disclosure described a different approach. The company post-trained a Qwen3.5-9B model using historical fraud outcomes and deterministic rewards that balanced fraudulent and legitimate examples. That specialized model exceeded Opus 4.5 across four fraud-detection metrics.

The reported gains were substantial. F1 improved by 9.6 percentage points, and dollar-weighted recall rose by 35.4 points. Those figures offer a practical counterexample to the idea that payment providers must wait for a larger general-purpose model to improve their fraud systems.

A smaller model trained around the task can fit the operating environment more closely. Its training signal can reflect the provider’s actual fraud labels, the relative cost of false positives and false negatives, and the kinds of transaction histories available at decision time. Deterministic rewards can also make the tradeoff explicit rather than leaving it to a broad model’s learned preferences.

The result still requires careful interpretation. Coinbase’s custom-model evaluation and its general-model comparison were separate evaluations. Performance in one benchmark does not automatically transfer to another. Training data can also contain changing fraud patterns, delayed labels, and historical decisions that reflect old rules. A specialized model needs monitoring for drift, calibration, and new attack behavior.

Even with those qualifications, the direction is clear. Task-specific data and objectives can matter more than a model’s release date or general reputation.

Coinbase also reported a production latency difference. Median end-to-end LLM request latency measured 0.683 seconds for the custom model compared with 1.515 seconds for Opus 4.5, a 55% relative reduction.

Latency affects where an LLM can operate. A slower request may create payment friction, increase infrastructure expense, or force a provider to use the model only for selected cases. A faster model can support more reviews within a decision window.

The latency figures came from a separate measurement, so they should not be treated as part of the fraud-detection comparison. They still highlight an important deployment tradeoff. Payment teams need to measure at least four dimensions together:

A fast model that misses expensive fraud creates a poor system. A highly sensitive model that blocks too many legitimate customers creates a different operational problem. A useful evaluation records both outcomes rather than allowing one attractive metric to dominate the decision.

Coinbase said the replay does not establish customer losses from deploying the tested newer versions. The company also said it could identify the regressions without identifying their cause. Those limits matter. The replay measures behavior on a historical cohort, not the financial outcome of a production rollout.

The related SR-Fraud research has another constraint: the proprietary dataset cannot be released for independent replication. That limits outside testing and makes it harder to know how closely the results would generalize to other payment providers, regions, fraud types, or transaction mixes.

Those caveats reduce the reach of the claim. They do not weaken the operational lesson. A payment company does not need a public, universally representative dataset before it tests a candidate on its own historical cases. A provider already has the most relevant starting point: its fraud labels, its customer behavior, its decision policy, and its cost structure.

The right response to limited generalization is local validation. It is not blind confidence in a newer model.

A model version number is a weak reason to change a fraud system. Evidence from the provider’s own task is stronger. The Coinbase replay showed newer Opus, Sonnet, and GPT versions losing ground under the same decision setup, while a specialized model trained for the task produced much stronger reported results.

The message for payment leaders is direct: every model upgrade is a fraud-risk change until the provider’s data proves otherwise. Teams should test the exact configuration, weigh missed fraud by dollar value, study the errors, and include speed and reliability in the same decision.

A newer AI model may deserve a place in a payment stack. Its release date cannot earn that place. Task-specific evidence has to do it.