The Behaviour Was Never Explicitly Designed
Agentic systems are already making operational decisions beyond predefined reasoning paths - what does this mean for enterprises deploying AI at scale.
The rules were approved. The thresholds were set. At the point of deployment, the fraud system was considered controlled. It wasn’t enough.
Card-not-present fraud accounted for around 70% of global card fraud losses in 2024 - totalling $33.41 billion, according to the Nilson Report. Payment networks process billions of payments each year. Risk decisions are made in milliseconds. The decision of which transactions go through and which do not is made by the system, in real time, before any human sees it.
An agentic fraud system deployed at scale will encounter situations it was never designed for. The question is not whether that happens. The question is whether the organisation has defined what it should do when it does.
The financial, reputational and regulatory liability that arises from those decisions sits with the institution, not the system.
This piece traces one journey in full: how an agentic fraud system moves from detection to decision to action - and what that means for the institution depending on it.
Dimitrius Pacheco and Neha Kabra cover it across five steps.
Neha Kabra, writes about AI, strategy and enterprise transformation from two decades inside banking and McKinsey, with a focus on how complex change actually lands.
Dimitrius Pacheco, writes about AI systems, embedded infrastructure, and operational dependency - examining how intelligent systems reshape decision-making inside institutions.
In this piece
1. The incident that starts it — how a live attack exposes the system in real time
2. What the system knows — how signals become interpretation, not certainty
3. How the system acts — decisions with direct economic consequences, made without human review
4. Where control actually sits — distributed across actors who did not design the system together
5. What becomes non-optional — and why the organisation is now dependent on behaviour it has not fully defined
1. The incident that starts it - how a live attack exposes the system in real time
A coordinated card testing attack starts quietly.
Low-value transactions begin hitting the system in rapid bursts. One dollar here. Three dollars there. Different merchants, devices, and geographies. Most attempts fail. A few pass. That is enough - to cause real damage.
The successful cards move quickly into higher-value transactions. The pattern has already shifted before a human fraud team can review it. The exposure is now live.
Stripe documented blocking over 20 million card testing attempts in a single day during peak attack periods. At that volume, every approved transaction carries potential loss. Every blocked transaction carries potential customer friction. Every extra minute changes the economics of the incident.
Manual intervention cannot carry that load. By the time a case is reviewed, thousands of decisions have already been made. The constraint is not intelligence. It is decision velocity — large payment networks processing thousands of transaction events per second, with fraud decisions expected within milliseconds, across issuers, merchants, devices, and geographies simultaneously.
Under live conditions, the attack changes faster than human review cycles can respond. The operational response moves inside the fraud system. Humans are no longer reviewing decisions before action occurs. They are supervising behaviour that is already in motion.
Before reading further, one distinction matters: between a traditional rules-based system and an agentic system.
A traditional rules-based fraud system operates on explicit logic. A velocity spike above a set threshold triggers a block. A transaction from a flagged geography routes to a review queue. The response is traceable back to a specific rule. When a regulator asks why a transaction was blocked, the answer is precise - this rule, applied to this condition.
An agentic system operates on goals. Give it the goal of reducing fraud losses while minimising customer friction and maintaining compliance - and it will pursue that goal through a series of actions based on its own judgment about the situation. It does not match the pattern to a rule. It reasons toward the response that best satisfies the defined objective under those specific conditions. When an attacker adapts — spacing transactions, rotating device fingerprints, mimicking legitimate behaviour — the rules-based system fails at the point the rule was not written for.
The agentic system reasons across the adaptation. The same objective. A different path to it every time.
With an agentic system, the path is generated through judgment. The institution can state the goal it gave the system. It cannot always enumerate the exact reasoning that produced a specific action. That gap is where the accountability challenge lives.
2. What the system knows — how signals become interpretation, not certainty
An agentic fraud system does not ‘know’ fraud. It observes incomplete signals and reasons through ambiguity in real time.
Every transaction arrives with contextual information attached: device fingerprint, behavioural history, merchant category, geographic location, transaction velocity, network relationships, and similarity to prior events.
Large payment networks evaluate these signals across thousands of transactions per second, with decisions expected within milliseconds. None of these signals independently prove fraud — and the agentic system is continuously determining which action best satisfies competing objectives under that ambiguity: reduce fraud losses, minimise customer friction, preserve authorisation speed, maintain compliance.
A transaction from a new geography may indicate account compromise. It may also indicate that the customer just landed in another country. A sudden increase in transaction velocity may indicate card testing. It may also reflect legitimate behaviour during a live event or holiday surge.
Under adversarial conditions, ambiguity compounds rapidly. Attackers continuously adapt to detection behaviour in real time. Previously reliable indicators can lose predictive value within hours. The system therefore encounters situations where neither the inputs nor the appropriate response are fully represented inside prior examples.
The agentic system interprets under incomplete information — and acts probabilistically before the ambiguity resolves. Two consequences follow: customer friction and regulatory exposure.
Customer friction lands first. When the agentic system balances fraud containment against customer friction, it is making a business decision with a P&L consequence. A false positive blocks a legitimate customer — that is a service failure. A false negative approves a fraudulent transaction — that is a financial loss. At scale, both happen simultaneously, continuously, without review. The customer experiences the outcome of a threshold set months earlier, applied to a situation it was never designed for.
Regulatory exposure follows. Regulators increasingly expect institutions to explain not just what the system decided, but why — and whether that decision sat within the defined risk appetite. When the system is making calls faster than human oversight cycles, the audit trail exists. The accountability chain often does not.
The system is making that call continuously, without referral.
3. How the System Acts — decisions with direct economic consequences, made without human review
The system is no longer limited to detection. It acts autonomously on probabilistic interpretation.
Once risk thresholds are crossed, the system can decline transactions, escalate authentication requirements, suppress merchant activity, freeze accounts, reroute approvals, trigger investigations, or dynamically adjust scoring behaviour across connected transaction flows in real time.
It increasingly generates behavioural responses dynamically - from probabilistic reasoning, model adaptation, feedback loops, and adversarial pressure operating simultaneously. Under live conditions, the system encounters scenarios never fully anticipated during training or deployment.
The organisation defines governance boundaries and operational objectives, but the exact path through which the system arrives at a decision is no longer fully enumerable in advance. Institutions are no longer directly governing operational behaviour itself.
In 2024, TD Bank was fined $3.09 billion for systemic AML failures. The fine was material. The consequence that mattered more was operational: the regulator capped TD Bank’s asset growth in the United States. The institution’s ability to expand was constrained not by capital or competition — but by its failure to govern systems making decisions it could not adequately explain. The accountability chain did not hold. The institution paid for the gap.
In an agentic system, that gap widens. The decisions are faster, the paths less traceable, and the conditions under which they are made increasingly outside the scope of what was defined at deployment. The institution that waits for a regulatory intervention to discover its accountability gap is already two years behind the problem.
4. Where Control Actually Sits - distributed across actors who did not design the system together
A payment transaction has always passed through multiple actors:
The card network sets interchange rules, velocity limits, and merchant category restrictions.
The processor applies routing logic, retry behaviour, and step-up authentication triggers.
The issuing bank approves or declines.
The liability has always landed with the bank.
That structure is not new.
What changes in an agentic environment is not fragmentation itself - but the behaviour emerging from fragmented systems acting autonomously at machine speed. In a rules-based world, the stack was comparatively inert. Each layer executed predefined logic independently, and the interactions between them were stable enough to model, audit, and reconstruct after the fact.
In an agentic world, each layer is continuously adapting in real time. The fraud system is adjusting thresholds dynamically based on observed behaviour. The processor’s routing logic is optimising simultaneously for latency, approval rates, and fraud exposure. The network is responding to the same attack patterns at the same time. Adaptive behaviour shifts across processors, issuers, and merchants simultaneously.
The outcome that emerges from those interactions was not explicitly designed by any single participant in the system.
The 2010 Flash Crash offers a useful reference point. Automated trading systems interacting across fragmented markets contributed to nearly $1 trillion in market value temporarily disappearing within minutes. No individual algorithm intended the outcome. The instability emerged from autonomous systems interacting under feedback pressure.
Financial infrastructure is moving toward similar machine-speed decision environments.
Visa processes more than 65,000 transaction messages per second during peak periods. Under those conditions, autonomous systems are continuously reacting to one another across infrastructure no single participant fully observes.
That creates an interaction observability problem.
Each institution can fully observe its own system.
The issuing bank sees its fraud model.
The processor sees its routing logic.
The network sees transaction velocity.
But no participant can fully observe the behaviour emerging from the ecosystem as a whole.
The institution ultimately accountable for the agentic AI outcome may be responding to behaviour that did not originate inside its own system - and cannot be reconstructed from within any single layer of the stack.
That is where control begins to structurally separate from visibility.
And once that separation occurs, governance becomes fundamentally more difficult.
5. What Becomes Non-Optional - and why the organisation is now dependent on behaviour it has not fully defined
The dependency is structural. An agentic fraud system running at network scale cannot be slowed without accepting loss. Fallback to human review becomes operationally impossible once the system is processing thousands of decisions per second across live transaction flows. The institution is dependent on behaviour it has not fully defined - and that dependency compounds across the distributed layers Section 4 describes.
Three things follow from that - and only two of them are fully within the institution’s control.
First, Model risk management at regulated institutions was built to validate model weights and backtest performance. Agentic systems require something different.
The question is not whether the model performs accurately under expected conditions. It is whether the objective the system is pursuing produces acceptable behaviour across conditions it was never trained for.
That requires adversarial testing, multi-model challenge mechanisms where independent models test and contest high-impact decisions, and human oversight designed around behavioural boundaries rather than output review.
Goal-space explainability is the second requirement. Regulators are already moving in this direction - the Federal Reserve’s SR 11-7, the EU AI Act, and DORA increasingly focus not just on model accuracy but on governance, explainability, and accountability under live conditions.
When a regulator asks why a specific transaction was frozen, output logs are no longer sufficient. The institution needs to reconstruct what the system was optimising for at that moment - and whether that objective was sanctioned for those conditions. Goal-level audit trails, in most institutions, do not yet exist.
The ecosystem gap is the third problem - and it sits outside the institution. The behaviour the institution is accountable for emerged from layers it does not fully control - processors, networks, authentication providers, third-party risk systems all adapting dynamically outside its direct oversight.
The full AI observability picture requires visibility across that ecosystem. The industry has not agreed on how to provide it.
Two of these three problems sit inside the institution. The third does not - and the industry has not yet agreed on how to address it.
Observability stops being a technical feature at that point. It becomes operational infrastructure.
I write about AI, strategy and enterprise transformation from two decades inside banking and McKinsey, with a focus on how complex change actually lands.
 | A guest post by
|
Ok, I'm about to take out all my money and hide it under my pillow. Joking aside. Are you saying you really don't have a solution for this problem yet? This is a serious topic! Why aren't the big AI companies already working on a solution? A friend of mine works on an evidence infrastructure for agentic AI, but his solution is not yet on the market.
This is one of the clearest articulations I’ve seen of where the accountability chain actually breaks; not at the governance layer but at the moment the system acts without a decision spine underneath it.
The framing I work with separates the model (what’s predicted) from the reasoning layer (what should happen given that prediction). In a well-structured agentic system, that reasoning layer (what I call Procedural Intelligence) handles the validation and decision gating between signal and action. It’s where Context Filters determine what the system is actually seeing, Decision Gateways hold the logic for what should happen next, and Fallback Paths define what the system does when the situation falls outside designed conditions. The gap you’re describing in Section 5 (goal-space explainability, behavioral boundaries, audit trails that don’t yet exist) become tractable once the decision architecture exists before deployment, not as a retrospective. You can’t reconstruct reasoning you never structured.
The ecosystem observability problem in Section 4 is harder and you’re right that it sits outside any single institution. But the intra-system version of the same problem is solvable and it starts with separating prediction from decision before the action occurs.