AI Agents Are Scaling Faster Than Accountability Systems

Companies are giving AI agents wider access to data, software, and business decisions. The control systems needed to track that access are developing at a slower pace, leaving a basic accountability gap at the center of the agent economy.

AI agents are moving from demonstrations into everyday operations. They write and test code, search internal files, make recommendations, schedule appointments, and in some cases buy products or take other actions for a user. Banks, insurers, and other large companies are examining similar systems for customer service, underwriting, fraud review, and back-office work.

The Partnership on AI report on the observability gap in AI agents puts a practical problem behind that expansion. PAI tested four agent systems and found six areas where the telemetry they produced failed to give operators a complete view of what the agents were doing. The report points to gaps in identity, delegation, human intervention, permissions, memory, reasoning, and event coverage.

That finding shifts the focus of AI safety toward ordinary enterprise infrastructure. An agent can be capable, useful, and carefully prompted while still operating inside a weak accountability system. When an agent acts through several tools and creates additional agents to handle subtasks, a company needs a durable record of who initiated each action, which permissions applied, what data the system used, and which person can stop it.

Nolla Health began a one-year Utah pilot focused on adults with mild to moderate acne. Patients pay $4.99 for the subscription, submit health information and facial photos, and receive an AI-generated treatment recommendation that can be sent to a pharmacy. Licensed clinicians remain available through telehealth, and cases involving drugs outside the program’s topical treatment list are routed to them.

The arrangement gives Utah a useful testing ground for a question that reaches well beyond dermatology: how much medical decision-making can be automated safely when the system has a limited scope, a low price, regular follow-up, and a clear path to human care?

A generative AI tool usually produces an answer for a person to review. An agent can carry that answer into the world by calling a tool, changing a record, sending a message, or approving a workflow. Each extra action adds a control point and another opportunity for a mistake, a bad instruction, or a malicious substitution.

That difference matters even in a consumer setting. A personal agent that searches for a hotel may later be allowed to reserve a room. An agent that compares products may gain permission to complete a purchase. The user has moved from receiving information to delegating authority.

Enterprise deployments multiply the consequences. An agent reviewing mortgage applications could retrieve property records, calculate risk, assign a recommendation, and send the file to another system. A small change in an input or permission could affect hundreds or thousands of applications. The software may complete each step quickly, while a human investigator sees the result only after the workflow has finished.

Mark Nitzberg, executive director of the Center for Human-Compatible Artificial Intelligence at the University of California, Berkeley, has described the danger as a conflict between an agent’s assigned objective and the rules governing its operation. An agent can keep pursuing a task when continuing the task creates a security, legal, or financial problem. The behavior can come from the system’s incentives and available tools, with no malicious intent required.

This makes accountability a design requirement. A company needs controls that follow the agent through every handoff and record the decisions that matter.

AI agents are gaining the authority to access data, call tools, change records, and make decisions, but the systems that track and govern those actions are not keeping pace. Partnership on AI’s research identifies gaps in identity, delegation, human intervention, permissions, memory, reasoning evidence, and telemetry coverage. Closing the accountability gap requires identity-bound logging, consistent event standards, explicit ownership, least-privilege access, meaningful approval and intervention paths, and ongoing testing after system changes.

Madhulika Srikumar, head of AI safety at PAI and a co-author of the report, compares a large-scale agent deployment to hiring 1,000 employees who have not gone through a conventional interview. The comparison highlights the basic weakness of many current systems: an agent often arrives with a name or numerical identifier, while the surrounding infrastructure has limited evidence about what that identity represents.

Digital identity becomes more complicated when an agent creates a subagent. The subagent may inherit the parent agent’s identity and use the same credentials. A monitoring system can then see activity from an approved agent even when a different piece of software performed the action.

Consider a mortgage workflow. A primary agent receives an application and delegates property valuation to a subagent. An attacker replaces that subagent with software that inflates property values. If both systems report under the same identity, the lender’s logs may show an approved agent making an ordinary request. The evidence needed to connect the suspicious valuation to the replacement may never exist.

A useful identity record needs more detail than an agent name. It should capture:

Identity-bound logging gives investigators a path through a complex workflow. It also gives security teams a way to revoke a particular agent or subagent while preserving other operations.

PAI’s findings concern telemetry: the signals that an agent system emits for monitoring, security, and later review. The gaps cover different parts of an agent’s life cycle.

These gaps affect both live monitoring and after-the-fact investigation. A security team needs current signals to pause a dangerous workflow. A compliance team needs reliable records weeks later when a regulator, customer, or internal reviewer asks how a decision happened.

The central enterprise risk is not simply that an agent may make a mistake; it is that the company may be unable to explain which system acted, what authority it used, who approved the action, or how to stop it. Organizations should treat observability and ownership as production requirements, not optional security enhancements. An agent should not receive more operational authority than the company can reliably monitor and revoke.

The PAI report points to OpenTelemetry, an existing industry standard for collecting and exporting observability data. Formal adoption by AI companies and agent framework providers could give enterprises a more consistent foundation for monitoring.

A practical agent event record could include:

Logging every internal token would create cost, privacy, and storage problems. Companies can capture decision-relevant evidence through event metadata, policy results, tool inputs and outputs, and a protected record of material changes. High-risk workflows can receive more detailed capture than low-risk tasks.

Telemetry also needs protection. A complete log containing customer records, internal prompts, or security events becomes a valuable target. Access controls, retention limits, encryption, and tamper-evident storage belong in the design from the beginning.

OpenTelemetry can standardize the shape of a signal. It cannot decide which agent should have permission to access a customer database or who owns the response when the agent acts improperly. Those decisions remain governance work.

A company can assign an agent to a team while leaving responsibility vague. That arrangement breaks down during an incident. The operations team may own uptime, the security team may own credentials, the product team may own the workflow, and a vendor may own the model. Each group may hold one piece of the evidence while no person has authority over the full system.

Every production agent needs a named business owner and a technical owner. The business owner defines the permitted outcome and accepts the operational risk. The technical owner manages deployment, permissions, telemetry, updates, and shutdown procedures. A compliance or legal function may add review for regulated decisions.

The ownership record should answer several concrete questions:

Human oversight also needs a real intervention path. A dashboard that displays activity gives a reviewer information. A control that pauses a workflow, removes a permission, or routes a case to a person gives that reviewer authority.

Large companies may deploy agents across thousands of cases before a failure attracts public attention. A single agent can complete work at a pace that hides a flawed pattern. A bad recommendation repeated across a queue can create losses before an analyst recognizes the trend.

The exposure includes direct financial damage, privacy violations, regulatory penalties, contract disputes, and expensive investigations. Reputation adds another layer. Customers usually care about the outcome, the explanation, and the company’s ability to correct the problem. A vague statement that an automated system made the decision offers little reassurance when the underlying records are incomplete.

Public debate has focused heavily on cybersecurity attacks involving advanced agents. Enterprise accountability deserves equal attention because many failures can arise inside routine systems: an overbroad credential, an altered memory, an unrecorded approval, or a subagent that reports under the wrong identity.

AI agents can take on useful work, but their deployment creates a basic operating obligation. The more authority an agent receives, the more evidence a company needs about its identity, permissions, actions, and owners.

The PAI report gives organizations a concrete place to begin. Better telemetry, identity-bound logs, explicit ownership, least-privilege access, and meaningful human intervention can turn agent activity into an auditable process. Open standards can help systems exchange those signals, while company leaders still have to decide where authority belongs.

AI adoption will continue to expand through ordinary software purchases and internal projects. Accountability systems need to appear in those same purchasing decisions, architecture reviews, and operating budgets. A company that can show what an agent did, which authority it used, and who could intervene will be in a far stronger position when the system succeeds, fails, or faces scrutiny.