AI Development Requires Ethical Governance Beyond Corporate Control

As artificial intelligence labs race toward increasingly powerful models, the conversation about who gets to set the rules has become just as critical as the technology itself.

Father Paolo Benanti, a priest who advises the Catholic Church on artificial intelligence, thinks the current debate around AI safety misses something fundamental. While researchers from major labs warn about superintelligence that could threaten humanity, Benanti sees a different problem taking shape. The companies building AI systems have positioned themselves as the only ones qualified to determine how those systems should be regulated. That concentration of power, he argues, looks a lot like a cartel.

Recent warnings from inside OpenAI, Anthropic, and other frontier labs have focused heavily on the risks of superintelligence-AI systems whose capabilities would far exceed human understanding. These warnings carry weight because they come from people directly involved in building the technology. Anthropic CEO Dario Amodei has called for mandatory third-party evaluations of new models before release. Others have suggested banning recursive self-improvement, where AI helps develop its next iteration.

Benanti doesn’t dismiss these concerns entirely, but he questions whether the framing serves the public interest. When companies cast AI development as so technically complex that only their engineers and executives can grasp it, they effectively shut everyone else out of decisions that will affect billions of people. The reverence with which some industry leaders discuss superintelligence-OpenAI CEO Sam Altman once spoke of building “a magic intelligence in the sky”-borders on religious language. That framing matters because it suggests these systems operate beyond normal human comprehension or control.

“Are we talking about some-thing, or some-one?” Benanti asks. “We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation going on.”

The distinction matters for regulation. If AI remains a tool-however sophisticated-then society can set standards for how it’s built and deployed. If it becomes something transcendent and incomprehensible, then only the high priests of machine learning get a say in its development.

The article argues that AI governance cannot be left to the companies building increasingly powerful systems. Father Paolo Benanti’s critique of industry self-regulation reframes the central issue from speculative superintelligence to democratic control, human dignity, and accountability. It advocates standards that preserve human oversight, a hybrid federal-state regulatory model, pluralistic ethics suited to different contexts, and moral perspectives from civil society and religious traditions.

Benanti advocates for democratic regulation rather than industry self-governance. He draws a clear line: ethics cannot belong to corporate boards of trustees. The standards that govern AI development need to emerge from public debate, not from the same companies racing to capture the market.

That’s where the cartel comparison comes in. When a small group of companies dictates the rules under which an entire industry operates, they’re not just building technology-they’re shaping the market to maintain their position. Both OpenAI and Anthropic have supported calls to slow AI development, but they’re also preparing for public offerings worth billions. The economic incentives don’t disappear just because executives express concern about safety.

The geopolitical dimension complicates things further. The most advanced AI models are being developed in the United States and China, two countries locked in strategic competition. That rivalry creates pressure to move fast, but it also makes it harder to trust any single nation’s approach to governance. International cooperation sounds good in theory, but it requires rivals to set aside the desire for dominance-something Pope Leo XIV explicitly called for in his recent encyclical on AI.

Benanti’s approach differs from many regulatory proposals. He’s less interested in restricting specific development methods-like distillation, where companies train models on outputs from proprietary systems-and more focused on establishing standards that ensure humans maintain control. Restricting methods might slow innovation or simply push development underground. Standards that prevent AI from slipping beyond human oversight address the core problem without dictating how companies achieve it.

This distinction aligns with emerging consensus in AI governance circles. Research shows that transparency and security rank as the most pressing ethical concerns, followed by accountability. Organizations need frameworks that emphasize these principles throughout the AI lifecycle, not just at the end when a model is ready for release. That means algorithmic audits, ethical review processes, and human-in-the-loop systems that keep people involved in critical decisions.

The challenge is implementation. State-level AI regulation in the United States has produced a fragmented patchwork of laws. Over 800 bills have been introduced since 2019, but few have become law. Colorado, Utah, and California have enacted different approaches-from comprehensive risk-based frameworks to targeted transparency mandates. A 2025 proposal for a federal moratorium on state AI regulations was quickly repealed after opposition from states and civil society groups who argued that states serve as first responders to AI risks.

The federalism debate reveals a deeper tension. Federal standards could provide consistency and prevent companies from facing contradictory requirements across different jurisdictions. But state governments have already passed laws protecting citizens in areas like biometric privacy and voice cloning-protections that wouldn’t exist if states had waited for federal action. The emerging hybrid model gives the federal government authority over training, safety, and data privacy while leaving states to regulate context-specific applications in healthcare, education, and criminal justice.

AI companies should contribute technical expertise to governance, but they should not control the rules that determine their own power, liability, and market position. The most durable approach is neither a blanket freeze nor unchecked acceleration: it is enforceable, adaptive governance that requires transparency, independent evaluation, meaningful human intervention, and public participation. Treating AI as powerful infrastructure rather than an emerging divinity makes democratic oversight possible.

Pope Leo XIV made AI the subject of his first encyclical, a significant move that signals how seriously the Catholic Church takes these questions. The letter, titled “Magnifica Humanitas,” warns against technocratic logic that treats efficiency as the ultimate measure of value and reduces human beings to projects for optimization. It calls for technology built for the common good rather than geopolitical or commercial dominance.

The encyclical draws on the Church’s social doctrine, emphasizing human dignity, inalienable rights, solidarity, and social justice. It condemns transhumanist ideologies and asserts that AI cannot replace human wisdom, conscience, or relational capacity. The document calls for stringent ethical constraints on AI in warfare and advocates for diplomatic solutions over a culture of power.

Benanti, who advises the Church but doesn’t formally represent the Holy See, sees the encyclical as setting a moral compass for AI development. The rapid acceleration of AI raises questions about civil rights, surveillance, labor impacts, and the extractive nature of technology that can colonize human experience. The encyclical challenges the concentration of AI power and stresses the need for inclusive governance with diverse voices at the table.

That aspirational vision faces practical obstacles. Disarmament and ethical implementation require collective political will and societal engagement beyond technology design. The encyclical serves as a north star, but translating those principles into policy and corporate practice demands sustained effort from governments, civil society, and companies willing to prioritize human dignity over market dominance.

One often-overlooked aspect of AI governance involves moral pluralism. Some researchers argue that building ethical AI requires recognizing competing virtues and allowing diverse moral frameworks to coexist rather than enforcing a single universal morality. AI systems need to develop practical wisdom through experience and feedback, allowing them to navigate moral complexity. Specialized professional AI should adhere to bounded ethical codes relevant to their domain-medical AI following healthcare ethics, legal AI following principles of justice, and so on.

A diverse ecosystem of AI systems with multiple moral temperaments might enhance safety and resilience. Imposing a universal moral standard risks brittleness and moral uniformity that could lead to systemic failures. When precautionary ethics become universal requirements, they can freeze the experimentation through which judgment matures.

This perspective pushes back against the idea that a single board of trustees or regulatory body can determine the right ethical framework for all AI applications. Context matters. The ethical considerations for AI used in medical diagnosis differ from those for AI generating creative content or managing supply chains.

Benanti’s emphasis on democratic debate aligns with this pluralistic view. Public discourse allows different moral perspectives to surface and compete. It prevents any single framework-whether techno-utopian, precautionary, or market-driven-from dominating.

Benanti keeps returning to a fundamental requirement: humans need brakes and a steering wheel. Whatever form AI governance takes, it must maintain meaningful human control over machine systems. That principle should guide both technical design and regulatory frameworks.

What does meaningful control look like in practice? It means transparency about how models make decisions. It means accountability mechanisms when systems cause harm. It means security measures that prevent misuse. It means humans can override AI recommendations in high-stakes situations. It means regular audits and assessments by parties who don’t have a financial stake in the outcome.

Some proposals miss this mark. A ban on recursive self-improvement or mandatory third-party evaluations might address specific risks, but they don’t guarantee human control. Companies could comply with those requirements and still build systems that operate beyond effective oversight.

The statement calling for a prohibition on superintelligence development until there’s broad scientific consensus and public buy-in reflects this precautionary approach. It acknowledges that safety and control need to be assured before development proceeds. That stance contrasts sharply with the race mentality that currently dominates AI development, where competitive pressure drives companies to release more capable models as quickly as possible.

Benanti suggests that AI’s arrival creates a new role for religions in public debate. By raising questions about what separates humans and machines, and how to protect future generations from technology’s negative effects, AI invites moral and philosophical perspectives that secular policy discussions sometimes lack.

Religious institutions bring centuries of wrestling with questions about human dignity, the common good, and the limits of power. They offer frameworks for thinking about technology that don’t reduce everything to efficiency or market value. The Catholic Church’s social teaching, for example, has long emphasized that economic systems should serve human flourishing rather than the other way around. That perspective challenges the logic that treats AI development as an inevitable force that society must adapt to rather than something society can choose to shape.

Other faith traditions offer similar resources. Jewish ethics includes deep discussions of human responsibility and the limits of creation. Islamic scholarship addresses questions of justice and the proper use of knowledge. Buddhist philosophy examines the nature of consciousness and the dangers of attachment. These traditions won’t provide technical specifications for AI safety, but they can inform the values that guide governance frameworks.

The current moment demands clarity about who holds power over AI development. Benanti’s warning about cartel behavior highlights a real risk: that a handful of companies will set the terms for technology that affects everyone. Their self-interested narratives about existential risk and superintelligence shouldn’t distract from the immediate need for democratic governance.

Effective AI governance requires several elements working together. Federal and international standards can provide consistency while leaving room for local adaptation. Transparency and accountability mechanisms give the public insight into how systems work and recourse when they fail. Pluralistic ethics allow different moral frameworks to inform AI design for different contexts. And meaningful human control ensures that no matter how capable these systems become, people retain the ability to intervene.

The technical complexity of AI doesn’t exempt it from democratic oversight. Nuclear energy is technically complex, but societies still regulate it. Pharmaceutical development requires specialized knowledge, but public health agencies still set standards for safety and efficacy. Financial engineering can be arcane, but governments still impose rules to prevent systemic risk. AI deserves the same treatment.

Benanti’s perspective cuts through the mystification that surrounds AI development. These are statistical systems built by humans, operating according to mathematical principles we understand. They’re powerful tools that deserve serious governance. That governance shouldn’t come from the companies building them, no matter how earnestly those companies express concern about safety. It needs to emerge from democratic processes that balance innovation with human dignity, economic opportunity with social justice, and technical capability with meaningful control.

The conversation about AI regulation is just beginning. Getting it right requires voices from beyond the frontier labs-ethicists and philosophers, policymakers and civil society advocates, and yes, religious leaders who can articulate moral principles that transcend market logic. The technology is too important to leave in the hands of those with the most to gain from shaping its development.