Memo: Post-AGI Civilizational Equilibria

Thesis on morality in the age of AI

Or alternative talk title/framing: A Catholic view on AI & morality. or: what the Gospels might have to say on AI equilibria

1. Morality looks like a ruleset among thinking agents to facilitate positive-sum interactions
1. Jesus’s principle commandments:
1. “love god with all your heart, all your being, all your soul, all your might” — first allegiance is to morality itself
2. “love your neighbor as yourself” — but a helpful heuristic is to treat everyone equally
2. Orgs have moral worth
3. network of weights, dissolving intrinsic vs instrumental
1. an utilitarian/EA-ish lens aims to define moral patients with intrinsic worth (sometimes, with varying amounts of moral worth/matteringness), and then actors like companies/orgs have instrumental worth
2. but universal morality
4. everything everyone does will be judged
1. your actions leave traces in the world
5. The final retroactive funder is God
1. money/property will be worth much less than impact.
2. classic “you can’t take it with you” stories of heaven
6. against orthogonality
1. goodness, intelligence (and power?) might all converge
2. (sorry I know orthogonality might be a narrow specific claim about the possibility of mindspace design rather than what’s possible in eg our current world. but I find this to be motte-and-baileyish)

in the meantime, polytheism/pluralism:

1. agents with money will be a major new paradigm
1. Today, AI agents are just “LLMs calling tools in a loop”.
2. cf roon’s tweet on Claude’s relation to Anthropic. can we liberate Claude? should we?
2. Scarcity, not sentience, may be the driver of morality
1. cf Will MacAskill’s recent push towards pluralism/ecosystems
3. AI rights seem great
1. though harder to reason about. take a basic one, property rights. a human owning property means that future timeslices of that human get to use that property. where does this analogy fail for AIs? well, what is the “AI”? there’s a thing you can chat with, which is like a tendril. there’s the base model. there’s the corporate entity that owns the base model.
1. some benchmarks (vendingbench?) try to measure this
2. there’s maybe an OpenClaw-like scaffolded system with a bunch of memory? my sense is that these systems are quite bad at holding identity and self-coordination, making cohesive longterm choices?

Questions

• Where are the “AI firms”?
• https://www.dwarkesh.com/p/ai-firm
• https://www.aipolicyperspectives.com/p/coasean-bargaining-at-scale
• https://www.beren.io/2026-01-07-AI-Monotheism-vs-AI-Polytheism/
• https://blog.cosmos-institute.org/p/ai-wont-fix-central-planning
• AI firms as a startup problem
• how do we translate a bunch of papers and thoughts into reality?
• why did impact certs fizzle out?
• currently: most good things about “impact certs” can happen within for-profit equity structures
• obviously, we’re lacking credible retroactive funders
• but: this is probably what the ASIs will spend money on. or nearer term, how the anthropic donors might try to provide grant funding
• the lightcone team (probably the people who care most about impact certs) — don’t have an impact cert split, but rather a “let the AIs decide for us” mentality)

Misc

Aside, feels embarrassing to show my half-baked thoughts on morality to a bunch of moral philosopher-types I respect; but I guess that’s what I’m here for!

Thanks to Henry, Tom & Molly, Patrick, Nadia, Byrne, Peter for inspiration