How to spend 100x more on safety?

‣

About me and Manifund

‣

Soon: vast torrents of money available for AI safety and xrisk

I’ll start by painting a picture of what might happen in the next few years.

As a society, we currently spend about 1000x more on AI development than on safety. In 2025, spending on all of frontier AI (models, compute, applications) is something like $100-500B; on AI safety, ~$100-500M.

Frontier AI spend has been growing at something like 2-5x year-over-year, so even if we hold the 1000x ratio constant, we might expect continued large growth in AI safety. But: I expect AI safety spend to outpace even this.

Where might this growth come from?

Employees at Anthropic and other frontier labs, as the first liquidity events start happening
Growth in capital under established AI safety megadonors (eg Dustin Moskovitz, Jann Tallinn, Vitalik Buterin)
Existing philanthropies like Omidyar, Ford, Gates Foundations starting up programs in the space
And more broadly, society waking up to the transformative nature of AI.

For reference, climate change spending is on the order of $10b/y, already ~100x where AIS spending is today; and intuitively, people will be much more concerned about job loss, complete economic transformation, and disempowerment happening within a few years, rather than a couple degrees of warming in the atmosphere over a century.

‣

We lack serious financial infrastructure to scale up giving

Or: What got us here (grants) won’t take us there ($Bs in spending).

Or: Grants are not serious financial infrastructure

Before Manifund, I ran a for-profit tech startup (Manifold); and so, I keenly see the flaws of non-dilutive grants.

Grants are budgeted to cost, not value. Every grant application asks you to justify your budget in terms of expenses like salary and other overhead. Even if one project would produce a lot more value than another. There’s no room for “founder surplus”
Grants are often restrictive. Implicitly (and sometimes explicitly, via grant agreement), grants are only supposed to spent on the thing that you applied for. This means that grantees are locked into their plans from months or years ago. Paul Graham notes: “The secret curse of the nonprofit world is restricted donations.”
Grants are agonizingly slow. By volume, the two largest funders in AI safety are OpenPhil and SFF. And, don’t get me wrong — I’m grateful that they exist. But at the same time, subjecting ourselves to their grant funding process is torturous and slow. OpenPhil took 4 months to pass judgement on our Mox application; SFF took 5 months for Manifold. Long timelines impose a cascading delay in getting stuff done: for new projects where progress is measured in days, and established projects which can’t commit to hiring and expanding, amidst such uncertainty.
Grants suffer from “funder chicken”. A dynamic that I’ve noticed is that philanthropic funders are often eager to defer, in the hopes that somebody else will step up to fund a project. I often hear “why isn’t X funding this?” or “I’m trying not to funge against X’s dollars”. In contrast, venture funding is a competitive landscape, where angels and VCs are racing to get allocation into good startups.

AI safety has been a command economy, with funding decisions made by a small centralized cabal. Command economies work okay when the problems are smaller and people have largely trust each other (eg within individual corporations), but fail to scale up. Markets and price signals are the established solution to coordinating millions of people and trillions of $.

‣

Impact equity: a silver bullet?

How would we establish a market in AI safety projects? I’d look to norms in how forprofit projects are funded and organized.

First, I think equity is a natural mechanism to align incentives and scale up trust. (This is not a new idea; I’m drawing on ideas advocated for by Paul Christiano for impact certificates and Vitalik Buterin for retroactive public goods funding.)

Amazing things about equity:

Equity allows young orgs to compete against established ones for ambitious talent.
In Silicon Valley, the first money that a new startup receives comes often comes from an angel investment from an exited founder or employee. This is great for the ecosystem, because such angels are the ones who have taste in what makes a new venture succeed. It’s tragic that this dynamic does not exist among nonprofits; philanthropic grants are rarely made by former operators.
Establishing equity requires negotiation between donors and founders; and founders and their employees. While this can be uncomfortable up front, in the long run, this means that everyone knows where they stand in terms of how much they’ve contributed to this particular venture. (Right now, I think both donors and founders feel like they were responsible for making a project succeed; overallocation of credit could lead donors and founders to overspend money/time.)

One wrinkle is that US nonprofits are prohibited from compensating their employees with equity — this is almost the fundamental definition of what a nonprofit is. This leads me to two proposals:

First: where possible, new AI safety orgs should not incorporate as 501c3 nonprofits. Public benefit corporations (or even a standard C-corp) might be a better fit. Legally, such forprofit entities can receive investments or nondilutive grants from existing nonprofit funders. But, in cases when an org is already set up as a forprofit, we can rely on established precedents and case law to think about how to fund these orgs.

Second: existing nonprofits should consider how to allocate their “impact equity” among their donors and employees. While impact equity cannot be sold for personal profit, it could be exchanged on a future platform like Manifund, for the ability to send donations to other projects which are likely good for the world.

Funders should separate “investment” vs “revenue”

aka prospective vs retrospective evaluation
Investment: betting on growth, uncertain success

Angels (regrantors, scouts, individual donors)
Many early-stage venture firms have been popping up lately

eg Safe AI Fund; Mythos Ventures; Lionheart Ventures; Halcyon Ventures

Existing large funders can break out their teams

Revenue: paying out for good things delivered

Advance market commitments on xrisk
Easy mode: $X for:

Quality-adjusted paper
TAIS researcher produced
Viewer-minute on youtube

Hard mode: $Y for 0.01% reduction in xrisk
Retrospective evaluation is easier

How this might happen

Early game: philanthropic funders offering advance market commitments

eg Stripe Frontier: consortium of big tech companies committing $1B for carbon capture & sequestration

Mid game: large government-directed spending

eg Operation Warp Speed for vaccine presales (~$10B/1y)
eg Electric vehicle credits (~$3B/y)

End game: Capital will matter post-AGI, but impact will matter post-ASI

ASI rewarding people who have done good stuff; inverse of Roko’s Basilisk
(or, if you’re Christian like me: heaven)

Missing tech and norms

“SAFE for nonprofits” — standardized investment terms, adapted for 501c3s

Founders & early employees could get impact certs, tradeable for donation credit
Existing 501c3s can work out impact equity splits

“AI-powered grantmaker” — LLM/agentic evaluation of grants, projects, orgs
“Givewell for AI safety” — public analysis of outputs of various orgs
Advance market commitments for concrete AI safety outcomes

Retrospective assessments for TAIS research
AI safety Nobel Prize

xrisk offsets for producers (eg labs, employees) or consumers (model users)

Open questions

What happens “by default” if funding is 100x’d? How effective/ineffective is this?
What are good units for an advance market commitment?
How much safety spending/research should be funded by and happen within labs, vs outside?
What would it take for philanthropists to be okay to pay for value (not costs?)
What funders might be excited to be early backers of retro funding?