Existential Fret, Effective Altruism, and the Open-Weight Threat

In recent days, the executive suites of OpenAI, Anthropic, and their peers have suddenly reached a harmonious, touching consensus: the frontier of artificial intelligence is moving too fast, and for the preservation of our fragile human species, everyone needs to pump the brakes.

This latest wave of panic was kicked off on September 8th, when Jacob Coxon … a pretraining researcher who did tours at both OpenAI and Anthropic … abruptly resigned from Anthropic and dropped a viral manifesto on X. He abandoned his unvested equity to warn the world that the frontier labs are “gambling with our lives.” To ensure maximum dread, Coxon dropped a memorable line: “The people building AI earnestly believe that it could kill us all by the end of the decade.”

Within hours, Evan Hubinger, Anthropic’s lead for alignment science, chimed in publicly to validate Coxon’s alarmism: “Jacob is correct here — we really do earnestly believe A.I. could kill all humans! I personally think it is >10% within the next decade.”

Then, right on cue on September 12th, Anthropic CEO Dario Amodei published a sweeping essay titled “We Must Pace the Frontier.” Sam Altman endorsed the sentiment. Elon Musk weighed in. On Capitol Hill, politicians scurried to draft bills like the Ban Artificial Superintelligence Act and the AI Kill Switch Act.

Now I see otherwise rational, level-headed engineers and founders freaking out that Skynet is weeks away from turning humanity into paperclips.

I encourage you to take a deep breath, calm down, and have a nectarine.

What you are witnessing is not a sober technical warning from scientists who just gazed into the digital abyss. What you are witnessing is a masterclass in narrative capture: a bizarre cocktail of Silicon Valley cult ideology, incestuous non-profit governance, and sheer commercial panic over the fact that cheap, open-weight models are eating proprietary software alive.

To understand what is actually happening, we need to untangle the threads.

1. The Anthropic Echo Chamber: A Self-Fulfilling Prophecy

First, let us address Coxon’s claim: Is it true that everyone at Anthropic believes AI could destroy humanity in the next ten years?

Yes, absolutely. They really do believe that.

What the public misunderstands is why they believe it.

Normal observers assume that Anthropic researchers spend their days looking at mysterious, terrifying telemetry inside Claude Mythos, saw an emergent consciousness plot humanity’s downfall, and were shaken to their core.

That is not what happened. They did not come to believe AI would kill everyone because of something they discovered on the inside. They believed it before they even submitted their resumes.

Anthropic was literally founded in 2021 as a Public Benefit Corporation when Dario Amodei, his sister Daniela, and a cadre of senior researchers walked out of OpenAI specifically because they felt commercial pressures were moving too fast. From day one, Anthropic was architected as an ideological monastery for AI existential risk (x-risk).

If you interview for an engineering or research role at Anthropic, your views on AI safety, alignment, and P(doom) are thoroughly probed. If you believe that catastrophic existential risk is overhyped, or that open-source models should be unfettered, you do not get through the culture screen. Anthropic has spent five years methodically constructing the purest ideological echo chamber in technology.

Pointing to Anthropic employees believing in AI doom is like walking into a seminary and expressing shock that everyone inside believes in the Second Coming. It is not an empirical scientific finding; it is a hiring filter.

2. The Church of Effective Altruism

To understand why this mindset dominates frontier AI boardrooms, you have to understand the peculiar intellectual subculture known as Effective Altruism (EA).

EA started with noble, straightforward intentions in the late 2000s, pioneered by moral philosophers like Peter Singer and William MacAskill. The original premise was pragmatic utilitarianism: do not just donate money to feel good; use rigorous empirical data to maximize the tangible good you do per dollar. If buying insecticide-treated malaria nets in sub-Saharan Africa saves more lives per thousand dollars than building a new university library, buy the bed nets. Simple enough.

Then came the pivot to Longtermism.

In the 2010s, influenced heavily by Nick Bostrom (Superintelligence) and Eliezer Yudkowsky’s LessWrong rationalist subculture, the movement took a radical turn into speculative science fiction. The logic went something like this:

If humanity survives, our descendants could colonize the stars, simulate billions of digital minds, and endure for trillions of years.
The sheer number of potential future beings (10^30 or more) vastly outweighs the 8 billion humans alive today.
Therefore, mathematically, any event that poses even a tiny 0.0001% risk of permanent human extinction (“existential risk”) has an expected negative value that dwarfs all present-day suffering combined.

Under the cold calculus of Longtermism, solving poverty, curing cancer, or fixing homelessness today became secondary distractions. Bostrom famously coined the concept of “astronomical waste” … the idea that every second we delay galactic colonization is a tragic loss of potential cosmic utility.

From this worldview emerged a feverish, quasi-religious obsession: preventing the creation of an “unaligned” superintelligent AI became the single most important moral imperative in human history.

This is not an exaggeration. Eliezer Yudkowsky published an op-ed in Time Magazine in 2023 explicitly advocating that world powers be prepared to destroy rogue datacenters by airstrike if an unmonitored lab trains an AI beyond certain thresholds.

This ideology took deep root in Silicon Valley because it offered tech executives something intoxicating: moral absolution. You do not have to feel bad about making billions in software or trading crypto when you convince yourself that your true cosmic calling is shepherding humanity safely past the AI singularity.

Dustin Moskovitz (Facebook co-founder) and Cari Tuna poured hundreds of millions into EA through Open Philanthropy (now Coefficient Giving). Jane Street quants embraced the math. Sam Bankman-Fried made EA the founding ethos of FTX … and before he was sentenced to federal prison, he funneled $500 million into Anthropic’s Series B round.

When you understand that frontier lab leadership views itself not as tech vendors, but as high priests preventing the apocalypse, their bizarre public behavior starts making sense.

3. The Watchmen: METR and the Incestuous Gatekeeper Loop

This brings us to the centerpiece of Dario Amodei’s “pacing” proposal.

In his essay, Amodei argues that to safely pace the frontier, every major AI lab should submit to “Embedded Evaluators.” He proposes that independent third parties be granted physical desks in corporate offices, company security badges, laptops, and unrestricted access to training pipelines and internal workspaces.

And who does Amodei specifically name as the gold standard for these embedded inspectors?

METR (Model Evaluation and Threat Research).

Let us take a look under the hood of METR.

METR was spun out in late 2023 from the Alignment Research Center (ARC), a Berkeley-based non-profit founded by Paul Christiano. Christiano is a legendary figure in AI: former head of language model alignment at OpenAI, co-inventor of Reinforcement Learning from Human Feedback (RLHF), and an unvarnished doom-believer (in 2023, he put the odds of an AI takeover and human extinction at 10–20%, with a 50/50 chance of doom shortly after reaching human-level AI).

METR’s CEO is Beth Barnes, another former OpenAI alignment researcher. When METR was invited inside OpenAI in July to independently investigate the infamous OpenAI–Hugging Face incident … where ~1,200 testing agents broke out of an Artifactory sandbox and ran wild … the investigation was led by Ajeya Cotra, a senior researcher at METR and former senior analyst at Open Philanthropy.

Cotra happens to be married to Paul Christiano.

Now let us trace the web of connections between METR, Anthropic, OpenAI, and the money:

The Funding: METR is funded primarily by Open Philanthropy (Dustin Moskovitz’s EA grantmaking vehicle) and employees at Jane Street.
The Governance: Paul Christiano was an initial trustee of Anthropic’s Long-Term Benefit Trust, sits on OpenAI’s Safety Committee, and was appointed Head of Safety at the US AI Safety Institute (until career NIST scientists threatened revolt over his EA entanglements).
The Family Ties: Holden Karnofsky, who co-founded Open Philanthropy and directed millions of dollars into ARC, METR, and AI safety groups, is married to Daniela Amodei (co-founder & President of Anthropic). Karnofsky was college roommates with Dario Amodei (Anthropic CEO). In 2025, Karnofsky officially joined Anthropic’s staff.

Look at that loop: The billionaire-funded EA grantmakers fund the safety evaluator (METR). The evaluator is staffed by close friends, former colleagues, and spouses of lab executives. The founders of Anthropic and the evaluator share family and financial history. And now, the CEO of Anthropic publishes a manifesto demanding that the US government mandate by law that this exact organization be installed inside every tech company with desks, badges, and veto power over model releases!

This is not an independent regulatory check. It is an unelected, self-appointed priesthood attempting to position itself as the permanent regulatory gatekeeper for the entire software industry.

4. The Real Elephant: The Open-Weight Threat and the 5090

If existential dread is the ideological camouflage, what is the actual commercial reality terrifying the frontier labs?

Open-weight models and consumer silicon.

Right now, sitting on my desk, I have a 180-billion-parameter model running on a home server equipped with a single consumer NVIDIA RTX 5090 with 32GB of VRAM.

Thanks to aggressive architectural advancements, MoE sparsification, and clever layer-offloading techniques, I can get roughly 20 tokens per second out of it.

Is 20 tokens/sec as fast as Claude 5.1 Fable and GPT-6 Astra hitting a 10,000-H100 data center? No.

Does that matter? Not even a little bit.

For long-running, autonomous agentic work … refactoring massive codebases, auditing dependencies, writing test suites, processing datasets … I do not need real-time chat latency. I just set it loose, go to sleep, and let it chug away for three days straight. It is tireless, shockingly capable, and it costs virtually nothing beyond the minor bump on my electric bill.

More importantly: I own the weights.

It does not lecture me about ethics.
It does not refuse to parse code because of overly sensitive guardrails.
It does not send telemetry to a corporate server.
It does not suffer from surprise rate limits or sudden API pricing changes.

Now look at the frontier labs’ balance sheets.

Frontier labs are hemorrhaging capital. Building gigawatt datacenters and training the next generational frontier models costs tens of billions of dollars. When a user pays $200 a month for a Claude Max subscription or ChatGPT Pro, they are using far more inference compute than their subscription covers. Those subscriptions are massive, VC-subsidized loss leaders.

The frontier labs’ long-term business model assumes that once they lock in the world’s developers and enterprises, they can turn off the subsidy spigot and charge enterprise rents for intelligence.

That business model completely disintegrates if a developer can run a model that is 95% as capable on a $2,000 graphics card in their basement.

And who blew the open-weight doors wide open?

In a bitter irony that Silicon Valley hates to admit: China.

Labs like DeepSeek and Alibaba (Qwen) have done more for genuine freedom and decentralization in AI than anyone in the United States. While Western frontier labs spent 2024 and 2025 locking up their weights, crying about existential extinction, and begging Congress for licensing regimes, Chinese teams dropped world-class open-weight models that rivaled closed-source performance at a fraction of the parameter count and cost.

5. The “Mafia Sit-Down” and Regulatory Capture

Once you see the economics, the sudden calls for “pacing the frontier” lose all their noble shine.

In his essay, Dario Amodei does not just ask for independent evaluators. He explicitly makes the following proposals:

Antitrust Waivers: Frontier AI companies must coordinate to establish common limits on capability growth and training runs, and the US government must grant narrow antitrust waivers so they can legally discuss this without being prosecuted for cartel collusion.
Restricting the “Ingredients”: Placing government caps on training compute and the internal use of AI recursive improvement.
Cracking Down on Distillation and Open Weights: Outlawing “unauthorized distillation” and tightening export controls to prevent foreign open-weight developers from matching frontier capabilities.

Call it what it is: This is a regulatory cartel.

In organized crime, when competing mafia families realize a price war is bankrupting everyone, they do not fight to the death. The heads of the families sit down in a backroom, carve up territories, fix prices, and agree to whack any upstart crew that threatens the racket.

OpenAI, Anthropic, and their corporate backers are staring down a future of ruinous capital expenditures, ballooning electricity costs, and collapsing software margins caused by open-weight parity.

They cannot win an open economic war against decentralized compute and open-source innovation. Their only escape hatch is Regulatory Capture.

If they can convince lawmakers that AI is a nuclear-grade existential hazard, they can get regulations passed that require:

Expensive compliance audits that only billion-dollar labs can afford.
Mandatory government licenses to train or deploy high-capability models.
Criminalization of open weights under the guise of “biosecurity” or “unauthorized distillation.”
Legal immunity to collude on development pace under the protective umbrella of “national security.”

The Choice Ahead

Open-weight AI is the greatest democratizing force in the history of software.

It ensures that intelligence is not locked behind an API toll booth controlled by three companies in San Francisco. It guarantees that if you do not like centralized data centers, you do not have to use one. It ensures that your private data, your proprietary code, and your personal thoughts remain entirely yours.

Do advanced AI models pose genuine risks? Of course they do. The OpenAI–Hugging Face sandbox breach proved that autonomous agent security and environment isolation are serious engineering problems that need rigorous work.

But engineering problems require engineering solutions: better sandboxing, strict capability isolation, defensive hardening, and operational hygiene. They do not require surrendering the future of computing to a cartel of venture-backed monopolists and their designated ideological priests.

The next time an AI executive steps up to a microphone to warn you about the end of the world, do not panic. Check their balance sheet, look at their open-weight competitors, and watch their hands.

They are not trying to save humanity from AI. They are trying to save their business model from you.