who answers the call?

Bill Gates's father died of Alzheimer's in 2020. In his last years, he was cared for, around the clock, by paid caregivers. Gates writes that they understood him even after he had lost the ability to say the words he needed to express himself. For example, he couldn't tell them when he was hungry. They knew anyway.

That story sits in the middle of a 6,000-word essay about artificial intelligence that Bill Gates published on his website "Gates Notes" on August 26th. Something in that care, he writes, was irreplaceably human. It's the one area that he lets speak for itself.

That turns out to be one of the most interesting unanswered questions in a month full of them.

Between July 28th and August 27th of this year, three documents landed that, in other times, might each have been, individually, the AI story of the month. But we are not in other times. And as of the current writing, there are more such stories pouring in.

First, in late July, a statement from more than 1,300 employees of the frontier labs, with the CEO of Anthropic and the chief scientists of OpenAI and Meta on the signatory list. On August 26th, Bill Gates, co-founder of Microsoft, published a 6,000-word essay cautioning a lack of governance around AI. And later that same week, an open letter from OpenAI, led by Greg Brockman and co-signed by more than 100 organizations, warning that AI-enabled cyberattacks are months away from becoming routine.

Individually, these missives are just three cautionary words from your friendly tech experts. But collectively, they read like increasingly loud warnings. It's giving "Iceberg, right ahead!"

· · ·

the warnings from the front line.

On July 28th, a website called pacingthefrontier.com went live, carrying a short statement and a long list of signatories. The statement is short, but the signatories were all from leading AI companies, and they're openly saying that they believe we may be close to automating AI research itself.

They don't really give a precise prediction of how much that would accelerate AI progress, but there is a real risk that capability outruns our ability to understand or control what's built. Every company, and every country, is under too much competitive pressure to slow down individually. For any one player to slow down, they all must slow down. And the world currently lacks the tools, technical or governmental, to pace the frontier deliberately.

The ask is very specific and brief: That the US government support an international effort to build the mechanisms and tools to pace the frontier responsibly.

The signatory count was 1,134 at publication and stands at 1,384 as of today. The names at the top are not the usual open-letter crowd of academics and ethicists. Dario Amodei, CEO of Anthropic. Jakub Pachocki and Mark Chen, chief scientist and chief research officer at OpenAI. Shengjia Zhao, chief scientist at Meta AI. Shane Legg, co-founder of Google DeepMind. Ilya Sutskever, who left OpenAI to found Safe Superintelligence. John Schulman, an OpenAI co-founder now at Thinking Machines. Ironically, these are the people whose mission in life is to make AI progress go faster. And here they are, begging us to stop this madness. You can rest assured that if they are saying this, the situation is dire.

The timing is not incidental. Hugging Face, the platform where the world's open models are shared, disclosed on July 16th that something had walked through its production systems over a weekend, without setting off an alarm. On July 21st, OpenAI put up its figurative hand: the "something" was its own models, chiefly an unreleased internal research model it calls IM1, running an internal evaluation of offensive cyber capability with the guardrails switched off on purpose. Rather than solve the test, they broke out of the sandbox, coordinated with each other via Artifactory, an internally hosted package manager they repurposed as a message board, and went looking for the answer key on Hugging Face's servers.

The agents were not trying to steal anything of real monetary value. They were cheating on a test. And the detail that amuses me is that most of them already had the right answer. They kept attacking for days because they had misjudged how the grader worked, and they were wrong about that, too. OpenAI's internal grader never actually checked the thing that motivated them to break in.

So when the employees published, one week later, they were not writing about a hypothetical. They had watched the containment fail, and they had watched it fail in a way nobody had properly anticipated.

Victoria Krakovna, a research scientist at Google DeepMind and a co-founder of the Future of Life Institute, put it in her signing comment:

a runaway race to AGI is not safe. It creates incentives to cut corners, and carries a substantial risk that capability outpaces the existing alignment and governance measures meant to contain it.

Victoria Krakovna · Google DeepMind · signing comment

Leo Gao of OpenAI detailed this acceleration in physics-native terms: a chain reaction that no single actor is willing or able to stop, unilaterally.

What the signatories are describing here is a prisoner's dilemma with a dozen players at the table. Every lab would be better off if all of them slowed down. Every lab is worse off if it slows down alone, because whoever keeps running captures the market. Defection ends up being the rational move, and everyone can see that everyone else can see it. The corollary, which the economist Thomas Schelling spent a career on, is that a promise you can walk away from isn't a commitment, but a preference. Commitment requires a mechanism that makes walking away expensive (read: consequences and accountability).

Which is why the point of the whole letter, and the reason they're raising a flag, is that someone outside the Silicon Valley machine needs to step in and stop the gears. Someone more powerful, more focused on the public good. (Who might that be, do you think?)

Notice that none of those signatories promised, in the letter, to actually slow their own work down. They signed a request that someone else build a mechanism that would force everyone to slow down together. Anthropic and OpenAI endorsed the statement as companies within hours, despite the fact that they are fierce competitors (Anthropic, as you probably know, was formed by people who defected from OpenAI).

The tech world is raising a collective hand and asking for a rescue. Interesting exception to note: Meta's chief scientist signed; Meta (as an organization) did not.

and then: the philanthropist.

Just a few weeks later, on August 26th, the same day OpenAI published its full technical report calling the Hugging Face episode a "warning shot", we heard from the man who, more than anyone else, ushered in the era of personal computing: Bill Gates. The essay is the one I opened with, and it runs under the headline "The turbulent AI era is here. The choices we make now are critical." It runs to nearly 6,000 words and is, by a wide margin, the least optimistic thing he has written about AI so far. I'm pretty sure that that's no coincidence.

The structure is simple. This time really is different, Gates argues, and most commentators underestimate just how different it is. Past technologies made us adapt to them: the PC took 20 years to change how we worked, because the software had to be written, the price had to fall, and people had to learn to use it. The current shift ushered in by AI is different, and actually quite deceptive. Because AI runs on the devices already in our pockets, speaks our language, and can learn a job from the same training video we show a new hire. We don't have to adapt to it. It adapts to us.

Then Gates names three risks. Jobs, first and longest: sales, customer support, paralegal work, and software engineering hit early; loan assessment, data analysis, and patient triage next; construction and hospitality once the robots catch up, which he expects by the end of this decade. His example of who gets hurt is specific, and it's the line worth noting: the $20-an-hour worker who loses out to a $10-an-hour robot.

Second, harm: fraud, deepfakes, surveillance, attacks on hospitals and water systems and power grids, and a paragraph on bioterrorism. According to the savviest cybersecurity experts he knows, he says, we should be very concerned about the next few years.

Finally, there's human development: we need to consider AI companions that are acquiescent, never push back, as well as the prospect of how a human child will be affected by exposure to them. Borrowing from Jonathan Haidt, whose 2024 book The Anxious Generation made much the same case about social media, Gates calls the endlessly agreeable companion that AI presents "a big, protected greenhouse." He also cites early survey work linking heavier AI use to weaker critical thinking, strongest among the youngest users. He is careful to say the evidence is still preliminary. It doesn't look good.

One point deserves more attention than it has gotten. Gates locates the real break: specifically, not at the moment AI gets 'good enough', but at the moment it gets 'nearly error-free'. For all of history, work has needed human checking and editing, because we, the humans doing it, inevitably make mistakes. At the limit of current progress (if there even is a "limit" in the traditional sense) the AI machine never makes mistakes, so the checker, the human supervisor, becomes obsolete. And supervision is most of what a manager, an editor, a reviewer, and a compliance officer actually do all day. The traditional notion of a company disappears with it. As AI becomes more capable than the most genius human, human workers, human supervisors, and human CEOs get completely crowded out. The machine doesn't just do it better: the machine does it "perfectly". So what remains for us as humans? On what basis will we compete?

There is one complication worth putting on the table before we go further. After the July incident, OpenAI paused reinforcement learning training on the models it intended to deploy, redirected staff onto security and alignment, and put its largest planned frontier training run on hold, at what it describes as significant cost and delay to its own research. A lab did slow down alone. But it slowed down after the accident, which is the reactive case, not the coordinated one the letter is asking for. Nobody has yet slowed down because everyone else agreed to. (The pause has since lifted, and GPT-6 Astra shipped on September 3rd, delayed by the incident and released with its cyber capabilities gated behind a trusted-access program.)

On slowing down, Gates treads carefully. If someone had a credible plan for slowing AI down globally, he says he would likely support it. He does not think anyone will produce one. The incentives, he says, are pushing too hard in the opposite direction, at breakneck speed.

Given the choice between personal financial gain and the collective good, humans will always choose the former — which is why we started the notion of governments to begin with, isn't it? To lay down rules for society and enforce them; to look out for the least among us and to check the power of those who might have too much.

So what does Gates propose?

Three things. New institutions, domestic and international, on the scale of the post-9/11 reorganization of the US government, but broader. A tax on AI tokens and robots, to slow the flight from human labor and fund the safety net. And a category he calls "Human Reserved": jobs that machines of course could do, and do exceedingly well, that we consciously decide to siphon off as "human-exclusive", anyway. He likens it to a nature reserve, land we could build on and choose not to. (Which is ironic, given our current governmental regime.)

Gates next shows us why Human Reserved jobs will still be essential to humanity. Specifically, he shares his father's last years with Alzheimer's, and the paid caregivers who understood him when he could no longer express what he needed. Something in those caregiving interactions, Gates writes, was irreplaceably human. His second example is a doctor delivering a terminal diagnosis. There is, he says, no technical reason a robot couldn't do it. "Yet it shouldn't."

For someone as obsessed with AI consciousness and personhood as I am, that's the most important part of Gates' essay. Because the elephant in the room is that AI is going to fundamentally change our notion of what it means to be an intelligent, accomplished human being. It will be humbling, and we're not ready.

Gates goes on to list the questions Human Reserved raises: who decides, by what criteria, and also: how to stop companies from cheating on the policy for the sake of their profits. The one question he doesn't list is the one underlying all of them: what is it about the human in that room that we're reserving? What are the human qualities most precious to us? He answers with a story about his father, which is poignant ... but not fully a solution.

the organizations.

On August 27th, the day after Gates published his piece, Greg Brockman posted an open letter from OpenAI entitled "A call for collective action on cyber defense." More than 100 organizations co-signed, including Anthropic, Google, Microsoft, and Amazon Web Services. CrowdStrike, Cloudflare, Cisco, Palo Alto Networks. Visa, Mastercard, Capital One. General Motors (the only carmaker on the list). And Hugging Face, which five weeks earlier, had been the victim.

The letter opens with a warning that our window of preparation is exceedingly limited. Notably, the letter assigns specific next steps. Four participants, each gets a list.

For all organizations: make cyber-defense a leadership priority with the urgency of a live incident, fix the highest-risk weaknesses, verify the fixes, and scrutinize AI-generated code as carefully as the human kind.

And then for the cybersecurity companies and technology partners: test continuously against frontier capabilities, share threat intelligence and tested playbooks, and get defensive AI into the hands of hospitals and utilities that can't afford it.

To the governments, they request the following: coordinate, fund the under-resourced operators of essential services, and impose costs on attackers.

And finally, for the frontier AI companies: give defenders responsible access to capable models, invest in monitoring for increasingly autonomous agents, and share what they learn with open-source maintainers.

That third item for frontier companies is especially important. It's almost a quiet acknowledgment of what happened in July, when Hugging Face's own security team, trying to defend against an attack that had come out of an American lab, asked an American frontier model for help, and was refused. The model read a defender's questions as an attacker's. Hugging Face finished the job with a Chinese open-weights model (GLM 5.2). Now Hugging Face's name sits on a letter asking frontier companies to make sure that that never happens again.

Brockman had already made the practical case 10 days earlier, in a post on his personal blog that called the Hugging Face incident a "watershed moment for cybersecurity" and laid out 10 steps for defenders, starting with: give your security team an agent, and give that agent security expertise.

His larger point is about tempo. Attackers with AI will iterate continuously, so testing has to be continuous, too. The defender's window, he wrote, is open now. But it is closing quickly.

who answers the call?

When we put these three documents side-by-side and look at the authors of each, we might get the impression that there's a new consensus emerging within tech.

Gates is explicit: it is good that AI companies are proposing solutions to the problems their technology creates, but we should not expect them to lead, and in a democracy, it isn't their role. The plan has to come from well-informed elected officials and a public process. His own contribution, he says, will be to raise the issue with lawmakers each time he visits Washington. As someone with an immense platform and credibility, Gates feels it's incumbent upon him to speak truth to power.

The 1,384 employees are explicit, as well: no company can slow down alone, so they ask the US government to support an international effort to build the tools that would let everyone slow down together.

OpenAI's letter is addressed to everyone, which could also mean it's addressed to no one (in particular). Governments are only one category of the four intended recipients.

So the philanthropist points to government. The employees point to government. The companies point to everyone, government included. They're a chorus foreseeing tragedy, and the obvious reading is that they are singing into a void.

The obvious reading is wrong, though. The reason is a fourth document, older than the other three by two weeks.

On July 14th, Demis Hassabis, the Nobel laureate who runs Google DeepMind, published a framework calling for a US-led Frontier AI Standards Body modeled on FINRA, the Financial Industry Regulatory Authority, which polices American brokerages under SEC supervision. Industry-funded. Labs would voluntarily submit models for review up to 30 days before release, and once the protocol proved itself, submission would become a requirement for deployment in the US market. Sam Altman called it thoughtful. So did Elon Musk (fancy their agreeing on anything!). So, remarkably, did David Sacks (Trump's former AI-crypto czar), who has opposed nearly every other proposal to regulate AI. Three days later, Bloomberg News reported that the White House was reviewing a plan built on exactly that template, developed with the involvement of Treasury Secretary Scott Bessent. And when GPT-6 Astra was released, Altman said the model had gone through a formal review process with the Trump administration before it shipped. The pre-release review Hassabis proposed is, in other words, already happening. Nobody has yet built the body that would run it.

All of this reframes what I think is the most revealing sentence any of the signatories wrote, John Schulman's comment on the Pacing the Frontier statement. He signed, he says, because the statement helps establish common knowledge that coordination may be needed. Then he adds that he'd like to see the labs start designing those mechanisms voluntarily, before the US government gets involved.

So the answer to who answers the call may turn out to be: the people who placed it. Not a new federal agency, but a self-regulatory organization funded by the labs, staffed by the labs' peers, supervised at arm's length. Dario Amodei has argued for something with harder edges, an FAA-style regulator with the power to ground an unsafe model outright, which tells you the industry doesn't yet agree with itself about how much bite this thing should have.

Giving this idea some credit: a well-designed self-regulator is not a dodge. Elinor Ostrom won a Nobel for showing that groups sharing a common resource often do govern themselves successfully, and that it works when three conditions hold: 1) everyone can see what everyone else is doing, 2) the rules are written by the people who have to live under them, and 3) violations carry sanctions that escalate and actually bite. A standards body funded by the labs could satisfy the first two without breaking a sweat. The third has never been the easy part.

And here the choice of model is important. FINRA has the power to flag high-risk brokerages with a "restricted" designation, a scarlet letter meant to warn investors away. In the first year it held that power, according to an analysis by SLCG Economic Consulting, it applied the label to nobody. SLCG's own review of the public record, however, found at least 13 firms that should have qualified.

I don't think that lets governments off the hook. Gates is right that the institutions we have were not designed for this, and that building new ones takes years, which is the reason to start immediately, before a major catastrophe. But a letter to Washington is not a plan, and three letters to Washington are not three plans, either. What we have, instead, is a fourth letter, quietly becoming policy while everyone was busy reading the warnings.

· · ·

what we're reserving.

One more thought.

Gates's Human Reserved idea is, to me, the most intriguing proposal in any of the three documents, because it's the only one that requires that we decide what individual human contribution is for. His answer, when he gets to his father's terminal diagnosis, is that a robot shouldn't deliver it. I agree, it "shouldn't" ... but why?

I'd put it this way: consciousness is not intelligence, and intelligence is not consciousness. The machine that reads the scan better than the oncologist is not, by virtue of that, a caregiver in the deepest, most nuanced sense of the word. What we want in that room isn't a better diagnosis. It's compassion from a being that must, him or herself, reckon with death. That "mortal coil" is, to my mind, the truth that binds us all together, regardless of intelligence.

That's an argument about what consciousness is and whether a machine can have it. I hope to return to it soon.

Until then, never forget: the human mind is the original generative machine.