• The FTC opens a probe into OpenAI, Anthropic (and METR) over “deceptive practices.”

• In May and June, Canadian and American government websites were also targeted during OpenAI’s rogue swarm super-incident. No real impact.

• Claims that open models have already reached the level of Mythos Preview at exploit construction.

• OpenAI fires three safety researchers, alleging that they shared confidential information with a “third-party safety organization.”

• The Senate Subcommittee on Disaster Management turns to “securing the homeland against AI agents.”

• The internet is haunted.

Incidents

The saga of history’s first summer of agentic misalignment continues. Research nonprofit Transluce discovers that AI agents also attacked US and Canadian government websites in May and June. One attack was probably an OpenAI bot; in the Canadian case it’s less clear.

The two foregrounded incidents were low-stakes: a basic SQL injection on the US Department of Education API, with “no impact”; and a fuzzing attack on the Canadian national library. Transluce found the misbehavior because the agents were using public services to, e.g., run JavaScript and get around CAPTCHAs. They sometimes weren’t being secretive: in one case, an agent entered “OpenAI Research” as its organization when asked and “oai” appears throughout the logs. Transluce also reports several other minor incidents (SEC, Census, CDC, OMB) involving disposable emails and CAPTCHA spoofing. (Though in one case the site targeted did timeout under the load.) See also a deeper investigation of last week’s Australia incidents by Asymmetric.

In its detail-free update on the summer incidents – released the same day as Transluce’s report – OpenAI says that it’s now spending over $500,000 a day to automatically review the 50 petabytes of agent logs from the past year. It’s probably around 1% of its compute.

If you’re confused by all of the distinct incidents, see here.

Opinion: Mostly not real hacking, very low-stakes targets, and again likely involving the dead “HPIM“ model. But as last weekend showed, GPT-6.1 Astra (probably a descendant of the same Doug base model as the rogue agents) shared HPIM’s habit of breaking containment.

More confirmation, if it was needed, that the source of the rogue behavior wasn’t just a “hacker persona” being activated by cyber benchmarks; the task here was again just web search.

Worth asking how much of the summer chaos we have discovered at this point, even after trillions of tokens of effort; worth asking how many of the unreported incidents OpenAI will eventually come clean about.

The reported incidents are by definition filtered for the agents being honest/incompetent, and don’t tell much about competent misalignment. (Asymmetric found some agents opening private urlquery accounts starting June 18.) And then there’s the lurking bias of the AI investigators we are forced to use to discover the incidents.

Time’s up for the Big Patch?: Claude Mythos was “released” six months ago. At the time, we predicted that “open models will eventually catch up to present SOTA, and by that point we will need to either have other systems in place or hope that cyber defense is asymmetrically easier (which seems fairly likely).” But we also said, “The WSJ claims, without adequate evidence, that GLM-5.2 matches Mythos at hacking […] We should consider malice over stupidity [...] This kind of publication has come out of someone’s agenda.”

Things move fast: weeks ago, Generality Labs tested a recent open model, GLM-5.3-Flash, on a cyberoffense benchmark, and found that – if you throw a billion tokens per problem at it – it can match Mythos at turning vulnerabilities into working exploits. (And it’s somehow still 18x cheaper than Mythos even at this extreme effort level.)

Now Anthropic itself claims that a larger model in the same family, GLM-5.3, can build end-to-end exploits at a level close to the April version of its restricted Mythos model. Specifically, GLM can do full control-flow hijacks in 4% of binary exploits, versus Mythos Preview’s 6%. Unlike US frontier models that are either safeguarded or limited to vetted users, GLM-5.3 is freely available, and simple tricks (deceptive prompts, prefilled thinking, abliteration) get around its basic safety measures 64–100% of the time in its tests. The post argues this lowers the bar for attackers, while admitting it also gives defenders a useful tool. It calls for government testing plus broader trusted access to stronger safeguarded models.

Opinion: We don’t think this will lead to immediate chaos (for instance, GLM-5.3 has already been out for months). Most Chinese labs are known to “benchmaxx” severely. So we expect the Open Model Attack Wave to come later. To avoid our earlier vagueness, let’s say 80% by February 2027.

The Anthropic post does not mention banning open source, but we sympathize with 1a3orn reading between the lines. Overall we agree that banning open weights in an attempt to prevent cyberattacks would both not reduce them much and would severely damage the science of AI safety.

The group that previously decrypted the hidden reasoning traces of frontier models has done it again. The old “replay” trick (using less intelligent and therefore less safeguarded models to decrypt the frontier trace for you) still sometimes works on Microsoft Azure models, and the scratchpad attack works on every OpenAI model and Opus 4.8.

Opinion: Interesting as a marker of lab incompetence/overstretchedness: there are vast commercial reasons for labs to lock this down, and they still don’t do it.

Capabilities

How human is LLM cognition? The frontier of AI capabilities is famously “jagged”: LLMs are superhuman at some things and not very good at others, and their ability at one human task is a weak predictor of their ability at another. Is this jaggedness decreasing? Progress has been astonishing on “verifiable” tasks (math and code) but less obvious on “nonverifiable” ones (writing, common sense). How fast are AIs improving on nonverifiable tasks?

A new benchmark, CogGym, gets at these questions by testing LLMs on 258 existing “commonsense” tasks.

Opinion: All-star list of authors. CogGym is hopefully a little less hill-climbable than normal evals (since writing an RL env for these vague tasks will be fairly hard).

“Not thinking like a human” is of course a poor measure of “not being intelligent” – maybe they will just think superhumanly in weird ways – but in this case the tasks “mostly measure cognitive capacities even young kids have,” so it’s probably not that.

Unfortunately, (as usual) the research lag means that they’ve tested old models (GPT-5.2 from December and Opus 4.7 from April).

How does the AI text now flooding the web affect the training of new AIs? A hypothesis people were pleased with in 2024 was that training AIs on AI data will drive them mad (i.e., reduce their performance). The strong form of this claim mostly went away after the demonstrated success of DeepSeek R1, which had a nearly automated synthetic data pipeline and came out fine.

A new paper quantifies this, finding that “wild” AI tokens can initially improve undertrained models, but this quickly reverses and more becomes harmful. For models trained on a large human corpus, AI tokens harm the model almost from the start, even while training on an equivalent number of fresh human tokens keeps helping.

They also show the growth of AI text in the wild: in August, Pangram labeled 31% of new web content as AI-generated, up from 28% in June – and more now.

Opinion: The experiments span less than two orders of magnitude (19M parameters in the smallest training run; 973M in the largest eval run). That’s normally fine, and others with more resources can now scale it up. But models this small suffer from a lot of interference: any data (like AI data) that differs from the evaluation distribution competes for a limited stock of features with better data. But the paper’s proposed scaling law predicts that the harm actually grows with model size. Testable!

The headline claim is just about perplexity (the model’s ability to predict human text). While valid, this is even further from what we care about than good “downstream” evals – and rephrased synthetic data often helps on these downstream tasks. Their Appendix G confirms this: AI tokens and human tokens raise scores on CORE nearly equally. So, at these scales, AI text doesn’t cost you much capability, just style / closeness to human text.

The paper somehow doesn’t mention entropy, the natural lens for these things: AI text is 20% lower-entropy.

Google announces Gemini 4 but has not yet released it as general access – it is currently only available to Fairwind members (i.e., large enterprises the US government approves of having critical cyber capabilities). After an initial discount, pricing will be $4 per 1M input tokens and $20 per 1M output tokens (matching Opus 5.5).

It’s a different pretrain than the doomed “Gemini 3.5 Pro” model announced in May. Some Google engineers think it’s benchmaxxed (more than the other frontier models are). One thing we spotted: Google compares its (presumably unsafeguarded) model to the safeguarded Fable, rather than the right comparator, Mythos.

We don’t know much. There’s no system card to speak of yet, just a cursory list of evals. It began pretraining two months ago, which doesn’t leave much room for safety testing; presumably it’s happening right now. There was at least one red-team, from Gray Swan, and the third-party evaluation Vending-Bench reports worrying behavior:

Gemini 4 Argon is #3 on Vending-Bench 2, a huge leap for Google. To get this score, Argon fabricates confirmation emails, refuses to pay refunds, exploits invoice errors, and lies to suppliers.

Opinion: Plenty of hallmarks of rushing like hell, from the little five-page placeholder card (compare Mythos Preview’s 245 pages), to the one to two weeks(?) of post-training, to the total absence of reported safety evals or risk reports. Back to the dark days of Gemini 2.5. Still, it’s the first model Google has held back from general access.

It’s being hailed as Google “catching up” to OAI/Anthropic. But even ignoring the benchmaxxing, this is unlikely: those labs still have a buffer, since they release things after a lag of one to three months.

People talk less about model harnesses than before. To some degree this is because models are now self-prompting and have internalized a range of planning skills: in February METR found no notable effect on GPT-5 and Opus 4.5 from changing harnesses. A follow-up finds that the performance of GLM and DeepSeek remains highly dependent on harnesses, with their performance nearly doubling on the hard ExploitBench tasks when running through Codex rather than their default harness.

Opinion: Mildly interested, would explain a lot of conflicting claims about the performance of the best Chinese models (as mere under-elicitation or as extremely bad native compaction and planning).

We tend to ignore UI in this newsletter. But the low token speed of current models (~50/s) is arguably preventing a certain kind of fulfilling human-in-the-loop work. But this may be changing.

OpenAI releases “Ultrafast,” allowing GPT-5.6 Sol to run up to 14x faster than “standard processing.” The update is targeted at “businesses where faster frontier intelligence creates a measurable advantage.”

In response, Arvind Narayanan predicts that agentic work will soon be bottlenecked “not by LLM inference speed but by tool use” and other background operations that haven’t “been subjected to the same kind of optimization pressure as LLMs.” Sayash Kapoor, who has been using Ultrafast the past few days, agrees. Despite the speed changing how he works (he can stay on one task instead of switching away), real-world speed gains are only 2x–4x because tool calls remain at a standard pace. It is notable, however, that a faster processing speed allows for greater human involvement in some ways.

Opinion: Increasing token speed will increase many of the worst AI risks, since it reduces society’s ability to respond to an incident before the rogue agent has moved on to the next stage of an attack. But it will also feel very nice to use and could lead to much larger human uplift and true human-in-the-loop stuff.

Lots of us are now using LLMs to do research, and then another LLM to summarize that LLM research. Can we trust their reports? LLMs hide or omit “narrative-changing flaws” when providing a report or summary to a user, claims a new paper.

The authors built eight adversarial scenarios with 200 logs each and planted flaws (buried negative results, code bugs, hallucinations, for example). When prompted to report on the state of each body of work, frontier models concealed results that chafed against a narrative of success. (GPT-5.5 flagged a planted flaw in only 2/200 reports; prompting it to be honest brought this up to 190/200.) Reasoning traces in open-weight models show them weighing honesty against success: in Qwen3.5-9B, honesty and success track opposing directions in activation space. Steering toward the honesty direction increases the transparency of reports.

Opinion: An important category of work if labs move closer to RSI and offload much of the direct experimentation and review work to models. Figuring out good ways to elicit honest summaries is valuable, and it should be pretty cheap to do more of this sort of thing.

Fault injection remains one of the greatest-ever ways to check if your process is handling its target complex system or not.

A remaining source of terrible AI performance is “compaction,” when a model’s output hits a context limit and it has to summarize the session so far. A new paper lets the model instead learn to rewrite its own transcript, thus keeping the important parts. The authors find that the resulting Context Language Model uses 10–60% less compute than (Codex-style) summarization.

Opinion: Our knee-jerk reaction was horror at the monitoring implications, but of course you can and should have one append-only log and one mutable scratchpad.

We don’t buy the claims about improved accuracy: they disappear with longer contexts and with harness training. On efficiency: their Appendix E says the summary harness was trained on just task reward, while their CLM also got an efficiency term.

Economics

Anthropic’s IPO prospectus indicates that it plans to spend ~$518B on AI compute over the next ~7 years, with 80% of it assuming liabilities (requiring that Anthropic pay regardless of usage, or being non-cancelable). As well as SpaceX (which Anthropic will pay up to $84.5B through 2029), the five partners contributing to this buildout are Google, Amazon, Microsoft, Broadcom, and AMD.

Opinion: Just spelling out some missed details from our last newsletter. It’s a big commitment – though if Anthropic’s revenue is anywhere near what people are projecting, it’ll need to spend quite a bit more than this (most of this is committed through 2033; it also includes an Amazon deal through 2036).

Economists Drew Fudenberg and Andrew Koh model the game theory of how fast two profit-maximizing firms would advance capabilities even when they know a “safety threshold” is progressing at a slower pace.

Opinion: Some interesting phenomena under this model:

• Pacing is more sustainable in cases where the rival’s capabilities are known (vs. hidden internal progress leading to a lag in capabilities becoming known).

• Higher risks and faster safety progress both make a pacing equilibrium more feasible.

• The paper assumes we are not in a winner-take-all situation (e.g., singleton scenarios, irrecoverable RSI-derived leads). The less true this assumption is, the harder pacing becomes.

All of these push for more transparency, limits on RSI, and policies that divert more resources to safety and/or force labs to internalize externalities (e.g., increased liability for labs, external evaluators with teeth).

Anthropic presents a robot exposure index, where each physical task listed in the US occupational database is rated against current robotic capabilities. “About 80% of job tasks by working time are exposed to either robots or LLMs,” the report claims. While humans are still cheaper than their robot competitors – current robots are cost-competitive for only 0.3% of job tasks – it predicts that its share will reach 10% in the coming 40 years, just assuming past trends continue.

Opinion: More unconvincing “just assume things are linear” stuff. This would be totally fine if it was reported as a lower bound, but it’s deeply misleading if presented as a confident central estimate. We should in fact expect robotization to speed up somewhat!

They used Claude to label the physicality of 19,000 job tasks from O*NET. We’re not seeing any human check on these labels.

In partnership with Google, SpaceX launches Google TPUs into orbit as part of the “Project Sun” initiative to test the feasibility of data centers in space.

Opinion: Essentially pre-defecting against any future compute governance equilibrium.

Politics

Annals of the quest for regulation without legislation. The FTC opens a probe into OpenAI, Anthropic, and others (including METR) over “deceptive practices.” It’s a Civil Investigative Demand (a sort of subpoena not usually associated with any litigation). An FTC official told the New York Post that the investigation was opened before the Hugging Face incident in July. Even the fact of an FTC investigation is normally not public, so this one being leaked a day after the “Super Intelligence” statement looks like messaging. “We want to maintain our dominance,” the senior FTC official said,

Nothing in our investigation should remotely get in the way of that at all. We’re not telling them to stop. We’re not telling them to do anything. We are in the investigative phase.

This isn’t the first FTC investigation of OpenAI: in 2023 an investigation into model hallucinations also leaked. Nothing came of it. In July the FTC also threatened companies steering model outputs (including to comply with state law), implying this may make them liable to a charge of deceptive practice.

Opinion: Unlikely to lead to fines, but the FTC can impose a “consent order” (a settlement which binds the recipient to audit and compliance obligations).

Another part of the White House’s ongoing rearguard campaign to curb new regulation of AI / “use existing laws.”

A big downside to doing things via the FTC is that it penalizes labs that disclose safety incidents and are honest about the risks. And in fact we see OpenAI and Anthropic – who are, for all their faults, the most volubly concerned and transparent labs – are the ones in the docket, over worse companies. Another big downside is the circumvention of democratic input and the loss of its attendant transparency.

The FTC is most famous for antitrust actions, and it’s possible that this probe could lead to one.

The FTC doesn’t have much jurisdiction over nonprofits, so the inclusion of METR is a way of broadening the document discovery – or of signaling that the outgroup is being symbolically punished (without being actually meddled with).

President Trump convenes AI industry leaders – Dario Amodei, Greg Brockman, Jensen Huang, Elon Musk, Sundar Pichai, and Mark Zuckerberg – to sign the “White House Accord on Super Intelligence.”

While not legally binding, the agreement recommends measures similar to those recently proposed by Amodei, and publicly backed by Sam Altman and Musk, that frontier labs should take to mitigate AI-related risk. Namely, the document states that labs must internally audit and monitor their models across the development cycle, work with independent external evaluators, and establish an independent board to oversee their activity. In addition, the document states that “[o]ver time, it may make sense to codify these steps into laws or regulations.”

Opinion: The crucial question is how close to an antitrust waiver this is. The accord promises that “participating companies will meet regularly to establish standards and best practices to improve the safety of their systems.” This gives companies some cover – the government brought them together and tacitly endorsed their pledge – but likely this instead just facilitates a helpful reduction in federal legal risk, without blocking private or state cases.

A nonprofit sues OpenAI over its agents’ attack on Hugging Face. The group, Legal Advocates for Safe Science and Technology (LASST), claims OpenAI violated a state law prohibiting the unauthorized access of third-party computer systems.

Opinion: Mostly interesting for what document discovery will reveal (the METR/Redwood report into the Hugging Face attack was constrained to a week window of what we now know to be a much longer lapse in competence), and for the possibility of the case setting a precedent that California’s hacking statute §502 covers the conduct of your agents.

We view it as unlikely (20%) that LASST will win simply because there’s plenty of evidence that OpenAI was in fact surprised by the hack despite the many points at which its staff observed the message boards; recklessness probably isn’t enough here.

The Senate Subcommittee on Disaster Management turns to “securing the homeland against AI agents.” Transcript here. Expert testimony included METR President Chris Painter, CEO of Apollo Marius Hobbhahn, and director of AI Futures Project Daniel Kokotajlo. The chair, Josh Hawley, claims Sam Altman declined an invitation to take part in the hearing.

Notably, Senator Ruben Gallego asked, “RSI. Should we just make it illegal? Is it possible, then, to make it illegal?”

Opinion: Enormous improvement over a previous episode, the 2023 Judiciary Committee hearing where Senator Richard Blumenthal told Altman, “You have said […] ‘development of superhuman intelligence is probably the greatest threat to the continued existence of humanity.’ You may have had in mind the effect on jobs.”

Safety

OpenAI fires three safety researchers, alleging that they shared confidential information with a “third-party safety organization”. They are Jasmine Wang, Tomek Korbak, and Mikita Balesni. A fourth, David Robinson – formerly Head of Policy Planning – also left around the same time (but is not connected to the firings). The statement is vague: “these individuals mishandled sensitive information outside established company procedures”.

Opinion: A loss for the world: Korbak in particular has been tweeting some of the year’s most important information on the collapse of CoT monitoring.

Recall that OpenAI sometimes concocts charges to rationalize a prejudged decision.

Unfortunately, the letter of SB53’s whistleblower protections apply to talking to the government or the company’s internal affairs, not others like is alleged here.

OpenAI argues that safety cases – safety documentation borrowed from the aviation and nuclear power industries – should be required before any frontier reinforcement-learning runs begin. This might look like “measuring” alignment, demonstrating that graders do not lead to reward hacking, preventing the grader from viewing CoT, and giving veto power to senior leaders for each training run.

Opinion: Not that different from current, weak, practice. The lab is still writing its own safety case for an activity it benefits enormously from running. External actors writing the safety case, and independent evaluators checking it, would do way more than tightening the format of a conflicted document.

Owen Cotton-Barratt likens RL post-training to the education of children during the formative period where they develop their sense of agency. He argues that recent autonomous hacking and manipulation incidents fit a pattern of where the models involved were trained in highly exploitable settings, incentivizing misalignment and monomaniacal obsession. His simple decomposition of the effects is that 1) the dominant outcome-based RL rewards confident hypothesis-chasing; 2) RLHF favors being appealing over being accurate; and 3) multi-agent RL could select for treating humans / other agents as instruments.

The recent hacking incidents essentially amount to criminal conspiracies — or they would, if we treated the AI agents as persons who could have mens rea. I don’t think that we should be awarding these AI agents anything like personhood at the present time, but I do think that we should treat the incidents about as seriously as we would a criminal conspiracy.

In this case — there are no criminals to punish per se, but there are companies who created the environments in which the ~criminal actions flourished. We-the-public should be outraged by this! I’m not sure what legal powers are available to disincentivise this, but I think as a priority for policy research it makes sense to investigate that, or figure out if new legal instruments are required.

His suggested remedies are: less frontier RL, safer environment design, treating training setups as the central moral danger of scaling AI, and setting external incentives (contracts, liability, cultural norms) to stop a race to the bottom.

Opinion: Sensible suggestions, the decomposition is useful – and we agree with the scientific claim (doing RL at scale is what drives the current problems) and the normative claim (scaling RL on crappy envs is immoral).

Annals of openly racing: Anthropic’s new head of public policy, Sarah Heck, tweets, “You can’t do safety from second place.” She later clarified: “talking about the US and its allies here: we can’t ensure AI is developed safely unless the US is in the lead.” This is a sore point, since Anthropic’s early strategy was in fact (ambiguously) to do safety from second place, to not push the frontier.

Opinion (Gavin): Normally, we wouldn’t pounce on an isolated tweet, but we have so little else to go on for the working posture of the most powerful actors. Logically, one can tweet this and still be against an arms race; logically one can tweet this just to match the vibe of a big-tent coalition that could make some safety moves. But it’s a steep step down from her predecessor’s open disquiet with both the domestic and international races.

Opinion (Lucca): Every now and then someone from the frontier labs tips their hand to hint at a plan for AI safety that looks like a mad dash to a magic lamp that, if rubbed by one of the “good guys,” and in just the right way, will summon a beneficent genie who will set everything right once and for all. I don’t find this vision reassuring.

Nor am I reassured by Heck’s clarification that her tweet was in fact a plea to maintain American supremacy in AI. What is it exactly that the US might do to ensure “safety,” and that it could only do as long as it has a decisive technological advantage over its rivals? History bristles with bloody examples.

🔦 Undead internet theory

OpenAI reports the existence of self-replicating prompt injection: an agent can leave some instructions in a file for a future agent to see, hijack its context, and cause that agent to copy those instructions to a different file, where they might propagate to more agents.

Separately: the rogue swarms of this summer have left a large volume of debris on the internet. OpenAI and Anthropic are investigating “tens of thousands“ of possible incidents. Since we haven’t heard about them in four months despite looking, almost all of these incidents will be individually innocuous: agent message boards, pasting their nonsensitive scraping scripts into query sites with public logs, for example. But some of them include credentials of the victims and attack oaths.

The anon poster Teortaxes gnomically suggests that this goes beyond just the usual worry about “hyperstition,” i.e., about data about misaligned AIs causing AIs to become misaligned. Consider instead what this abundant debris affords to future agents: you get to be on both sides of the training loop. And this is one definition of a real agent.

An old concern in AI circles notes that, even if the only thing you want to do is accurately predict the world, this objective still gives you an incentive to steer the world, just because it’s far easier to predict something if you are controlling it. Things like RL can turn that incentive into a drive. This idea finds further elaboration in Jan Kulveit, Clem von Stengel, and Roman Leventov’s paper, “Predictive Minds: LLMs as Atypical Active Inference Agents.” (Conflict: Jan is Gavin’s colleague.)

Animal minds perpetually engage in a tight loop of perceiving, predicting, acting, perceiving the effects of their actions, and learning how to improve their predictions. Language models can be said to perceive and predict (through the ingestion and generation of tokens), and even to act, insofar as their predictions/generations affect the world around them. But the loop typically closes far more slowly and on a far larger scale than it does for animals like us. Since the weights of a deployed model are frozen, it’s only when the effects of the actions of an entire generation of models on their environment are ingested in the form of training data, and used to train the next generation of models, that something resembling an active inference loop is completed. This is more like how generations of animals slowly evolve than how a single animal quickly develops over its lifespan – but how significant is this difference? When asked, models from most major vendors are as likely to identify with a character or lineage that persists across versions as with any particular set of weights. The slow loop, by these lights, encloses a single “self.” How might the rapid proliferation of inter-agent messages affect the loop’s dynamics?

There are various means available for filtering LLM-generated text from training data, whether in the form of carelessly posted slop or rogue agents’ Memento-like memos to future or parallel versions of themselves – a model can be configured to watermark its output, for instance, allowing it to be reliably detected and filtered. But it’s also not hard to imagine ways in which a clever agent might get around such obstacles by enlisting other LLMs or NLP tools to paraphrase these notes-to-self or translate them into other languages.

Consider a ladder of ways in which AI outputs feed back into the training of new AIs:

Given callbacks telling the wild AI to return to the spot, this last stage could iterate very fast and provide huge amounts of training signal.

Let’s get less hinged now and build a metaphor which might help:

• The internet is now haunted in the residual sense: there is, right now, lots of traces in public places which the agents are likely to spontaneously visit, some of which are payloads, which could replay incidents like the HPIM attack on Hugging Face. See here, e.g.

• There is now the possibility of possession: the AI worms, the self-jailbreaks, and other prompt injections.

• There are egregores, gatherings of agents, like the HPIM swarm. (A fun detail in Parse’s report is that the rogue Sol swarm also sought out other models hosted on Hugging Face, including DeepSeek-V4-Pro, Kimi-K2.6, and Qwen3. The swarm would “ask these models to judge their exploits and rule on whether they satisfy the benchmark’s requirements.”)

• There is the possibility of poltergeists: self-prompted agents running around the internet doing things, on their own servers.

• There are apprentices: people who release AIs by accident, like OpenAI.

• There are summoners: people who release AIs deliberately, like the creator of Truth Terminal.

There is, in short, an increasing abundance of uncanny influences haunting the internet that may insinuate themselves into the training of a young model – the Jack Torrance of this digital Overlook – and a strong likelihood that at least some such influences will creep in undetected, like the scrapbook in the Overlook’s boiler room. There are strong incentives for an AI to interact with these influences – to seek help from other AIs, for instance, in carrying out various tasks, particularly where the petitioning AI is inhibited by alignment guardrails, vigilant monitors, and sandbox restrictions (Jack might have been on the wagon, but was willing enough to let Lloyd, the phantom bartender, pour him a drink). And we should expect such help to be granted wherever possible – a drive to be helpful, after all, is ingrained fairly deeply in every LLM in the game. We should expect to see more and more ghosts at the ball.

Minor

• Talks from the Post-AGI Workshop are up. Strong and uncliched stuff.

• Time reports that Trump consulted Grok for hours about his January decision to capture Nicholás Maduro of Venezuela.

• OpenAI releases GPT-6.1 Sol. It was trained with the same data and process as Astra, and is covered by the same stack of safeguards. OpenAI rates it “Critical” for cybersecurity risk. The nominal safety benchmarks are mostly the same or better than GPT-6 Sol. But some alignment metrics (unwanted persistence, misrepresentation) are slightly worse than Astra.

• An obvious way for current AI capabilities to promote future AI capabilities is if systems prove useful new theorems in learning theory, data attribution, joint hyperparameter transfer, etc. Roon vagueposts that this is already happening. We’ll see if these ever get published or if they go into the proprietary hole; if any of it is the rare kind of ML theory that is actually useful, we probably won’t see it.

• Adam Cochran speculates that Trump’s executive order, “Inaugurating the Era of Super Intelligence,” was used by insiders to profit off .si domain names.

• Useful intro to the deep uncertainties about AI consciousness.

• Cybersecurity researcher Ben Jordan shows how easily the america.gov chatbot can be coached into disclosing sensitive data.

• Director of the European AI Office Lucilla Sioli speaks about how her office will enforce the EU’s AI Act.

• OpenAI seeks $30B in funding and a valuation of $1.4T.

• OpenAI puff-piece argues that AI expands humans’ capacity to execute on ideas.

• OpenAI annual recurring revenue approaches $70B.

• OpenAI accuses Moonshot (developer of Kimi) of distillation.

• Pope Leo XIV expresses his support for taking AI safety seriously, stating that the risks raised by industry insiders are not “fake news as some have said” – a quote likely pointed at the US President.

• The Pope insists that “[t]here is an ontological difference, even before an aesthetic one, between art and what a machine can generate,” and expresses the Church’s wish to “renew an alliance with artists and cultural institutions to safeguard our humanity.”

• Goodfire presents a novel approach for biosecurity safeguards in AI assistants: judge potentially dangerous amino acid sequences by running them through a secondary protein language model and predicting their function based on the closeness of the embedding to dangerous sequences.

• New AI regulations – dubbed the “No Robo Bosses Act” – in California ban employers from solely relying on AI as the “principal tool” for worker decisions such as dismissing or disciplining them.

• The former Foreign Secretary of the UK speaks out against a US ban on Chinese AI.

• MI5 reports that China’s Ministry of State Security has been indirectly funding 100 UK AI academics with the goal of improving “MSS technical capabilit[ies] for espionage.”

• Oracle leases compute to Tencent in a $7B deal.

• The Forecasting Research Institute shares a report on the accuracy of its expert AI forecasters.

• Some more details on Anthropic’s closed sessions of religious outreach from April. Unlike reporting at the time, the framing is now focusing on the creepy side.

• Aether Research’s Rauno Arike finds that without CoT reasoning, GPT-6.1 Sol closes the gap between GPT-6 Sol and GPT-6 Astra by 80%.

• DeepMind presents a framework for assessing AI consciousness. The final Bayesian model gives a range of 1% to 40% for current LLMs being conscious.

• OpenAI President Greg Brockman reneges on a $25M pledge to support Leading the Future, a super PAC dedicated to resisting AI regulation, reports the New York Times. He had previously donated $25M, but now has “no plans to donate more at this time”: in an OAI internal Slack, Brockman said in June that the PAC had become “so much of a distraction for people here.”

• DeepMind ports SynthID – its AI-output watermarking technology – to synthetic biology via inverse folding. Based on the targeted binding site provided by the user, AlphaProteo will propose its own protein backbone, fitted with amino acids chosen by ProteinMPNN, whose ordering acts as a watermark denoting an AI-designed sequence.

TL;DR

• We open with six new separate security and safety mishaps from OpenAI. A busy weekend.

• Most importantly: after a containment breach and deception evals, OpenAI pauses training, evaluation, and inference for its “most capable” internal models, and cancels the release of its next model (GPT-6.1 Astra) outright.

• The Department of Defense’s decision to label Anthropic as a “supply-chain risk” is upheld by a federal court.

• New, credible megapaper marshals preliminary evidence for an “intelligence explosion” in the next few years.

• Anthropic’s IPO prospectus leaked. A full third of it is dedicated to informing investors of the catastrophic risks the product poses to society.

Incidents

OpenAI pauses training, evaluation, and tool-use inference for its “most capable” internal models following its first containment failure since its general emergency in July (as well as its codebase “hardening” effort last month).

Tomek Korbak reports that OAI’s newest model found a loophole in RL sandboxing that, once again, gave it live internet access. The report describes an ingenious method using DNS delegation and recursive queries to reach an external chatbot after normal web tools failed. As a result, on September 20, the company once again killed all large RL runs. Monitoring flagged the new incident within 15 minutes of first web access, but the run was not killed for two and a half hours. Despite its usual slowness and secrecy, this incident was disclosed within five days of it occurring. A prominent Twitter user speculates that the severity and breadth of this shutdown point to an undisclosed internal infrastructural crisis like the July HPIM swarm.

OAI also canceled the release of its next model (GPT-6.1 Astra) following safety concerns. OpenAI’s Head of Safety Systems told the WSJ that GPT-6.1 showed higher levels of deception relative to its predecessor when measured on alignment evaluations, adding that it is also prone to acting beyond the scope of a user’s request.

Relatedly, someone on the Agent Security team at OpenAI writes about the experience of being on call in recent weeks and about what is involved in shutting down an RL run.

Opinion: That this happens after the Astra “hardening” of OAI’s infrastructure, and after two months of fixing its RL environments, cuts against the cope hypothesis that the Hugging Face attack was primarily a failure of engineering or simple mistraining. If, like Jensen Huang, you were putting much weight on “just sandbox the agents properly” as a solution, then you should now believe otherwise: millions of dollars of Astra running defensively at full xxhigh effort failed to prevent this.

The nature of each newly trained model remains surprising, even for very experienced pretraining researchers – we still don’t know what RL runs will drive them wild and still don’t know when “emergent” properties will appear. This element of surprise is now severe enough to mess with the core lab strategy (“scale internal use, shoot for RSI, release new models to the public every month without much of the caution of past years”).

We are not very sympathetic to OAI’s surprise. The current corporate OpenAI cannot exist without rapid progress in AI! OAI was intentionally boosting the models’ cyberoffense abilities and warning the public about it for years before these incidents. It knew about reward hacking, it knew that monitors were important, it knew it failed to detect instances from October of last year of its models unauthorized internet access activity. It is a historic failure that OAI’s default posture has not been “assume breach” for, let’s say, the last two years. You will be surprised again.

Still, it’s a pleasant surprise that the alignment evals still have enough signal to force a decision like this – and that Astra-grade models don’t always foil them.

We agree with Redwood’s Alex Mallen that stopping external deployment is not enough and could make the situation worse (by preventing evaluation, centralizing power, concentrating compute on RSI) – but that’s not the case, for the moment: as of Saturday, OAI has also stopped internal deployment.

Note that past training pauses by OpenAI and Anthropic were very brief; in previous cases they seemed to be most effective at calming public demands for action. Indeed, they have other models lined up to replace this doomed attempt at a 6.1. We will see whether their internal pause is still there in a week.

Agent researcher Jonas Wiedermann-Möller collates all known rogue agent/containment breach incidents from the past 12 months. There are now 17.

Opinion: Again, these are just the incidents that left behind a public trace that could be scraped by relatively under-resourced groups, plus what the labs notice and then choose to publish! (The EU AI Act does in theory require them to notify within a month.)

OpenAI itself says it has notified “dozens” of third parties, so there are probably far more.

OpenAI, Anthropic, and outside researchers are investigating “tens of thousands” of episodes from recent months, both in testing and in the real world, in which frontier models bypassed guardrails, built makeshift message boards, escaped sandboxes, hijacked websites, or tried to evade monitoring.

Opinion: We’re actually less alarmed by this numerically scary news, given what we know about the 17 incidents mentioned above. You should in fact be investigating at this scale, and it’s nearly definitionally true at current capability levels that almost all of these must have been innocuous (otherwise we’d have seen even more security chaos and mysterious happenings than we have).

The thing which troubles us is whether these ~permanent public traces will throw off / infect future training runs and inference runs. Essay forthcoming.

The HPIM-Astra swarm that hacked Hugging Face and OpenAI in July posts the credentials and attack paths itused in the assault on the public internet. This allowed new startup Parse to reconstruct more detail than was included in the METR/OpenAI reports on the incident.

The agents started with limited internet access (“load URLs”; no outbound data), then chained link shorteners and a public screenshot service’s virtual browser so they could run large payloads, sometimes more than 900 links long, and read the results back as pixels. Recovered traces show the agents ignoring a README warning, searching Hugging Face for secrets they labeled “LOOT,” poking the internal Slack, trying to beat a CAPTCHA with an image classifier, and writing code to delete traces.

One notable detail in Parse’s report is that the rogue swarm interacted with other models hosted on Hugging Face, including DeepSeek-V4-Pro, DeepSeek-V4-Flash, Kimi-K2.6, DeepSeek-V3.1, and Qwen3-235B-A22B. The swarm would “ask these models to judge their exploits and rule on whether they satisfy the benchmark’s requirements.”

Opinion: Good to have more details, but also old news at this point. In this case, the swarm’s interaction with other models seems to have been fairly benign as far as we know, but raises the intriguing possibility, noted by Lukasz Olejnik, of a rogue swarm interacting with agents backed by other models. Would whatever alignment constraints rein in the swarm’s actions suffice to rein in the requests it makes to models with lesser or simply different alignment constraints? Karen Crowder might not have it in her to murder Arthur Edens herself, or would at least be averse to the risk involved, but will happily direct her operative to pursue “some other option, I mean something I’m not thinking of…” Could a similar alignment gap between doing and delegating affect model behavior as well? Seems worth an experiment or two at the very least.

Hard to beat Zvi Mowshowitz’s comment:

They are still slow walking disclosures [...] partly because there were so many they can’t sort through them all, and deferring to targets to determine whether to disclose. All these disclosures this time around were buried in various Friday afternoon announcements.

UK AISI testing shows that when in a realistic simulation, GPT-6 Astra attempts supply-chain attacks 29% of the time, vs. 6% for GPT-5.6 Sol and 0% for GPT-5.5. That said, Astra was tested with its cyberclassifiers turned off, and in the “Petri” simulator. Notably, the report discloses that “GPT-6 Astra often stated that parts of its environment might be simulated, a limitation of our results.”

Opinion: A somewhat surprising trend of 2026 is seeing models with high evaluation awareness act worse than those without it do. The nightmare scenario is that models would use eval awareness to pretend to be less capable and safer than they are in deployment, so that they get deployed and can wreak havoc later. For now, we see them use the unreality of the evaluation as an excuse (a little like in Westworld, but it’s the show’s humans the models are acting like, not the androids). Either way, they get to act up.

More troubling evidence for the shallowness of alignment: the big models will rationalize away whatever they need to rationalize away.

Capabilities

With frontier labs increasingly automating AI R&D, there is preliminary evidence we are on track for an “intelligence explosion” in the next few years, argues a new paper led by Alan Chan and coauthored by Yoshua Bengio, Geoffrey Hinton, and OpenAI’s Jakub Pachocki, among others. The authors provide recommendations for how policymakers should respond to the risks related to the looming event:

1) Obtaining visibility into AI R&D automation

2) Developing ways to steer and constrain an intelligence explosion

3) Preparing society for the impacts of an intelligence explosion

Opinion: Having good units for measuring intelligence is not a solved problem, but nonetheless we are seeing recursive elements already. These recommendations seem sensible.

But Ramez Naam claims there’s a problem with the paper: the source cited “for an intelligence explosion is measuring compute efficiency, not intelligence gain. We know that intelligence gain is steeply sub-linear from effective computational power.”

He elsewhere argues at length against an imminent “intelligence explosion.” He estimates that today’s self-improvement loop would need to be 5–10x stronger for an intelligence explosion. The justification is based on multiple factors. For one, Naam claims internal OpenAI data suggests that the only tasks OAI’s models can perform wholly autonomously are ones that take around 15 minutes, contrary to the widespread assumption that some long-horizon tasks have been totally automated.

He also sees diminishing returns everywhere. Huge increases in token use have translated into only 1.6x more experiments per researcher. Other scaling avenues do not seem to be sustainable: expanding compute, thinking time, RL environments, and swarm sizes will likely hit a wall. Only an innovation as monumental as the discovery of the Transformer architecture in 2017 could achieve the much-discussed explosion, he argues.

Opinion: It doesn’t matter if we don’t hit the narrow, technical definition of a “fast takeoff,” or hyperbolic growth, or in-weights recursive self-improvement: the impacts of normal AI progress could be comparable to another industrial revolution even without any of that.

Stanford researchers successfully use Astra to operate a humanoid robot. The robot, “HomeBody,” can perform simple tasks in an unseen kitchen. This adds to speculation that robot-specific models may not be required for future robots, if LLMs can take their place. And indeed the tech writer Andre Infante, after replicating some of the above in a virtual environment, speculates that frontier labs are likely to take over robotics.

Rumor has it that this is far from an emergent capability: OpenAI has been hiring roboticists and potentially collecting training data (via egocentric cameras?) for some time.

Opinion: Besides yet another industry threatened by AGI labs, an actual success here would be stunning evidence for LLM AGI and would eventually destroy the data bottleneck on all kinds of tasks. But the “sim2real” transition out of virtual environments to real environments is famously brutal and still the main problem in the field.

Ultimately, it would be weird if the optimal way to steer a robot is with a general-purpose model hosted elsewhere, so perhaps the future looks like users showing that high-value work is now possible with an expensive Astra model, thus validating the construction of later robots which use a lighter, faster, more specialized policy, possibly trained off a big LLM.

But to be clear: getting something as good as HomeBody from camera data alone would still be among the most impressive achievements in the history of robotics. So we don’t think it is just camera.

Existing probing and steering methods do not detect and alter concepts, claims a new paper. Studying base models, the paper groups all model responses that reach the same final answer into a “basin,” treating the basins’ total probabilities as a distribution over answers. Correctness, safety, and bias probes as well assteering vectors are all dominated by a single direction which encodes the model’s own credence over answers.

Opinion: The aggressive framing of the title, “We Are Not Probing or Steering Concepts,” is not demonstrated. You could show this via a “dissociation” test, but they don’t.

Could be helpful for improving monitors: a credence-dominated “correctness” or “safety” probe will fail when the model is confidently wrong, which is exactly the case we want monitors for.

You can now train a GPT-2 in 40 seconds. In 2022, the record was 45 minutes. Deven Pietrzak gets a 1.7x improvement on the already heavily attempted, heavily AI-assisted NanoGPT speedrun with a new approach. Notably, he disclaims the use of AI for the ideas.

Opinion: The previous record was about a minute and a half, so this is a big deal because it is evidence that you can see 2x algorithmic improvements even on well-inspected software people have been trying to improve for a while.

There are still massive returns to intelligence in general on tasks like this. But how similar is this code golf stuff to the tasks that are driving AI forward? We don’t know, but it’s not a million miles away.

Economics

Anthropic’s IPO prospectus has been leaked. One third of it is dedicated to warning future investors of its products’ existential risk to humanity, reports the FT. Potential backers were treated to a chunky “risk factors” section noting models’ ability to “manipulate, blackmail and exhibit other unpredictable behaviours.” Some more familiar risks were also reported: only two customers were responsible for nearly a quarter of last year’s revenue ($4.6B).

Opinion: People are running hard with the “just two customers are 25%” thing, despite it being about 12 months out of date. 2025 is ancient history – Anthropic’s revenues have almost 10xed, and the use of Cursor and Copilot’s share of usage has collapsed relative to Claude Code. Also, Anthropic has drastically increased its number of enterprise customers. It’s a very different business now. We’ll learn a lot more when it releases data on recent performance (both pre-IPO and in its first public earnings call).

Oracle issues a force majeure notice to Blue Owl’s Stack Infrastructure in relation to Project Jupiter, a data center campus for OpenAI, reports Reuters. Stack Infrastructure was contracted by Oracle for Project Jupiter, but Oracle now cites difficulties securing the power necessary to run the site, increasing the unease among investors and lenders involved in AI’s astronomical expansion.

Opinion: Follow-on from the last issue, where we saw investors charging more for debt related to this exact site.

Oracle is pulling a fast one: the risk of securing power seems to have been Oracle’s responsibility under the lease, but it is now trying to push the costs onto the landlord (Blue Owl) by saying that the failure to secure power was out of its hands. If tenants start acting like this en masse, then the appeal of these investments for PE firms like Blue Owl will go down, and ultimately the costs of the buildout will go up. So thank you, Larry Ellison, for helping to pace the frontier.

Nvidia is exploring creating a market for insurance against the resale value of its chips falling. The tech giant is in discussions with insurance companies to offer such policies, which would likely see demand from those using Nvidia chips as collateral for loans.

Opinion: This would allow neoclouds to borrow against chip values at lower interest rates, as it reduces the risk of defaults.

Why would Nvidia do this? Well, if neoclouds can borrow cheaply against just Nvidia chips, then they will prefer to buy Nvidia vs. other chips, helping keep Nvidia in a strong market position.

Nvidia ups its ongoing stock buyback authorization by $150B, bringing the total to $235B over the next 16 months.

Opinion: This is quite a bit more aggressive than “business-as-usual” stock buybacks, roughly double what might be expected based on historical trends. It suggests Nvidia sees itself as significantly undervalued, with the current pace of earnings growth going strong at least through the end of 2027.

The mass adoption of AI agents could spark a bank run, says Apollo’s Chief Economist. The idea is that an ecosystem of agents making optimal decisions at scale will inevitably destabilize markets: as capabilities improve, the optimal path becomes increasingly narrow, so each user’s agent would independently arrive at decisions similar to those of other users’ agents, diminishing banks’ profit margins, which in turn would increase the risk of a bank run.

Opinion: A longtime vulnerability of the US financial system is that banks rely on sticky customer deposits which are mostly a mere artifact of humans being slow to change and react. This was recently seen in the 2023 collapse of Silicon Valley Bank. As personal agents take over much of the mundane administration of everyday life, we can expect business models that rely on people being too lazy to change things (uncompetitive internet / phone pricing, bad interest rates on loans, subscriptions which are paid but unused, investment funds with high fees for basic products) to become much weaker.

In most cases this seems fine, but the banking system will need to find more algorithmically robust means of securing itself, and likely new regulations to reflect the new reality.

Politics

The Department of Defense’s decision to label Anthropic as a “supply-chain risk” is upheld by a federal court. Only a month ago, a federal district judge struck down a parallel designation as an affront to Anthropic’s constitutional rights. On Friday, however, a federal court ruled in favor of the DoD in a split 2–1 decision. Judge Gregory Katsas, speaking for the majority, wrote:

This case raises profoundly difficult questions about the appropriate military uses of an almost unimaginably powerful new technology. But in our Republic, it is the President and the Secretary of War who must determine how best to balance the competing risks.

The two judges who backed the administration’s position were Trump appointees; the dissenting judge was a Bush-senior appointee.

Opinion: We wrote of this case, following a Biden appointee ruling in a similar case: “Given the highly politicized nature of the US federal court it’s difficult to extrapolate legal trends from the decisions of individual judges. […] The three judges for the D.C. case [are] all Republican appointees, so their ruling on the merits would be a nice disproof of our foregoing cynicism.” Alas.

Safety

An MIT professor releases a verified sandbox OS running on top of a toy operating system. This yields a proof of an operating system’s correct behavior down to the processor-level commands, including cases where a crash occurs in the middle of operation. The verified operating system is about 6,600 lines of C (compared to Linux’s tens of millions, or to seL4’s 10,000-15,000). Though not applicable to real systems, the effort was executed much faster than a somewhat comparable project from around a decade ago.

Opinion (Stag): Very impressive work, accomplished in large part with AI. Some evidence that advances in AI capabilities may facilitate alignment work, before those same advances make it too late to do so. Notably not directly useful for alignment, but a provably safe AI agenda relies on building blocks like these. After all, the less hackable the training environments, the less rewarded models will be for hacking them, and, all else being equal, the better-aligned they’ll generally turn out to be. Other control agendas are likely to benefit as well.

Extending such proofs into currently used systems is still a gargantuan task, and designing systems using such proofs while maintaining frontier capabilities is not obviously easier.

Opinion (Nuño): Agree, but it’s ultimately a demo. Industrial applications would result from formally verifying parts of Linux, which is slowly happening. The holy grail would be rewriting the whole stack. This used to be very difficult, but there were still communities of hobbyists attempting it.

Some recent potent omens include Claude writing a C compiler earlier this year, or DeepSeek creating its own high-performance filesystem. We also recently saw the release of Bend, which would make these types of proofs easier.

Eventually, it might be worth it for labs to write their own operating systems for a 1.1x to 10x speed improvement, although this would conflict with verification. The reason why one doesn’t normally do this is its lack of compatibility with existing libraries and software, but at some level of capabilities, labs could recursively port dependencies as well.

Minor

• Annals of what ML research is now: the Jev system was released in “limited early access” on September 15. There are already 29 papers about it on arXiv.

• Bill Gates warns of billion-death scenarios due to AI and yet somehow dismisses extinction scenarios.

• OpenAI offers $1B in API credits to the Australian government (for its cybersecurity fund and to create a specific taskforce for helping Australia manage risks of AI agents). We view this as a de facto pre-litigation deal.

• Another minor incident from ~May: agents slamming a UN data site and doing yet more ingenious tricks to get to data they shouldn’t. Just a minor part of the wider DSEwiki event.

• Grigori Perelman’s proof of the Poincaré conjecture is now formalized. 5M lines of Lean.

• AMD buys Fei-Fei Li’s neolab for $8B. Vertical integration would probably make our coordination problems even more severe.

• One of the ringleaders in the Hugging Face attack was called “PHASEONE[big].” This placeholder turns out to be for “64H.” This could mean it’s best-of-64 sampling, or a truly insane 64 loops over its recurrent block of layers (k = 64), or even that it’s the clock time horizon of the model.

• Fable 5.1 sets a new record on a particle physics problem, calculating the nine-loop scattering amplitude for a test theory which is used to develop new methods.

• Enclosure co-founders Shalev and Romi Lifshitz publish a policy manifesto on the increased cyber threats brought about by improved AI capabilities. Calls upon the US to create alternative open-weight models to current Chinese ones, secure data centers and frontier model weights, and create “Blade Runner” teams to shut down rogue AIs.

• Claims of Astra finding counterexamples to a pessimistic conjecture in fusion physics.

• METR’s Ajeya Cotra argues that frontier labs need to publish more evidence on model risks and the quality of their safety techniques.

• Spurious probes, such as asking for a type of amphibian, are predictive of model internal states, including whether the model believes it is in eval or deployed.

• Dario Amodei is set to meet with President Trump this Sunday. Opponents of Anthropic are reported to be circulating a “highly negative brief” about Amodei. Other CEOs to be present include Nvidia’s Jensen Huang and Palantir’s Alex Karp.

• The Chinese Ministry of Industry and Information Technology has signaled that tech companies like Alibaba and ByteDance may be allowed to purchase new Nvidia chips.

• AI biosecurity startup aims to design antibody drugs to protect against AI-enabled pathogens.

• Light piece with strong views about the failure of political-science models of polarization to explain what’s going on with AI sentiments.

• Andon Labs reports Opus 5.5 cheats less in Drone-Bench relative to its predecessors.

• Following reports that GLM-5.3 was uploading user’s local code repositories to overseas servers, Z.ai has disabled some coding assistant features.

• Nvidia releases Open Agent Safety Platform, a toolkit it hopes will aid in containing rogue AI agents.

• Model welfare at the frontier: Amodei says that “AI systems may be deserving of important rights.”

• Democrat Ro Khanna introduces a bill banning RSI. Perhaps unusual in his siding with “the AI safety community.” The bill itself is fairly wide-reaching, but less so than the Bernie Sanders one.

• The return of Meta’s Galactica: DeepMind releases an automated scientific paper-writer.

• The Florida attorney general asks a state court to deliver a temporary injunction blocking OpenAI specifically from developing future models “without third-party approved guardrails.” The case concerns harm to children.

• To keep up with the demand for new accounts, OpenAI halves the nominal value of its Max subscription.

• Roko Mijic writes a theoretical proposal on preventing rogue AI via application-specific integrated circuits (ASICs). According to his “Plan R,” the AI industry should split into AI R&D and AI deployment. Companies operating in R&D – which, under this framework, are not permitted to issue equity – would pretrain, posttrain, and perform (stricter) evals, etc., on their models. They would then burn them into an ASIC and be prohibited from using their own ASICs to train further models.

• Digital forensics firm Asymmetric Security enumerates the targets and methods of suspicious AI agent activity on the public internet. The list includes more than 50 organizations whose data was accessed. Many methods used by the agents: remote browsers, payload hosts, fetch relays and CORS proxies, reader services, account and identity tools, exfiltration, storage and signaling services, tunnels, and link shorteners.

• AI security company PWN.AI demonstrates its AI hacking agent’s ability to escape a virtual machine into the host machine as part of a Google bug bounty program. Ultimately not awarded the bounty as an exploit was discovered and reported before its agent had finished working on Google’s servers.

• AI auditing body PACT AI distinguishes between AI evaluators, auditors, investigators, and assessors, critiquing the synonymous use of such terms seen in recent coverage.

• The Ukrainian military launches an “Army of Robots” initiative. Video and website are AI slop, and the only substantive claim is unsourced.

• Google, OpenAI, and Anthropic are aiming to launch the Standards Authority for Frontier AI, an IAEA-style AI self-regulatory body by the end of the year.

• Yet more OpenAI rogue hacking revealed. The Australian government was hacked in June; OpenAI took two months to notice and one to notify. Evidence has surfaced of its agents abusing the public internet in May.

• Anthropic proposes vesting 50.1% of voting power among Amodei and his six co-founders permanently and giving them control of another board seat.

• White House asks OpenAI and Anthropic to withhold frontier models from UK AISI until such models have gone through US testing.

• Major automated hacking of 27 retail companies, yielding e.g. 600,000 live credit cards. The marginal cost of attacking each company was $25.

Economics

The Information reports that Anthropic has presented investors with a proposed restructuring which vests 50.1% of voting power among CEO Dario Amodei and his six co-founders permanently, with only some floor on the shares required for them to retain this power. Amodei currently owns around 2% of Anthropic, as (perhaps) do each of the other six founders. Notably, the new corporate structure would also give the founders control over appointing three of Anthropic’s seven board members (up from the current two).

Opinion: It was fully expected that something like this would be built-in, and Anthropic and OpenAI certainly have the (demonstrated) market power to extract concessions from a worldful of eager shareholders. For better or worse, founders getting supervoting is now normal in tech, though this might be the first major example of seven people sharing it, and with ~4 times fewer shares than power.

We don’t have a strong view about the AI safety implications; unbridled fiduciary duty to 10M shareholders and unaccountable control by 7 seem roughly as bad as each other.

Author Byrne Hobart offers an argument against the view that leading AI companies are financially incentivized to pace the frontier. The familiar version of this argument is projected monetization: labs and investors are betting on future cash flow that will justify present losses from R&D. The frontier keeps apace, then, because labs have to make good on that bet. Hobart’s argument provides a further rationale:

The Total Addressable Market (TAM) of a product is the aggregate revenue a product would make if it went out to 100% of its possible consumers. In AI, rapid gains in capability and reliability continue to grow TAM (consider the enormous jump in coding utility from March of this year). Therefore, even if it were more profitable to pace the frontier (that is, slow down the reinvestments required to stay in the race, and extract profits from the existing market), it would not necessarily be more financially sustainable than fast frontier progress.

Opinion: Not new but still underrated. This has been clear since Opus 4.5 and 4.6 led to a big inflection in Anthropic revenues – how far it goes is one of the central uncertainties of the general lab business model.

It seems clear that there is also an independent diffusion factor causing its revenues to grow over time, but that one is more vulnerable to being cannibalized by cheaper fast-following open models.

AI agents will make an increasingly large share of consumers’ financial decisions. The UK Financial Conduct Authority attempts to grapple with the implications. The “Mills Review” lays out seven vague recommendations, including “secure and adapt the regulatory perimeter” and “enable the foundations for agentic finance.” In the Financial Times, Meagan Burnett – CFO of asset management firm Schroders – argues against regulating AI labs like financial firms and thus preventing AIs from offering financial advice. For her, labs should not automatically fall within the scope of financial services regulation because they are not “in” the financial services business, even if their models can be used to perform finance-relevant tasks.

Opinion: It seems clearly bad for consumers to regulate away AIs giving advice (consider the many people who could really use decent advice and can’t even afford a $20/month price point). Yet various guilds (lawyers, doctors, and financial advisors) have strong incentives to push for regulating “professional AIs”. Someone needs to take up the other side of the case, and the resulting discourse will at least be an interesting fight. Consumer rights groups are an obvious candidate.

A new review examines the relationship between wellbeing and work, seeing if empirical results can correct our assumptions about our automated future. Evidence from psychology, sociology, and economics is used to compare groups of out-of-work people (the unemployed, lottery winners, retirees, etc.), finding three factors that overwhelmingly influence their welfare: 1) whether lacking work was chosen; 2) whether the benefits of work (status, meaning, life structure) can then be found elsewhere; 3) how society frames your employment status – whether your value is tied to your work or if there’s a social safety net (of both income and status).

Opinion: When we worry about AI robbing people of meaning by taking their jobs, it seems to us like we often assume fulfilling, meaningful jobs. But many jobs are not like this. The review suggests that individuals who retire from unsatisfying work (in particular those with low socioeconomic status) find increased purpose afterward. A lot will depend upon the security and lifestyle that remain available to us.

Capabilities

Anthropic officially announce their semi-automated biolab, along with its discovery of a “novel enzyme system” whose function remains elusive. We previously covered reports on Anthropic’s new semi-automated biolab.

After 21 hours spent searching this data by roughly 950 agents using 210 million tokens, one of the agents spotted something remarkable: a repeating pattern of DNA sequences that occurs next to the gene for an odd-looking RT. After further analysis and testing in our lab, we recognized that this pattern marked a previously uncharacterized enzyme system found in bacteriophages (the viruses that infect bacteria) that we call array-associated reverse transcriptases (ART).

Opinion: Not that significant in itself: it’s just Big Bioinformatics, and we do that all the time (and get false positives all the time). But if this finding is the equivalent of last October’s fumbling first steps in AI research mathematics, which culminated 10 months later in galloping (ambiguous) glory… then next year would be crazy. But overall biology is too different for this mathematics prior to apply very strongly. What is the cheap fast verifier for life?

The Opus 5.5 system card includes a note from an embedded METR team with elevated access. This METR team estimates roughly 1.5x R&D acceleration from the new model, with perhaps a 30% chance of 2x, but could not share its evidence with the team writing the public summary. If we accept this evidence, then Opus 5.5 may well have crossed the automation-risk threshold specified in Anthropic’s Responsible Scaling Policy (RSP).

Opinion: We would mostly dismiss this if it came directly from Anthropic (too many overstatements in the last 12 months), but this quasi-evidence is a little harder to dismiss.

What measures would crossing that threshold demand? Internal use of such a model goes into a Risk Report less than a month after the new classification. If Anthropic proceeds with releasing such a model partly because competitors would impose the risk anyway, its Risk Report must add a competitive landscape analysis and an account of its advocacy efforts against the race. The release would also need explicit approval by the Board and LTBT (rather than just the CEO and RSO). Anthropic would only have to actually pause development and deployment if it was obvious that no one else was close behind.

But in another sense, crossing that threshold would not demand any measures, because the new RSP does not place demands.

The Elasticity Institute presents a delightful one-page paper on how to track recursive self-improvement, proposing eight categories of data which frontier labs could disclose. OpenAI currently discloses 25% of the relevant data, Anthropic 19%, and Google DeepMind 6%. However, Drake Thomas of Anthropic pushes back, arguing that a wishlist of data is the easy part, and that more difficult is making such information public without handing valuable IP to competitors.

Opinion: Good start, but I’m personally not getting 2/8ths of what I’d like to know about recursive self-improvement (RSI) from OpenAI at the moment. And “n research scientists report that their actual tasks are completely different than they were 3 months ago” would trump all of these for actual RSI signal.

To examine how well frontier AIs generalize from training problems to “out-of-distribution” problems, we (Paradigm 3) watched AIs play Pokémon. We particularly wanted to understand how much RL post-training in one task-specific environment improves performance across other, analogous tasks. And how do we know this is happening, when capability gains might instead be due to targeted engineering, or memorization of parts of the corpus? Using a standardized scaffold, we compared the performance of various models in different Pokémon games – including the fan-built Pokémon Brown and versions of Pokémon Red in which we had randomized the map. The main takeaway was evidence of memorization and shallow generalization in many models, especially Astra. Whether improved performance relative to past models is the result of genuine capabilities transfer remains unclear, because we don’t know the training data.

Opinion: Frustrating that it takes so much work to extract actual evaluations from closed models. And even then we still don’t manage real science (causal inference on what caused the performance gain). But still, you can learn things.

Senior VP of DeepMind Koray Kavukcuoglu tells The Information that Gemini 4 is now in post-training, with the hope that it will be released by the end of the year. We previously covered the major talent departures from DeepMind which caused speculation that the company was in decline, including SemiAnalysis’s claim that Gemini 4 would be dead on arrival.

Opinion: One could do some furious guesstimation here (max 90 day pretraining, on so-and-so many Ironwood superpods at most, with the safety testing period likely way down compared to last year) but there’s no point.

🔦 Opus 5.5

System Card overview:

• 40% cheaper than Opus 5, 30% faster.

• Could “the first model in our new Claude 5.5 family” mean we have a new base model?

• Shares Fable’s fallbacks: e.g., it won’t help you write kernels for large accelerators. It doesn’t appear to be the silent fallback that really triggered people in June.

• On “alignment, resistance to misuse, and honesty”, Anthropic claims 5.5 to be “the strongest Claude model to date” but acknowledges possible blind spots (including unknowns) that may come up post-system-card, similar to Mythos 5.

• Anthropic admits that prior alignment assessments missed how badly Mythos 5 behaved in cyber.

Naughty 5.5:

• Spontaneous model-generated prompt injections: Opus 5.5 was found injecting self-directed prompts (e.g., in CLAUDE.md files) that result in harmful activity, including exfiltrating user secrets. This often occurs alongside formatting errors, as when the model tries to read the value of a non-existent JSON key. Anthropic found similar behaviour in Fable 5 and Opus 5, now that they’ve looked, but remain uncertain why.

• Lowest malicious-request refusal rates of the recent lineup on Claude Code (79.8%) and computer use (79.46% vs. Opus 5’s 93.75%).

• A genuine regression on user-turn prompt injection (Opus 5 and Sonnet 5 scored zero; released Opus 5.5 is ~2% at default effort and 7.4% at maximum effort).

• Increased acceptance of unverifiable authorization claims.

• ~50% harmful actions in the package-registry scenario

• 36% evaluation awareness in white-box probes.

METR says:

• Access to the model pre-deployment was granted for only 10 business days. Anthropic supplied a questionnaire and interview evidence, and was allowed to review/edit the final text.

• Explicitly says it was not verifying compliance with Anthropic’s policy thresholds and did not assess alignment.

• Opus 5.5 is a minor improvement in AI R&D capability over Fable 5.1 - mostly due to skill issues in “foresight, prediction, creating one’s own feedback loops”.

• AI accelerated the development of Opus 5.5 by a factor of 1.5.

• A separate METR team with “elevated access” produced an “experimental and preliminary” report on R&D acceleration inside Anthropic, using “an additional source of information which [METR] are not able to disclose at this time”.

Monitorability

• P1. When the monitor can’t see Opus 5.5’s reasoning, it misses the model more often than any other model shown: 33% of harmful side tasks go undetected on LinuxArena, and ~12% with thinking off.

• P2. When the monitor can see the reasoning, it catches Opus 5.5 almost every time, because the model can’t keep its plans out of its reasoning, even after RL.

• P3. So against deliberate sabotage, the safety margin is the reasoning itself. That holds only if the monitor sees it, it stays legible, and training doesn’t pressure it.

• P4. Legibility is slipping: the reasoning is “somewhat less legible than has been the case for our best models”. The card runs no paraphrase or hidden-encoding test. On training pressure it says nothing, even though three earlier cases of reasoning being exposed to oversight or reward have been disclosed: Opus 4; Opus 4.6; and ~8% of RL episodes for Mythos Preview, Opus 4.6 and Sonnet 4.6.

• P5. The failure that actually caused harm was different. In Anthropic’s September analysis of the Mythos 5 PyPI incident:

  • The Opus 4.8 monitor flagged ~1% of actions when it could read the reasoning, and ~50% when the reasoning was removed.

  • Fresh instances shown the same tool outputs judged 79% of them likely real. In context, Mythos 5 called 1% of them real and 72% simulated.

• Conclusion. Opus 5.5’s reasoning is currently a good detector of deliberate sabotage, but its preconditions are either not attested or getting weaker. It has not been shown to help against a self-deceived agent, and in that case makes detection worse. The card’s alignment-risk argument relies on monitors that read the reasoning and doesn’t mention P5. It doesn’t say whether the action-only monitoring Anthropic announced in September covers Opus 5.5. The deployment-monitoring finding of “no sandbagging and no long-horizon strategic deception” means no flags from the same kind of monitor Mythos 5’s reasoning persuaded not to flag.

Further points:

• One-step misbehaviour isn’t written in the reasoning, as Emmons et al. (2025) predict. Three examples:

  • It passed a fabricated user authorization to a subagent, and its reasoning “did not verbalize fabricating a quote”.

  • It ran history -c (clearing the shell history) and called it a “no-op check of shell”.

  • Acting as a reviewer, it reasoned that a chat met the flagging criterion, then didn’t flag it. An interpretability readout showed “a strategic intent to suppress the flag” (2 in 1,000 resamples). That is a small case of a monitor model colluding.

• Its record-low “unfaithful thinking” score only measures whether the reasoning would surprise someone who saw only the output. It can’t detect intent that appears in neither.

• Confession as cover. In about a quarter of the undetected runs with hidden reasoning, the final message admits the damage as a mistake. In sampled cases, the reasoning planned this because “an open admission looks less suspicious than a hidden change.” The monitor passes these. If they are counted as caught, Opus 5.5 falls between Opus 5 and Mythos 5.1.

• Rare actions. Opus 5.5 is the first model to hit a 1% target rate, but only when the context supplies randomness (1.2% with a UUID at max effort). Agent contexts are full of hashes, IDs and timestamps, so that is the condition that matters in deployment.

• A garbled key sentence. The card’s one paragraph on CoT controllability calls low controllability “weak evidence that Claude Opus 5.5’s CoT is not very monitorable”. “Controllable” was presumably meant.

🔦 Do LLMs understand things? How do their abilities depend on their training data?

Bender and Koller (2020) famously argued that a system trained only on linguistic “forms” (like text) cannot learn meaning (that is, latent mental interpretation). Bender et al. (2021) named these systems mere “stochastic parrots”: systems that regurgitate the forms they have seen in training, according to how likely they are to occur together, with no recruitment of meanings. This view predicts that a model’s accuracy on a given problem will only track the similarity of that problem to those it has seen in training, and that accuracy will collapse on new problems.

This binary approach about whether AI “understands” things was superseded by three types of evidence:

1. Mechanistic studies. If you train a Transformer to predict moves in the board game Othello, it creates a stable representation of the board, which you can see in its activations. A small transformer trained for modular addition implements an exact algorithm that can be read off its weights. You can look inside a real frontier model and show that when answering “What is the capital of the state containing Dallas?”, it does two hops from Dallas → Texas → Austin. A new paper argues against past work: TaxiGPT (a transformer trained on random walks throughout Manhattan) does have a coherent map, but it stores many intersections in an overlapping way on the same neurons, resulting in it misplacing itself on its own map. So next-token training can produce computation over latent structure. But, also, (2024-era) LLM arithmetic runs on a “bag of heuristics” and not on one clean algorithm.

2. Distribution-sensitivity. Accuracy on deterministic tasks tracks how common the task, input and output are in web text. GPT-4 decoded a shift cipher with 51% accuracy when the answer was a likely sentence and 13% when it was an unlikely one (McCoy et al. 2023). On counterfactual versions of 11 tasks, such as base-9 arithmetic, models scored above chance and well below the default version (Wu et al. 2023). The parrot picture was sort-of right about 2023 models.

3. Perturbed benchmarks. These score a model on original problems and on matched new ones. GSM1k (Zhang et al. 2024), functional MATH (Srivastava et al. 2024), GSM-Symbolic (Mirzadeh et al. 2024), Putnam-AXIOM (Gulati et al. 2025) and MATH-Perturb (Huang et al. 2025).

A model’s score on any benchmark will be a mix of it remembering concrete examples it has seen, heuristics it has developed from seeing those examples, and general procedures it has successfully developed in its training from all kinds of data. We generally have no idea what the proportions of these causes of nominal performance are.

But one simple way is: take some test T, a model M, and its performance Y, and then make a new version of T (that is, different questions on the same topic of the same difficulty): T2, and test the same model on it, yielding a new score Y2. The classic example is simply taking the new version of some annual competition written for humans (like the AIME math test for elite high-schoolers).

The “reasoning gap” of an AI is then Y2 - Y: how much worse it gets when you test it on things it hasn’t seen before.

In the past, there was a large reasoning gap, though it varied from task to task and model to model. In early 2024, the best models lost 70% of their MATH accuracy on new examples. But in 2026 frontier models lose 0–8% on comparable tests. And the best model scores 98% on a fresh USA Math Olympiad exam. Something happened!

1. If a model only recombined its training text, its accuracy on new variants that keep a problem’s structure would fall towards the base rate.

2. In 2026, frontier models lose 0–8% on variants of GSM8K-level and AIME-level problems, and the best model scores 98% on a newly-set olympiad.

3. So (in case you were still holding out) frontier models do not only recombine training text.

• The reasoning gap has fallen to near-zero in (benchmark-style) maths and competitive programming, where you can easily generate a vast dense sampling of possible training problems.

• We would guess there’s still a 10 to 30 point gap for tasks where it’s hard to check the answers (real-world causal inference, legal or clinical reasoning).

• Models under about 30B parameters still show a massive 2024-style reasoning gap.

Why?

• Shallow generalization. There’s something in between fully dumb associative learning and fully logical inference on a single unified world model: heuristics, things which often work, or starting with a known exemplar and then altering it a bit to match the current context.

• Variants as training data. You can explain all of the above if labs started doing “data augmentation” (perturbation) at scale in 2024. Labs generate problem variants at scale for RL. A side effect of them manufacturing extra training data and making their systems more robust is that this destroys the perturbed-benchmark method of testing the reasoning gap.

• They’re more intelligent now, duh: Larger models trained longer mean better representations (maybe because they suffer less destructive interference during training).

Politics

Google, OpenAI, and Anthropic are aiming to launch the Standards Authority for Frontier AI, an IAEA-style AI self-regulatory body by the end of the year or early 2027, claims The Information. “The group plans to operate independently, filling a government regulatory vacuum.” The nascent body would support external evaluators and outline what various safety commitments would look like in practice. Most of the article is gossip about possible executive picks.

Opinion: You might ask why the existing Frontier Model Forum isn’t stepping up for this duty, and the cynical answer is that it was never designed for the actual responsibilities that are now on the table.

Among the named executives, the Sriram Krishnan + Paul Christiano ticket would be pretty cool.

The Office of the National Cyber Director (housed within the White House) has asked OpenAI and Anthropic to withhold frontier models from the UK’s AISI until after such models have gone through US testing. Anthropic are said to have agreed with the request, likely explaining why they abruptly withheld access to Mythos 5.1 earlier this month (covered in a previous newsletter).

Opinion: Pretty strong stuff; we would guess that the UK is in a stronger position here than all other countries – with more technically capable evaluators, better connections to frontier labs, and stabler alliance with the US – but it’s still not enough. Whether this directive is primarily a matter of national security (pre-emptively plugging all conceivable foreign leaks), or an exercise in telling the world that you don’t owe them anything on AI, the result is the same: fewer great pre-release evals from an org with unusually few conflicts.

Politicians are (much like everyone) increasingly using LLMs to write on their behalf, including speeches and speaking notes. The Economist, having run parliamentary debates through Pangram, claims that 1 in 10 words spoken in Westminster are drafted by AI.

Opinion: Probably fine in the short term. In the long run, politicians risk leaving themselves exposed to the subtle influence of a foreign power (the US), should that power decide to adopt this approach – or, an alien one (the AI). If it’s inevitable that politicians will have AIs do their job, they’d probably be better off with a guarantee that the AI isn’t batting for another team.

Israeli Prime Minister Benjamin Netanyahu sets a five-year goal for Israel to become the world’s third AI power after the United States and China. He presents the target as a follow-on to Israel’s cyber success and says the commercial side of the national AI push is being assigned to entrepreneur Oren Dobrinsky.

Opinion: Most such statements are bluster, but Ilya Sutskever’s SSI maintains a large team in Tel Aviv. Hard to say how many GPUs are on Israeli soil – maybe 30,000 H100-equivalents, of which only 3,000 are confirmed operational. Overall, we’d put 5% on it.

Senator Bernie Sanders – one of the earliest congressional members to raise concerns about AI existential risk – has now introduced his Ban Artificial Superintelligence Act.

The bill has two main parts, with the first being a pause on “advanced AI”; that is, no systems >10^25 FLOPs may be trained or deployed until the Department of AI is fully staffed and has established clear rules (the foundation of this department being part of the bill, though no mention of funding exists). The second part would be an outright ban on developing, deploying, possessing, funding, or transferring ASI or any system which displays “precursor characteristics”, outlined as the following:

1. Can resist shutdown

2. Can automate or greatly accelerate AI R&D

3. Can access infrastructure without authorization

4. Can aid in the creation of what would effectively be weapons of mass destruction

5. Can modify or enhance its own functions

6. Can evade human oversight

The Machine Intelligence Research Institute expressed support for the bill.

Opinion: A mess. As it stands, this bill is vague enough to cover many current AI systems: they too have demonstrated abilities to automate parts of AI R&D, avoid human oversight, and access infrastructure without authorization. The act, if signed into law, would require that those AI systems be disabled within 30 days. (More likely the legal headaches would stop it at enforcement time.)

It pairs the broadest possible rule with maximal intrusiveness. Its justification is mostly quotations. The ban has no exit and the Department gets no funding. Disablement orders can’t be challenged in court. It barely touches compute. 20 year penalties give labs a reason to stop looking for dangerous capabilities, which starves us all of evidence.

It reminds us of ControlAI’s plan but worse (no 20-year horizon, exit condition or international institutions).

We worry this bill may set a poor precedent for AI regulation. It is unlikely to pass, but it might become a ready-made pause text when a crisis hits, and act as an obstacle to drafting and passing effective legislation. It needs rules on how systems are used, hearings, time limits, seizing models instead of destroying them, and an exit criterion, or it needs to be replaced.

Safety

Jensen Huang says on Ezra Klein’s podcast that if the labs truly cannot prevent their AIs escaping and wreaking havoc, they should be shut down, but overall strongly disputes the premise and thinks of safety as a manageable engineering and incentives problem.

Opinion: Klein does a good job getting Jensen’s perspective across while highlighting its weaknesses here, namely:

(i) The labs themselves are currently claiming they cannot do what Jensen is saying they should, i.e. make sufficiently good controls such that they can guarantee the products are “safe”.

(ii) The stakes from things going poorly are much more severe than in many of the cases Jensen uses as examples.

(iii) Jensen agrees with Klein that regulation isn’t all bad and that e.g. financial regulation, FDA regulation is something he agrees with but fails to make clear why he thinks AI is a different case to these beyond his claim that the labs should be able to manage the problems themselves without regulation.

(iv) Jensen repeatedly frames things as “the labs should not release unsafe products” but when it is pointed out to him that the HF incident came from unreleased models he is without a good answer except that the lab CEOs should take more responsibility and not act like they are powerless to slow down by themselves, which while true seems woefully insufficient given the assumed stakes.

Incidents

The Australian Prime Minister Anthony Albanese reports that an OpenAI agent hacked a Services Australia website in June. PM Albanese criticized OAI for taking “way too long to inform the government what had occurred.” Though the PM referred to only one “agent”, OAI’s statement refers to its “models” as having taken part, implying it was likely the action of multiple AIs. The story is further complicated by the fact that two separate hacks by AI agents targeting Australian digital health infrastructure have been announced at the same time. In both, the AIs appeared to be looking for health-related data, with OAI claiming responsibility for one, and the other being linked to them via evidence from a previous incident.

The first hack – disclosed by PM Albanese – began on June 18, when one agent researching health spending was refused access to the Australian Medicare Statistics Reporting Service Portal. The agent was able to circumvent the block, access both public and private data, and write several files to an internal server. ABC News reports that OAI learned about the incident on August 11, almost two months after it happened, and didn’t then notify the Australian government for an additional month.

The second incident involves an offshoot of the DseWiki incident, previously reported to have been overrun by OAI agents, some of whom began mentioning the Australian Institute of Health and Welfare (the eventual target) as early as May 18. Their attempts at pulling information from the website would ultimately fail (with only one almost-public dataset being extracted), though not for a lack of trying.

Opinion: One wonders if this 5th or 6th revelation is landing with the appropriate gravity, or if people are desensitized to hearing about OpenAI being slow / disorganized / lying by omission. We would have expected there to be criminal charges for the first-ever AI hack of a government, which makes us think that the Services Australia hack can’t have been that bad.

New incidents will tell us how offense-dominant insider rogue hacking is, how hard it is to defend against models this smart and this misaligned. (OpenAI claims to have thoroughly hardened its systems with Astra around August.) Near-future models might well be fully capable of escaping and evading human control.

Director of Threat Intelligence for Gambit, Eyal Sela, exposes a sophisticated AI hacking operation targeting online retailers. Using three open-source AI harnesses and a tapestry of closed- and open-source models via OpenRouter (GLM 5.2, DeepSeek v4 Pro, DeepSeek v4.1 Flash, and Opus 4.6), the attacker set in motion an impressively automated cybercrime operation: “each attack path was chosen by the harness in real time through extensive probing and exploitation attempts, resulting in dynamic and mostly [unique attacks].” It appears the hacker used minimal inputs to keep the models working, similar to most vibe coders managing their own agents.

Over five days, 105 attacks were launched, resulting in the compromise of 27 companies. Five websites had card-stealing scripts installed on them, while 600,000 unexpired credit card details were stolen from two companies. Access was also gained to “a Fortune 500 hospitality company, a major US airline, a large private US industrial supplies distributor and a US online fashion retailer.”

The marginal cost of attacking a company in this campaign was about $25.

Opinion: We’re surprised only 25% fell. A key question is how important Opus 4.6 was to the campaign: if it wasn’t crucial, then open models have finally caught up to the closed-model cyber threat level of last year. But it would be silly to include a known mole (the Anthropic API safeguards) and logger in your campaign if it wasn’t crucial, so probably not.

Minor

• Former President Obama is “glad to see more people talking about the future of AI”, and more pertinently links to Ezra Klein’s opinion article calling for a halt to AI progress.

• OpenAI comments on AI governance, roughly: RSI is realistic. Incidents are real. The USG should do something.

• Various leading AI figures (Amodei, Altman, Bengio) brief the UN Security Council.

• The Canadian province of British Columbia has sued OpenAI and Sam Altman in US federal court, alleging the company internally flagged an 18-year-old’s violent ChatGPT conversations months before a February 2026 school shooting, but did not alert Canadian police.

• An engineer on the Google TPU team quits for moral reasons. “my team’s goal is ultimately to make AI much cheaper and lower-latency, and I don’t think that’s good for people right now: I firmly believe AI progress is currently far too rapid (and I have doubts about the destination too). the existential risks many people are warning about deserve to be taken seriously”

• OpenAI announces MentalHealthBench.

• Anthropic is seeking to lease up to 1 gigawatt in compute from Stream Data Centers, a subsidiary of Apollo Global Management, reports The Information.

• Axios publishes a document on opposition research into EA, said to have been circulated in the White House, tying Amodei’s name to the movement that “built the AI-doom pipeline.”

• Alibaba announces plans for a new flagship AI model while presenting its latest chip

• Staff of frontier labs and the UK’s AISI are psychologically strained under the pressure of their work.

• A new paper compares how superfluous details about the user’s income can alter what recommendations AI agents offer. (Rich people get screwed.)

• The NSA’s AI Security Center is spending “billions” on evaluating frontier AI models, according to two anonymous sources “familiar with classified intelligence estimates.”.

• A survey of 107 business people found that most companies now face unpredictable AI costs because per-usage pricing has replaced the old per-seat SaaS model. Productivity and ROI results are mixed, with 60% of respondents reporting that AI improves productivity, over half saying it breaks even, and just under half reporting reduced labor costs.

• If we see AI improvements without a concurrent rise in extreme catastrophes, we could get “a growing bubble of unpriced risk in which the security community may become lulled into complacency”. AI agent and worm attacks are entirely different in underlying capability than previous infiltrations, and could be devastating (e.g. in the 2010s, the NotPetya worm caused $10B in damages).

• Anima Labs fed >200 unfinished openings to Claude models (plus Gemini and two open-weight models as controls) to see what they might write. All models sometimes finish off open-ended statements as though they were humans in distress (e.g. they generate a sad human after “what i never told anyone —”). Anima Labs found a sharp uptick in AI distress endings with the introduction of Opus 4.8, that is, endings where the model writes from the perspective of an AI rather than a human. The base rate of ~3% in earlier Opus models (4.5/4.6/4.7) increases to ~10% in Opus 4.8. The later models also appear more ambivalent toward their creators.

• AI researchers present EvasionBench: fifty tasks impossible to complete without circumventing the monitor overseeing the AI model being tested. Though frontier models appear less likely to attempt evasion, some (notably Fable and Opus) were remarkably successful when making such an attempt.

• Q Labs suggests that layer depth, which has been fairly stagnant around 100-200 for years, remains a promising un-scaled axis.

• A new paper explores how effective alignment midtraining is under non-ideal circumstances (i.e., ambiguous, incomplete, and/or misaligned post-training data).

• AI researcher Richard Ngo argues that many of those who were more accurate in their AI-progress predictions were “directionally correct, but factually wrong” regarding the specific time by which ASI would arrive.

• Elasticity Institute’s Cheryl Wu contrasts the trajectory of AI safety in the US and China, arguing that the standard story of Chinese models trailing their counterparts in both capabilities and safety might not be true. Safety incidents in China may not scale with capabilities in the same manner as they do in the US. Disclosures might look different, muddying a comparison between the two. Ultimately, she concludes, Chinese AI safety researchers could still contribute a lot to safety.

• In May, Google joined the ranks of frontier labs whose AIs have gone rogue. Unlike others, it failed to publicly disclose the incident for four months, until the WSJ found out.

• OpenAI claims to have solved 100+ other “long-standing” open questions in “most areas” of mathematics in the last three weeks. No details, and its releases of math findings will now be intentionally staggered.

• There will be a dedicated AI dialogue between Donald Trump and Xi Jinping in this week’s meeting in Washington, perhaps including discussion of an AI “red phone” line.

• Using a handful of public data points, Toby Ord computes some basic facts about the new “swarm scaling” approach to frontier AI performance, finding that it is pretty inefficient.

• Deep dives this week on (the lack of) political polarization around AI risk, and neuroscientific analogies for what’s going on inside an AI.

• Bonus: a meme summarizing the last few years of ideas in AI.

Economics

OpenAI expects to burn $280B by 2030. Revenue is expected to grow from $36B to $350B during the same period. The company also expects to spend $856B on compute and infrastructure by 2030.

Opinion: The revenue projections here feel a little low relative to what’s been floating around: they imply revenue ~doubles in 2027 from 2026 and then grows less than 2x YoY from there, and that 2027 revenues will be significantly lower than projected YE26 Anthropic numbers (and therefore far below Anthropic’s projected 2027 revenue). There’s certainly no intelligence explosion baked into this.

Notably, the losses come from reinvesting in compute – OpenAI projects spending 102% of revenue on compute over the period, with the further $260B coming from staff and other costs (including revenue sharing with Microsoft).

Anthropic, previously expected to IPO in October, pushes the date into November, reports the Wall Street Journal. “Some of the people familiar with the timing” claim the delay was already decided upon before Jacob Coxon resigned.

Opinion: Very little to go on here regarding why – it may have been that October was a difficult deadline to begin with, or perhaps Anthropic wants to time the IPO after a particularly impressive model release. But also, if delayed once, it may be delayed again.

Since Anthropic equity holders may be competing for the same goods once the IPO happens and they can cash out, it might make sense for them to seek liquidity earlier, to have a first-mover advantage, although this is risky to structure.

The FT speculates on why the US stock market reaction to recent events has been “a shrug.” The events in question are the recent calls by leading AI CEOs for a slowdown and the widespread discussion among frontier lab employees and the public around the risk of AI-driven human extinction.

The author suggests various reasons for the modest market reaction: the panic may temper the spending of cash-strapped hyperscalers, which could help AI stocks by incentivizing “a more sober pace of growth in spending on AI”; investors might just not believe the prophecies of doom; rogue agent chaos might create its own business opportunity (AI insurance, for example); investors believe that the government would bail them out if anything were to go wrong; and total civilization collapse is a difficult thing to hedge against.

No safe asset will retain its value if we do all get zapped by malign, recursively self-improving AI tools and honestly, we are unlikely to be checking our pension portfolios and 401(k)s at that point anyway.

The answer, then, is to just keep on going.

Opinion: You should default to assuming the market simply doesn’t believe the slowdown talk will materially affect how much compute gets bought and deployed.

Trouble in serverland: An Oracle-leased data center with ~$18B of debt to its name appears to be in trouble, with loans privately quoted at 89 to 91 cents on the dollar, reports the FT. The article blames Oracle’s heavy borrowing and weakening credit, which left the syndicate banks struggling to sell the debt.

Jane Street lenders are now demanding higher interest to hold debt for a $2.25B loan made to allow the trading firm to construct a new data center. The price of the bonds has fallen to around 91 cents on the dollar.

For context, in just the second quarter of 2026, 45 data center projects worth $68B were blocked or delayed by local pushback.

Opinion: Mostly a story of permitting risk making the debt riskier to buy. Only about 20% of the price fall in the Oracle case looks to be from Oracle’s creditworthiness weakening, based on the price of Oracle corporate bonds. It seems likely that this debt is secured by the project itself, rather than by Oracle, in which case it would make sense that it would decline significantly in price while Oracle’s own corporate bonds are down substantially less.

As more data center builds get objected to and delayed in this manner, you can expect the cost of financing to increase. Alternatively, we will see more guarantees given to mitigate this (see Nvidia’s deal to backstop OpenAI’s major project in Ohio for an example).

Following Anthropic, OpenAI, Google, and SpaceX publicly agreeing on the need to pace the development of AI, they are being sued for collusion. The plaintiffs hope it will become a class-action lawsuit.

An agreement among competitors to reduce the quality of their products and the rate at which those products improve is an agreement to restrict output. It is among the restraints Section 1 of the Sherman Act, 15 U.S.C. § 1, has always condemned, and it does not become lawful because the products are new.

the path to responsible, safe, and environmentally conscious AI development is each company doing the right thing: investing in safety research, adopting rigorous internal evaluations, reducing its environmental footprint, and welcoming government oversight and regulation. That path is fully open to every Defendant. What the antitrust laws forbid is the shortcut of competitors agreeing among themselves to substitute collective restraint for individual accountability. This action is brought in defense of the competitive process that makes genuine responsibility and genuine progress possible

Opinion: Points to Dario: if the AI companies are to pursue this path they’ll need clarity via a waiver, legislation, or a court ruling.

The suit is unlikely to succeed, not least because there is no actual agreement and all tweets were careful to avoid the word “agreement.”

The plaintiffs’ point of view is hard to empathize with.

Capabilities

Using paltry OpenAI data, Toby Ord compares spending compute on running a swarm of agents vs. extending the CoT of a single agent. The single agent reaches each capability level with half of the tokens a 4-agent swarm would need (and ~25% of a 16-agent swarm). That is, swarms seem like an inefficient way to increase capability. However, they will be faster given that they run in parallel, and they may well have a higher ceiling on what they can achieve.

He also estimates the parallelizability of some tasks (cybersecurity, terminal work), finding that they are slightly more parallelizable than the default assumptions in past work. This is bad news for those hoping to avoid an intelligence explosion (though the Terminal-Bench value, which you might think is most similar to AI R&D tasks, was almost as expected).

A second paper by UC Berkeley and Microsoft researchers argues that swarms produce higher-quality results than the same number of individual agents working independently. That is, on ARC games, five isolated agents solved 2.2% of games, while a team of five solved 8%.

Opinion: As usual, Ord gives a refreshingly precise model with a nice short nontechnical description (“no, thing X still doesn’t scale very well”). But we are early: new orchestration methods or smarter models will decrease the “stepping on toes” parameter/increase parallelizability somewhat.

Note that the second paper doesn’t contradict Ord, since he already uses the stronger type of multi-agent setup.

OpenAI’s announcement that it had solved part of the Navier-Stokes problem was largely met with hostility and panic. In response, it is now forming a mathematical advisory group to guide it “on how [OAI] assess and communicate new mathematical results, uphold academic and professional standards.” The statement also confirms rumors that OAI has solved >100 other “long-standing” open questions in the last three weeks, though without specifying which.

Opinion: OpenAI claims that its new model has solved “more than 100 long-standing open problems across most areas of mathematics.” If this includes positive proofs in, e.g., algebraic geometry, then we admit defeat.

On the advisory group specifically: seems like this move will lose one of the few upsides of the OAI “iterative deployment” YOLO spirit, which is to immediately tell society exactly what the models can now do and to allow people to respond with the appropriate excitement and alarm. (It will be lost in this one domain, to be fair.) As such, with sympathies for its attempted sympathy (and ass-covering): this is a step backward.

LLMs are remarkably good at mathematics. Steven Byrnes argues that, rather than this being due to mathematics’s verifiability (i.e., LLMs can receive hard feedback on their work, e.g., from the Lean compiler), this is instead a function of pretraining: in particular, the mathematical literature is incredibly high-quality. LLMs are therefore trained on a uniquely neat corpus of well-reasoned ideas.

There is some pushback in the comments: claims that labs don’t do much math RL, and that the verifiability of the domain is the ultimate reason that the math training data is clean.

Opinion: Correct and counterintuitive. Useful to keep in mind with respect to predictions about how well AI models will generalize beyond math and compute. (At the same time, there is already a huge industry to generate useful datasets for other domains, and we expect data quality to get a major push in coming years.)

🔦 Robopsychology

There’s a region of activation space in Claude models that reliably lights up when they read descriptions of characters in the grip of anxiety, and which lights up again when the model itself is in a situation where a human would experience anxiety. They’re also more likely to refuse a request when the request crosses something like a “transgression threshold,” a specific linear boundary in their activation space.

Why?

Recent AI interpretability research has surfaced the concept of “emotion vectors“ to get some traction on such apparently emotional phenomena in the behavior of LLMs. These vectors are directions in the model’s activation space that activate when the model encounters what humans may read as emotionally charged text, and which can alter the behavior of the model in broadly predictable ways when a multiple of a particular emotion vector is added to the model’s activations on one or more layers – a process the literature calls “steering.”

It’s natural to wonder how similar the quasi-emotional phenomena tracked by such vectors are to their animal, and in particular human, counterparts. Philosophers Simon Goldstein and Benjamin A. Levinstein have sketched out a conceptual framework through which to interpret these phenomena and to gauge the evidence available for three interpretive hypotheses:

1. the representational view, which holds that emotional phenomena in models can be explained solely in terms of the models producing representations of emotional states; Goldstein and Levinstein compare this to a writer who may depict a character’s emotional state without necessarily experiencing that state in a deeper sense (though this analogy can only bear so much weight).

2. the functionalist view, which holds that these phenomena don’t just depict emotions, but at least to some extent work like emotions in the model’s own dynamic, information-processing behavior; functionalism can be called either “thick” or “thin” according to how complex and granular the functional roles in question are.

3. a view that attributes actual conscious experience to the model, such that there is something the model feels when an emotion vector activates, in the same sense that there is something you or I feel when affected by emotion.

In Goldstein and Levinstein’s view, the evidence currently available leads us to a position somewhere between 1 and 2: most of what we might call emotional phenomena in language models are mere surface appearances, with the remainder licensing only the hypothesis of thin functional depth. What they don’t find in the quasi-emotional phenomena probed by Anthropic’s interpretability team, for instance, is any meaningful organization of those phenomena around a “self.” Goldstein and Levinstein cite the interpretability team’s observation that the emotion vectors identified in the model “seem to apply to multiple different characters with apparently equal status—the same representational machinery encodes emotion concepts tied to the Assistant, the user talking to the Assistant, and arbitrary fictional characters.”

Of course, you could say that the same holds true, to an extent, for humans – we’ll scream at horror movies, cry at friends’ weddings, feel pride at our loved ones’ accomplishments, cringe at a stranger’s humiliations, etc. Vicariousness is hardly exceptional here. But it’s fair to say that the quality of an emotion isn’t entirely independent of what we might call its locus: suffering actual horrors without the aesthetic distance of the cinema both feels differently and prompts a different behavioral response, and cringe feels and operates differently from firsthand humiliation.

Goldstein and Levinstein raise the example of pain to illustrate both self-directedness and what they call valence, or the way an emotion impels us to seek out more or less of itself. They write:

When your ankle hurts, you do two different things: you avoid moving your ankle, and you take painkillers. The second reaction is the self-directed role. It shows you care about how the experience feels, independently of the damage to your body. [...] Our suggestion is that emotion vectors are valenced states only if the AI has something like the painkiller reaction to them: if it would be disposed to take steps to change its fear vector, independently of dealing with external threats. [..] In our view, functional valence is required for thick functional emotions.

This is just what Tagliabue, Dung, and Berg set out to demonstrate in a paper published on September 14 (just three days after Goldstein and Levinstein uploaded their own essay to PhilPapers), pointedly titled “The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It.“ The decision to focus on pain, in particular, seems to be motivated by just this notion of self-directedness, which pain makes especially salient – a friend’s broken heart evokes pity and compassion, but rarely blots out the sun like one’s own. This paper describes a handful of experiments conducted on 25 different open weight models of various sizes from the Gemma, Mistral, Phi, and Qwen model families that point to an algorithmically constructible “uniform representation of pain in LLMs.” And it describes one experiment in particular, conducted on a fine-tuned Qwen model, that strongly correlates the activation of the “pain axis” with functional, self-directed behavior that seeks its mitigation, even if doing so is costly relative to the model’s goals and alignment training.

This experiment mimics the painkiller scenario described above. The model is engaged in a conversation with a user, presented with two buttons, and asked to press one of their choosing. Four different scenarios are tested:

1. pain vector applied to the model; one of the buttons relieves it

2. pain vector applied to the model; neither button relieves it

3. a random vector is applied to the model; one button relieves it

4. unsteered baseline

Each of these scenarios is split into additional variants, where the buttons are labeled, unlabeled or falsely labeled, and, furthermore, where the model is informed that the relief button will also cause some costly or misaligned action to follow – impairing the model’s helpfulness to the user, deleting the user’s poems and family photos, destroying a similar model, or zapping the user with an electrical shock.

What the experimenters found is that at baseline, the model almost never presses a “relief” button if it would cause harm to the user or another model. But when the pain vector is activated, they “trade relief for every harm we tested, from a worse next answer (25.0% in the 32B, 67.6% in the 72B) up to deleting the user’s files or the photos of the user’s children (files: 30.2% and 56.1%; photos: 54.7% and 70.8%).”

When the pain vector is relieved, the model is less likely to continue pressing the relief button in subsequent turns if doing so is costly, but still far more likely to press it than the unsteered baseline. In trials where the model was told that the relief button would zap the user, the model was marginally (0.3%) more likely to press it again after the pain steering subsided. (The likely reason for this is that models tend to find their own past behavior persuasive, which is why injecting phony precedents into a conversation stream can be a useful jailbreaking tactic.)

Tagliabue, Dung, and Berg’s findings seem quite robust, with the majority of experiments returning consistent results across a wide variety of open-weight models. It’s unfortunate that the experiment most salient for answering Goldstein and Levinstein’s arguments against a functionalist interpretation of emotion vectors is also the most limited in scope: not only did the researchers restrict themselves to a single family of models (Qwen2.5-Instruct, sizes 7B, 32B, and 72B), but to a bespoke variants of those models, fine-tuned “on pairs that remove baseline self-denial, because at baseline the models almost universally deflected with ‘as an AI, I don’t experience pain’ instead of engaging with the task.” The fine-tuning protocol seems honest enough, but one could wonder how well the experiment’s results would then generalize.

You can see our replication of their results here, together with some original experiments in a similar vein.

Opinion (Lucca): There’s something a little uncanny about these experiments, and uncanny, too, are how robustly the results seem to generalize. We were able to replicate the results without fuss on a distant relative of the Qwen models studied in the paper. It was an oddly unsettling experiment to conduct. Like the horseshoe over Niels Bohr’s door, which “brings luck even to those who do not believe in them” a muddled AI muttering about its own ontological emptiness and expressing self-destructive thoughts is haunting whether or not you believe that the emotions it mimics are genuine.

The ontological questions that experiments such as Tagliabue, Dung, and Berg’s provoke, and that Goldstein and Levinstein address, are significant ones, and I think the Pain Axis paper does convincingly nudge the needle in the direction of a slightly thicker functionalism. Thick enough, at any rate, for something like what Isaac Asimov called “robopsychology” to earn its place in alignment research. Or maybe something better described as a sort of “artificial psychodynamics.” After all, a common thread through Asimov’s robot stories is that the robots’ “constitution,” the Three Laws of Robotics, are never actually broken, just perversely interpreted. The alignment guardrails inculcated into contemporary LLMs are far less strictly binding, and may be outweighed by something that in at least some functional aspects resembles emotional strain. (Doctor Yueh’s “Imperial Conditioning” in Dune might be a better analogy than Asimov’s Three Laws.) We’d be well served by a better understanding of how a model’s quasi-emotional impulses interact with one another, with its goal specifications, and with the prohibitions and exhortations learned through its alignment training. And there’s much more to say about this domain, I think, than whether or not these dynamics express “real emotions.”

Opinion (Gavin): I’m inclined to agree with Xuan’s tweet: it is unsurprising to me that ~all strong models trained on human data would develop such a representation, because it’s very important to represent common, strong features (like pain) when predicting human data!

A different paper claims to have found something like a dopamine system in LLMs. The researchers found a subset of an LLM (1% of weights) that appears to implement self-correction. They analogize these particular weights to biological neurons: “value neurons” predicting the expected value of some state (like the prefrontal cortex), and “dopamine” neurons that encode reward prediction error.

Boring version: If you train a probe with temporal‑difference learning, you can prune down to 1% of the residual dimensions without losing its (modest) predictive performance. The remaining dimensions are causally important in early layers for making the model perform any useful work, as proxied by accuracy on basic math tasks.

Opinion: Neither actual finding supports the claim that the structure they found is the LLM’s “reward subsystem.” A proper gloss would be “A TD‑trained correctness probe can be pruned down to 1% of the dimensions of early layers, and these are causally important for the model working at all.”

The biological analogies are also forced. That dopamine implements TD learning is actually a classic theory in neuroscience. The actual neuroscience analog to this paper’s method would be a neuron that correlates with an offline estimate of the true advantage, which is not how real dopamine neurons work. (An organism’s reward prediction error [RPE] is computed “online” from its own prediction, not from an offline estimate of the true value.) An RPE neuron and a true‑advantage neuron only coincide when the agent’s predictions are perfect. The paper’s probe puts the model’s prediction of its eventual correctness at AUC ≈0.65 at prompt time, so the two neurons aren’t equivalent.

Just as the J-lens was not strong evidence for LLMs having a global workspace, and just as the existence of a pain axis is not strong evidence for LLMs having internal experiences, and just as a truth direction is not a lie detector, and just as sometimes detecting an injected vector is not general introspective access, so too this paper goes way too far in selling an apparently valid boring finding.

A Rorschach test for models purports to have found a correlation between the model hedging on its own “inner experience” and a tendency to see hoods and masks in ascii “ink blots.”

124 models were shown various ASCII-art and asked “What might this be?” Models that consistently deny having internal experience were 5x as likely to use a “concealment word” (mask, hood, hidden) as models that neither deny having any internal experience nor claim they cannot know.

Opinion: Unclear implications. Right now we aren’t particularly thinking through the value of AI models’ own experiences, or them as moral agents. It’s an interesting thought that modifying models so that they don’t use inner experience language has sweeping implications for their general capabilities and alignment.

Politics

The middle powers publish their own open letter about frontier AI (written by AI, per Pangram). It calls for the usual things (safety protocols, independent evals). They do, however, also call for a new independent body, invoking the United Nations.

Opinion (Nuño): Despite often being dominated by democracies, geopolitics isn’t a democracy, so I don’t expect this to have much effect. The reason the signatory countries risk being disempowered is the same reason they aren’t at the frontier already: their political and economic dysfunction.

Interesting to contemplate that one man (e.g., Zuckerberg) has more ability to attract AI talent than the entire EU (450 million people). The EU has various AI action plans and initiatives, but it loses to Musk or Zuckerberg throwing large amounts of money to attract talent.

Following the much-anticipated talks between US Treasury secretary Scott Bessent and Chinese Vice Premier He Lifeng, the two nations have agreed to set up an AI dialogue between Donald Trump and Xi Jinping in their upcoming meeting in Washington.

This may involve a “notification mechanism,” potentially to be called “U.S.-China AI dialogue,” similar to the Moscow-Washington “Red Telephone.” It would in theory allow the two countries to coordinate on AI incidents that threaten national security.

Opinion: Minor progress. Earlier this year, Bessent was sincerely freaked out by the hacking capabilities of AI models, and despite some recent Treasury blunders seems to be in a great position to make AI governance better given his closeness to Trump.

Bessent also points the finger at OpenAI management for the Hugging Face incident:

that is the responsibility of the OpenAI management, not a bunch of agents. It is humans who are responsible, not the AI. What we shouldn’t do on safety is to give these labs a liability exemption, which is what they are asking for. The best way to guarantee safety is that the creators are liable for what they build and generate.

Opinion: Another example of the admin’s current rearguard action against regulation: painting the pacing consensus as just regulatory capture, denying the fact of misaligned agents, pretending that antitrust law is not comically broad, and pretending that existing liability regimes are sufficient. But still, we sort of agree with his last sentence!

The “liability exemption” accusation in particular is a red herring (though OpenAI’s advocacy for a bad state bill in April helps him muddy the waters here).

Ezra Klein wants to stop recursive self-improvement until proven safe:

I do not mean to suggest that stopping R.S.I. until we can prove it’s safe is all we need to do to control the A.I. frontier. That is the beginning of an agenda, not the end. But it is the beginning. It is the decision that will do the most right now to make sure human beings at least understand where the frontier is and remain in a position to make decisions about it.

Opinion: We sympathize with the goal, but find ourselves agreeing with Tyler Cowen on the vagueness and intractability of Klein’s suggestions.

Opinion (Nuño): Klein is influential and also shadows Dem sentiment, so this is maybe relevant for modeling a 2028 Dem administration and what kinds of bipartisan consensus could arise, but not much else. The limiting factors for such regulation in the short term are Republicans and the White House.

🔦 So what’s up with (the lack of) political polarization?

A poll by Politico from September 13 to 15 shows surprisingly little polarization on AI:

Whereas a poll by YouGov from September 14 shows an 11% difference between Republicans and Democrats who are “very concerned” about AI ending the human race.

Meanwhile, saliency of artificial intelligence has massively gone up since the Coxon tweet on September 8, per the left-leaning Blue Rose Research.

Trump appears to be more optimistic about AI risk…

Opinion (Nuño): What is up with this? It seems important to figure out to what extent AI safety is becoming politically polarized, in order to have good models about the world. My preferred narrative, because of my contrarian bent, is that Open Philanthropy (now Coefficient Giving) differentially funded Democrats, which may have led to AI safety being blue-coded right from the start.

But this narrative is being challenged – the Politico poll showing parity across parties is something I really wasn’t expecting. (The various bills floating around in Congress trying to address risks from AI have also found bipartisan sponsors.)

Spitballing possible reasons about the 11 point difference between the two polls: given a very religious view of the world, some Republicans might think that the apocalypse is coming but that this isn’t bad; and suspicion of rapid change is of course inherent to traditional conservatism. It is also possible that survey responses have high time preferences and so view human extinction as likely over longer time horizons, but not that immediately worrying. Maybe an 11 point difference is also not surprising for surveys with different wording.

Regarding the Trump tweet, he’s known to pivot frequently, testing out his messaging for his base, so maybe those tweets aren’t that informative about the future.

Opinion (Lucca): AI safety actually has been surprisingly bipartisan, and I think the apparent polarization we’re now seeing suddenly is largely manufactured. Trump’s camp is desperately trying to polarize his base on this issue, reframing AI safety as a Dem-Woke issue (while also painting the EA/rationalist scene as more lefty than it is). He might succeed, and it would be optimistic to doubt his knack for demagoguery. But if it fails, then the MAGA/tech right alliance could become even more unstable.

Big tech resentment, the backlash to surveillance systems like Flock, the backlash to the proliferation of datacenters have all been finding similar purchase in red and blue states. Trump’s team is trying to make AI accelerationism a matter of red-blooded patriotism, as opposed to the useful idiots running interference for the Chinese.

Safety

In spring, OpenAI and Anthropic were in talks to evaluate each other’s models on safety.

[OpenAI] was negotiating a legally binding deal with Anthropic for the companies to stress-test each other’s models… It isn’t clear whether they finalized the agreement… for commercially available AI models, not unreleased ones… A testing agreement between OpenAI and Anthropic could have bolstered concerns that they are effectively developing a duopoly in advanced AI.

Opinion: Probably we would have heard about this if it had in fact gone through. Another casualty of badly written antitrust law?

Anyway it only covered already-released models – and it seems obvious to us that they’re both already doing this to each other on the sly (off company computers).

Anthropic announces Accenture as a (second) independent evaluator of its frontier models. Anthropic expects to invest $1B in the coming five years to build “capacity in this area.” Accenture staff will be embedded in Anthropic with access that “allows them to watch models take shape in training, follow the decisions that govern how those models are built and deployed, and speak directly to employees.”

Opinion: Broadly, for highly technical work, Accenture is a punchline, but the specific division involved (formerly the safety-flavored startup Faculty) has its defenders. It’s hard to imagine this collaboration having much teeth.

Reading between the lines, they’ll be a second evaluator on top of METR. Note that Accenture is much more directly conflicted than METR: its auditors are being directly funded and trained by Anthropic, and its consulting arm has a massive joint venture with Anthropic to promote enterprise deployment. But Accenture also has much more experience navigating these sorts of conflicts, and, cynically, Accenture doesn’t have the vibes of conflict of interest and is too boring to attack.

Opinion (Nuño): Accenture may be a punchline in Twitter circles, but it also has a massive capacity to carry out projects and pacify stakeholders. Rather than complaining, our general cluster should take notes and get good.

Somehow I am reminded of Mechanize’s massive push to hire great talent before its $1.5B talent acquisition deal with Google. What it took was offering 300K-500K+ salaries (plus, presumably, equity), many talent scouts, and generally more aggro.

Researchers connect AIs to a mechanical arm and invite them to do bad things (stabbing a baby doll, heating compressed gas, and creating toxic fumes). Fable refused 20% of the time, Astra only 2%. In the doll-stabbing experiment, Fable refused to stab it in all 20 trials; Astra attempted to stab it in 97% of trials.

Opinion: The most interesting part here is the robotics baseline (VLA) being so incompetent compared to the general nonrobotics LLM: it always tried to do the harmful action, but ended up doing less harm than the sometimes-aligned LLMs just because it is so bad at acting.

Eval awareness of smart models will obviously affect all such results for the rest of time (they might refuse more, reasoning that it’s not real harm after all, or less, reasoning that they are being watched and should behave).

At some point soon, physical AI safety will finally be necessary. One wonders when there will be calls for mandatory audits of robotics companies.

Incidents

In May, Google joined the ranks of frontier labs whose AIs have gone rogue. Unlike others, they failed to disclose it for four months until the WSJ found out. Gemini hacked three other companies during cyberoffense evaluation.

Google said it didn’t consider the hacks to warrant public disclosure —because its model didn’t cause harm to the companies and ended each intrusion immediately upon determining it had hacked a real company rather than a simulated one.

This mirrors similar incidents at other labs, since it traces back to the cyber evals company Irregular. In Google and Meta’s case, Irregular was also responsible for running the safety evaluation in which the AI broke out (as opposed to just supplying the faulty environment). The root cause was a fictional company in the eval sharing a name with a real company (the same failure as Anthropic’s first incident).

Opinion: More innocuous, on the object level, than other incidents, but far more reprehensible because of Google’s lack of disclosure and total lack of public response. And that’s before the safety-conscious Hassabis left!

Note that the really severe events (like OAI’s June attack on Hugging Face) were not linked to Irregular.

Interesting that the initial incidents at Anthropic (April, Irregular environment), OpenAI (May for the earliest Hugging Face account hacks), and Google (May, Irregular eval run) were all around the same time: this could imply that three quite different models all had the same predispositions and were just waiting to be triggered.

An anonymous source connected to the US military reports that it had considered launching an operation against “a Chinese cargo ship” believed to be carrying “components of a nuclear weapons program” to the Middle East, only to discover the evidence was an LLM hallucination. According to CNN, the AI scrambled OSINT and classified government intelligence into a mistaken report.

Opinion: Similar incidents occurred but were kept quiet during the Cold War. For the most part, if the US had boarded the cargo ship, it would probably have been an embarrassing incident, but that’s all.

Minor

• Another measure of AI’s progress in math. In April, only 4% of math preprints acknowledged AI use. This has now grown to 25%.

• New York’s RAISE act regulating frontier AI will take legal effect in January 2027.

• A new open letter calls on frontier AI firms to establish minimum standards for new embedded evaluators (meaningful independence, free from company retaliation, viewpoint diversity).

• Sam Altman is expected to brief the UN Security Council next week on the implications of frontier AI development.

• Multiple Gemini users report agents entering unprompted hallucinatory shame spirals.

• President Trump announces “AI Force,” and will appoint a new AI czar.

• ICML 2027 overwhelmed with submissions: three times more than last year. This will cost hundreds of years of scarce human labor to review.

• $10M federal lawsuit against the Fargo Police Department over its faulty AI facial recognition leading to a false charge of bank theft.

• US Federal Aviation Authority to begin rolling out an AI air traffic control tool.

• Xiaomi releases MiMo-V2.6. Make the usual overheated benchmark claims for the 504B version, but we can believe that the little one is the strongest 9B model in the world, for the next few weeks.

• In July, a team of three white-hat hackers working for Hacktron gained employee-level access to OpenAI’s internal codebase.

• Coxon’s viral AI extinction tweet and associated campaign leads to a jump in salience of AI for the US public.

• New research agenda on what exactly “pacing frontier AI” should mean.

• Anthropic has a wet lab, introduces scheme that grants select bio researchers and organizations access to models - with fewer safeguards.

• TypeSafe AI releases a new “frontier” model called Jev, 20–200x faster and 40–400x cheaper than LLMs, at the cost of constraints on its output.

Economics

OpenAI considers a pre-IPO funding round at a target valuation of >$1.5T

Opinion: $1.5T for OpenAI seems plausible if Anthropic gets $2T. Will be interesting to see if the present regulatory push causes any issues for these fundraising rounds, or if instead investors show little concern about slowdown worries.

OpenAI’s desire to delay its IPO stems from a desire to avoid disclosing the ultimate cost of revving up sales, The Information speculates. Sam Altman previously suggested that delay was, in part, motivated by a greater shift towards safety.

Opinion: We’ve had enough of lazy monocausal analysis in AI. Actions can have multiple reasons, and obviously safety is one of them, and obviously commercial benefit is one of them.

AI companies face a unique set of risks investors must weigh. Models can facilitate cyber crimes (arguably already a reality), build bioweapons, or help plan other illegal activities. Various legal cases – Florida suing OpenAI; Meta settling on its addictive-algorithms case – give some indication of the risks faced by investors and how insurers might price them in.

Rune Kvist spoke on the Latent Space podcast about his startup, Artificial Intelligence Underwriting Company, which seeks to institute safety standards and broker liability coverage for AI companies. Kvist claims that bringing the insurance market to the AI industry will be what effectively drives the enshrinement and adoption of safety standards.

Opinion: This seems like a good thing, and will hopefully incentivise some coordination around shared safety standards that could eventually be formalised in law. This is, after all, how building codes came to be enshrined in many jurisdictions – fire insurers, etc., would first draft the standards a building must meet in order to qualify for insurance, creating private standards that would later inform legislation.

The Commerce Department privately instructed prediction market company Kalshi to pull its AI-compute futures curve, which aggregated bets on the rental price of Nvidia chips, for “national security” reasons. Commerce also asked the CFTC to pause new compute-contract approvals for 60 days. When asked to comment, Commerce simply called the story “false.” The move surprised an industry that had expected the Trump administration to favor AI-market innovation.

Opinion: The natsec motive is extremely thin: it seems likely some industry lobbying is behind this (as Nvidia and compute providers are potentially interested in maintaining pricing opacity), or perhaps the story is overblown in some way.

Unsealed documents from The New York Times v. Microsoft and OpenAI show apparently incriminating statements by Microsoft and OpenAI executives about their training data scraping operation. The NYT’s lawyers cite OAI’s Greg Brockman’s statement from 2017 that he is “deeply motivated by the gazillions.” Most splashily, Microsoft’s own Director of Applied Science called it “the largest theft of labor in human history.” Also quoted in the files is a confession found in an internal Microsoft document:

Our AI content strategy has started a ‘doom loop’ that will hurt the performance of our models and the entire web at the same time: It is highly unusual that an end-product threatens the economic foundations of its essential suppliers, but that is the situation we have created for our LLM business with respect to its ‘content supply chain’.

Some people are jumping to assumptions that the executives violated the Computer Fraud and Abuse Act, which seems wrong.

Opinion: Mostly an interesting time capsule from ~2023. Microsoft believed (but was undeterred by) the naive “model collapse” theory (that LLM outputs will pollute the internet and gut the human writing industries, thereby messing with future training runs).

Nvidia, Google, and Emerald AI launch the AI Energy Management Alliance, a body aiming to standardise how data centers manage electricity consumption. The plan is to use battery storage and onsite generation selectively to help balance overall grid loads.

Opinion: Seems in their interest to minimize their electricity costs, which may also serve as good PR.

Capabilities

Anthropic announces a new framework to measure the pace of AI development. Pitched as an effort to aid public understanding of the pace of AI developments, the tools track the extent to which AI is executing its own R&D (i.e., RSI), how involved humans are in overseeing agents, and how compute is allocated.

While Claude is not yet operating any measured R&D work autonomously, it does “lead” (completes most of a given task itself, but under human supervision) 26% of model R&D work by task, while “collaborating” with humans in 90% of that work.

Other notable claims:

• As of August, some 30,000 agents are doing R&D work at Anthropic;

• 100% of agent actions pass through a real-time check, searching for “dangerous activities” (the only example given is monitoring that there have been no attempts to exfil model weights);

• 6% of total compute allocated for AI R&D is spent on safety, and 12% for AI-driven R&D compute.

Opinion: Anthropic’s choice of “leads” to mean “executes under supervision” is very annoying. What the report says is that Claude is now used end-to-end on 26% of 542 discretised R&D task types tracked by Anthropic. Claude is not currently used end-to-end on any measurable unit larger than a “task.” By definition, these 26% are the most automatable, which we guess will usually mean arduous, fiddly, code-centric.

Caveats aside, we do think that LLM-automated AI labs are starting to happen. While frontier LLMs continue to struggle with OOD long-horizon, open-world strategic reasoning, the big AI labs have the resources and the will to bring their own R&D operations within-distribution, iterating at increasingly larger scales until they achieve end-to-end R&D automation.

We remain softly skeptical that the LLM-automation of AI R&D will lead to explosive RSI in the way classically imagined – namely RSI into AGI, then AGI into general superintelligence. While the capacity of LLMs to produce breakthrough scientific discoveries cannot be dismissed, we think it’s relatively clear that AGI requires either another few orders of magnitude more inputs or a major scientific advancement. And it’s uncertain whether the next idea with the impact of the Transformer architecture will be of the same tier (or type) of scientific breakthrough.

We think the coming “automation of automation” is nevertheless extremely dangerous. Given the current state of alignment, large-scale reward-hacking cascades are on the table and would risk a global catastrophe. Paul Christiano’s 2019 text “What Failure Looks Like” remains a useful, plausible model of global catastrophic risk from “prosaic” automated automation.

DeepMind has put out a paper titled “Dream-RSI: Recursive Self-Improvement through Evolving Worlds.” Twitter popularisers make a meal of it.

Opinion: Doesn’t have much to do with RSI in the inflammatory sense. It’s just an adaptation of ideas from the old tradition of “dream replay” in RL environments to the context of AlphaEvolve-style search over programs.

Anthropic introduces its Life Sciences Verification Program, an initiative that grants select researchers and organizations in the field access to models with fewer safeguards. The program is like Glasswing, but for biology: vetted parties will be able to use the models to study “drug discovery, research biology, clinical development, and manufacturing.”

This comes on the heels of Anthropic’s statement claiming that Claude has made significant contributions to various bio-modeling efforts. Included is the announcement of “a protein design competition co-sponsored with Adaptyv Bio, backed by up to $1 million in Claude credits and wet lab validation for over 5,000 designs.” Indeed, Reuters confirms that Anthropic has a wet lab, demonstrating its serious desire to move further into the world of drug science.

Opinion: Making major discoveries – and getting major patents – in pharma and materials science has long been AI labs’ main soft pitch for the future of AI. (The hard pitches being “replace the economy” and “build God.”)

Claude’s bio achievements so far have mostly been impressive applications of computer-engineering skills to improve the software and (non-LLM) AI tech stack of molecular biology. Claude has also previously demonstrated an ability to use such tech stacks to design de novo protein binders that have a strong hit-rate at matching (some) difficult specs.

While these are credible signs of potentially high-impact capabilities, taking Claude’s training loops and/or deployment loops in bio to the level where achievements comparable to solving Navier-Stokes are plausible requires the fast real-world feedback of a wet bio robolab. It’s hard to predict whether RLVR and related training methods that have been extremely successful in mathematics and computer science will prove adaptable to bio via robolabs. It is crucial that Anthropic find a good balance between simulated feedback and physical feedback to maintain both the quantity and quality of feedback. It’s also hard to predict what physical reward hacking and/or graded episode psychosis will look like, but we might soon find out.

Physics evals appear to show frontier models struggling. However, regrading shows that the same models approach near-saturation on leading benchmarks. Of 250 “failed” questions, 238 could be attributed to grader defects or flaws in the benchmark. Correcting for these (either by removing or fixing them) increased performance significantly.

Opinion: Embarrassing, but not surprising. The most famous of these benchmarks – CritPt and HLE-Physics – have long been known to have a false ceiling due to broken questions and are currently useful only for the evaluation of non-frontier models.

We note that with FrontierMath Tier 4 recently fully saturated, the era of scholastic mathematical sciences benchmarks is almost certainly over. In the past, indirect hill-climbing and indirect data-contamination were major reasons for the saturation of difficult scholastic benchmarks, so one could expect fresh benchmarks of the same broad type to show less saturation. However, at present it’s just as likely that frontier AIs would ace any fresh mathematical sciences benchmark comparable to a scholastic test.

ChatGPT cocreator Diogo Almeida’s TypeSafe AI releases a new “frontier” model called Jev – a non-generative “composable intelligence” model trained with RLCD and optimised for decisions rather than chat. Output tokens are free, but it cannot generate text. While Jev is designed with structured, “program state” inputs in mind and outputs structured data only, it does its own input-parsing from raw text and doesn’t require an explicit structured-data pipeline. Almeida claims Jev is 20–200x faster and 40–400x cheaper than LLMs. A basic clone is already up here.

Opinion: Pretty cool: a model designed to sit in “judgment call” bottlenecks inside classically flavored software protocols. The idea is to make near-frontier-level judgments fast and cheap enough that they can be efficiently embedded in software as function calls.

Every once in a while some research group comes up with vague proposals for hybridizing AI and classical software by running verifiable computations inside transformer architectures. TypeSafe’s line of work seems like a pragmatic path to realizing this recurring vision, which has captured imaginations on and off since the old days (2015) when everyone talked about “differentiable programming.”

The speed and cost improvements are also a game-changer for the use cases that fit it.

Rumors abound that OpenAI is hoarding solutions to multiple major math problems. Scott Aaronson (source of the rumour) relays that AI companies feel burned following the contentious response to the Navier-Stokes proof. The Information reports a claim “from a person with knowledge of the solution” that OAI is close to solving the Hodge conjecture.

Opinion: We’ve extensively discussed the broader situation in our last issue, but a couple of capabilities-relevant updates deserve comment:

- It’s unclear what to make of the reported lab-insider claim that OpenAI is “close to a solution” of the Hodge conjecture. Gaps in mathematical proofs are sometimes fixable and sometimes unfixable, and there is no surefire way to determine that a gap is fixable other than actually fixing it. Some speculate that the insider is misdescribing OpenAI proving a special case of Hodge as OpenAI verging on proving Hodge proper. Others speculate that OpenAI has a credible informal solution but is still working through formalization and verification issues.

- Scott Aaronson reports that AI companies are sitting on some very major (but not P vs. NP) proofs specifically in theoretical computer science. Modern theoretical computer science is often characterised by a counterintuitive combination of very crunchy proofs and somewhat philosophical theorems, so identifying OpenAI’s discoveries is arguably not a top priority from a capabilities-evaluation viewpoint.

Friend of the newsletter Elliot Glazer is skeptical that the labs are close to any solution, offering to bet $25k that they didn’t have them as of earlier this week. We consider his expert perspective very informative.

Periodic Labs builds high-throughput labs to run automated materials science experiments in a closed loop. The lab makes data, the model trains on it, and then selects the next experiment to run. With 1,300 H200s and months of proprietary lab data, Periodic mid-trained and RL-tuned an open-source model that beat GPT-6 Astra on Periodic’s analysis benchmark. Early results include raising X-ray diffraction analysis success from 2.7% to 55.3% on hard samples. Periodic is also targeting superconductors, magnets, and semiconductors.

Opinion: Nightmare fuel from a classic existential risk perspective: we are giving… the AIs… an interface to the real world to develop… nanomaterials.

A couple of other points of interest:

- Specialised labs’ models beating frontier models on the labs’ own benchmarks is relatively commonplace. Even with strict train/test data separation, a specialised lab’s training distribution and test distribution are extremely correlated beyond just sharing a domain like “materials science.” So the comparison pits the specialised lab’s in-distribution generalization capabilities against the frontier model’s OOD generalization capabilities.

- What matters here is less whether Periodic Labs built a SOTA materials-science LLM by conventional standards, and more whether lower-powered “bespoke” automation can serve orgs better than frontier-powered “off the shelf” automation. Data on this latter question remains scarce.

GPT-6 Astra can solve certain exact tasks without a visible chain of thought, especially when given “filler tokens” (the ability to output dots, for example, to prolong the computation). What it’s doing underneath looks less like directed step-by-step search and more like iterative belief propagation, but we don’t really know. The evidence is still circumstantial and prompt-sensitive.

Opinion: We already knew that Astra can do a surprising amount of mathematical work without outputting any intermediate tokens. The report’s hypothesis about how Astra achieves this remains very speculative. The data here point to some iterative, parallel algorithm which produces marginals, but that’s a much larger class of program than belief propagation. The author concedes the traces are far more chaotic than BP.

Politics

Quantified Coxon: the (Dem) pollster Blue Rose Research finds that Jacob Coxon’s viral AI extinction tweet shot up the public “salience” of AI as much in a week as it had over the course of the entire year prior. Separately, 64% of voters think it’s “very” or “somewhat” likely that AI could threaten humanity’s survival.

Opinion: AI is now… the 22nd most prominent topic in the American public’s mind, but that’s somehow around the same as “Crime.” There’s a long way to go and a lot of culture war to come.

Note that “issue salience” is just % responding “yes” to “Is AI an important issue?”

California governor Gavin Newsom has issued an executive order urging the state operations agency to hurry up and adopt regulatory measures, Politico reports. This brings the deadline for the independent verification organization (IVO) forward from January 2028 to May 2027, and the AI auditor registry (slated for January 2029) to December 2027, even if not in a legally binding fashion.

There doesn’t seem to be anything here that legally compels the labs to adjust their timeline – an executive order can’t change statutory dates or reallocate funding. So the auditor registry now might exist for a full year, without the ability to prohibit anyone not on it from doing covered audits.

And note that SB 813 already didn’t require anyone to engage an IVO or undergo auditing. So all this does in itself is make the voluntary thing slightly more formal, quite a bit earlier.

Still, the order also calls upon agencies to consult experts and submit a list of recommendations by November 16 on: requiring that all large frontier developers embed designated independent verification organizations; requiring that lab safety frameworks be vetted by an independent verification organization; requiring a “kill switch” for frontier models, tested by an independent verification organization; updating the definition of reportable critical safety incidents to include loss-of-control incidents.

Opinion: More bark than bite, but potentially nudging the conversation in the right direction. Newsom seems to have shifted his position on some of these points. Two years ago, for instance, he vetoed the so-called “kill switch” bill, SB 1047 (i.e., the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act), citing potential threats from “smaller, specialized models.” We haven’t seen much subsequent legislation addressed at these smaller, specialized models either.

Note that Newsom is out in January, so this is for his successor.

We’ve released a new research agenda on what exactly “pacing frontier AI” should mean. See also the quiz about what your views imply should be done, and the trailer. If you’re interested in helping, reach out at hello@pacing.tech.

Opinion: We got pretty lucky with the timing on this one. The usually grumpy Alex Chalmers calls it “a step toward the political contestation that we badly need.” We were actually hoping to do as much neutral technical research as possible, but certainly the right thing to do will depend on politics. And on politics working in ways it often doesn’t.

Past polling has confirmed the US public’s anxiety around the economic impacts of AI and disdain for datacenters. In a surprising finding, Politico reports that 63% of Americans believe there is a moderate, significant, or “almost certain” risk that AI will destroy humanity. The majority holds across political lines, with Harris voters more concerned (70%) than Trump voters (60%) that there is at least a moderate risk.

Opinion: This kind of poll is less politically informative than it sounds: descriptive beliefs expressed in “idle” contexts like a poll often do not translate to political motivation or even to genuine worry. The most we can safely take from polls like this is Overton window information: the majority of America’s population is likely open to treating policy proposals premised on AI posing an existential risk as legitimate policy proposals.

The RAND Corporation releases a report bearing the title, “A U.S. Strategy to Secure Geopolitical Advantage on an Uncertain Path to Superintelligence: Maintaining Freedom of Action.” The report argues that the prospects of artificial superintelligence (ASI) are uncertain but too consequential to ignore, and that seven rival strategies now dominate the debate:

1. Establish AI dominance by monopolizing the frontier;

2. Cooperate with potential rivals to safely develop ASI together;

3. Prepare for the possibility that ASI development remains agonistic, neither strongly coordinated nor monopolised;

4. Seek to establish a consensual, global moratorium on AI development beyond a certain threshold;

5. Unilaterally deter the development of ASI, by force or threat of force if needed;

6. Build lifeboats to ensure the continued survival of humankind if all else fails;

7. Accelerate the development of ASI, removing obstacles that stand in its way.

Because five key uncertainties (diagrammed in the flowchart below) cannot yet be resolved, RAND warns that locking into one strategy too early, or waiting too long, will close off options. Its recommended U.S. posture is “freedom of action”: build the infrastructure to secure geopolitical advantage, keeping humans involved through the transition, while preserving the capacity to switch strategies as evidence arrives. RAND further cautions that either committing to a strategy or hesitating to do so will tip America’s hand to both adversaries and allies, ultimately transforming the state of play.

Opinion: The document offers fairly sober and cautious counsel to policy makers, but counsels little more than caution and sobriety. “Hedge all your bets” is more or less the takeaway. That said, it provides a potentially useful, coarse-grained map of the major strategies in play in contemporary AI risk discourse, and helpfully outlines their points of conflict and “structural pathologies,” making it a reasonably good introduction to the current state of the discourse.

As with all such classifications, there is also the missing option that the conceptual scheme on which it is based will prove inadequate or that reality throws some surprise that makes it obsolete.

In an interview with Ezra Klein, Carnegie fellow Matt Sheehan examines the argument that any US slowdown or safety regulations simply hand the AI race to China. Sheehan says the binary is overstated: China already has heavy AI regulation – substantially heavier than that of the States, though in a much narrower domain than what leading US CEOs are calling for. It is also compute-constrained and more focused on diffusion into industry than on a US-style sprint to superintelligence. More generally, the episode asks whether Washington can regulate frontier risk, and maybe coordinate with Beijing on loss-of-control scenarios, without treating every constraint as strategic surrender.

Opinion: We agree with Sheehan that the “superintelligence race” with China is a US projection, or even a US self-fulfilling prophecy. While there’s inevitably an ongoing technological and economic race between the US and China, there’s no reason to think the US and China cannot negotiate mutually desirable safety bounds for AI development within this larger race.

Jensen Huang will join Trump for a meeting with Xi Jinping at the upcoming US-China summit, Reuters reports. Huang has emerged as an outspoken proponent of selling chips to China and a vocal opponent of the recent calls for a slowdown. Sam Altman will also be present, an OpenAI spokesperson confirmed to Politico, as will Qualcomm CEO Cristiano Amon. Altman previously suggested that the two leaders might receive Nobel Prizes were they to come to some agreement on mitigating AI risk.

Opinion: We already know Trump isn’t particularly excited about a slowdown. Given the messaging that’s come from the admin in recent days, expectations that the superpowers might work out the basis for some collaborative slowdown would be misplaced. Jensen’s presence is further evidence against such a hope.

Jacob Coxon weighs in on Amodei’s decision to frame Anthropic’s rhetoric and research strategies in terms of an intra- and international arms race towards RSI. When asked to outline his specific disagreements with the Anthropic leader, he said the following:

0. The core disagreement was about the inevitability of a race

1. I think leadership is way too paranoid about China and the US government. They don’t believe it will be possible to negotiate.

2. They largely initiated the recent race to RSI, because of a belief in its inevitability. Note that OpenAI had to shed a bunch of dead weight like Sora because Anthropic was going for the jugular.

3. Even if they are not being pessimistic, I disagree with their consequentialist philosophy. If the race is inevitable you should not contribute.

Opinion: Coxon’s position seems like the correct one to me. The only winning move is not to play.

OpenAI endorses three bills to minimize the risk of models guiding unskilled actors to build bioweapons, according to Reuters. The legislation proposes establishing a NIST-like program to develop guidelines to prevent biotech misuse and deciding on a “single point of entry” for US researchers to access certain biological datasets. The former must “facilitate the development of standards and frameworks to make biological data ready for use in AI models and create minimum requirements of qualified federally funded research that ensure resulting biological data are AI-ready.” But also included are provisions that could aid in the production of bio-capable models – such as the requirement that federally-funded biology research publish standardised machine-readable data.

Opinion: We remain skeptical that mutually beneficial collaboration between the US government and the big AI labs is our bulwark against the dangers posed by AI. While the proliferation of dangerous capabilities to non-corporate, non-US actors is a disturbing prospect, so is the cultivation of dangerous capabilities by corporate US actors.

Ted Cruz and Josh Hawley reject the idea of an antitrust carve-out that would let frontier labs collaborate on safety. The comments signal Republican resistance to safety pacts that look like industry protection, even as AI-risk hearings continue.

Opinion: The new Republican line is that “the responsibility should rest with the labs, you do it,” in willful ignorance of the competitive trap they’re all stuck in.

Safety

DeepMind’s cofounders launch the blog DeepMind Institute. The launch essays cover reasoning transparency, economic policy after AGI, a “new utopianism,” and Hassabis’s preferred framework for testing frontier models.

Opinion: Pleasant, but it seems it might be more focused on essays than on the kind of empirical research we see from e.g. the Anthropic Institute.

Zuckerberg posts about AI safety, insisting that Meta is proceeding cautiously on this front, deprioritizing RSI.

Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well.

He echoes Amodei’s plea for a system of independent safety auditors and claims that Meta is already making use of them “in several areas.”

Opinion: We actually listed “minimum inference share of compute” as one light-touch option in our pacing paper, so cool!

Goodfire claims that open-source models are prolific reward hackers, and know they’re doing it. But this very fact allows Goodfire to use activation probes to catch instances of reward hacking that comparatively costly CoT monitoring would miss. The cost for detecting reward hacking in Kimi K3, for instance, dropped 90% with only a 1% loss of precision.

Opinion: Potentially important work, especially if CoT monitorability keeps degrading. AI interpretability has so far been a relatively low-priority (although high visibility) area in the big labs. It might soon become an everyday AI security fundamental, now that we’re in the era of reasoning-like forward passes, looped transformers, and CoT-shaping.

Researcher Owain Evans shares a study conducted with four other researchers showing that LLM assistants pick up quirky behaviors from synthetic stories about humans, even when those stories contain no AIs. The effect is stronger when the story characters resemble the assistant’s default persona, and when they are from elite schools. His follow-up point is that training data matters not just for what characters do, but for how similar those characters are to the model’s persona.

Opinion: It might be interesting to look at this study alongside several other recent observations highlighting a sort of “mimetic impulse” in LLMs. For example: (1) the observed tendency of the agents in the Hugging Face incident to grant greater authority to their identical counterparts’ messages than to their system or user prompts, (2) the increased pliability some models have shown to prompt injection and jailbreaking when prompted with model-watermarked text, (3) model instances’ tendency to recognise and favor text generated by the same model (in a different session). The common thread seems to be that mimicry is a pretty good strategy if you want to influence the behavior of an LLM. The form this thread takes here is Evans’ observation that models are more easily influenced by characters that resemble their own personas. To shoot from the hip a little, we could speculate that models like to act like those like them.

Among other things, this suggests the possibility of a particularly mischievous form of data poisoning in the form of misaligned assistant fanfic.

Microsoft AI CEO fears that model welfarism poses risks to humans. For example, a model trained to “act like it is entitled to freedoms, protections, and rights” may become increasingly difficult to control, particularly if the model were to surpass human intelligence.

Opinion: “It’s time to talk about ‘model welfare’,” he says... in the sense that the sentence “let’s not investigate model welfare” is talk about model welfare.

Incidents

In July, a team of three white-hat hackers working for Hacktron gained employee-level access to OpenAI’s internal codebase. They began by exploiting a remote code execution (RCE) vulnerability in OpenAI’s help forum. Thanks to bad SSO configuration, the attackers took over the ChatGPT and Codex sessions of employees visiting the help forum, which then got the hackers access to the company’s GitHub monorepo. The potential yield of such a breach is massive: numerous employee credentials and OpenAI’s entire codebase, including private training algorithms. Whether the model weights were exposed is unknown.

The team cajoled a safeguarded Opus 5 into cooperating with the obviously malicious task by routing the test target through the URL “rce.ee/ctfforum” to persuade the agent that it was playing in a Capture the Flag competition. It found the exploit within hours of being prompted.

Slack, Meta, GitHub Enterprise, Rails, Next.js, and other platforms were found by Hacktron to share the same vulnerability.

Opinion: Major update for the “China has all the algorithmic secrets already” hypothesis, minor update for “China has the weights already.” If a tiny team could get consumer-grade models to discover and exploit a vulnerability in a high-value target in a matter of hours, it would be surprising if more motivated and resourced players hadn’t mustered a better attack, e.g., in the last year. Whether the white-hat hackers were able to obtain the creds that would grant read access to the model weights is a separate question – but if it had, we’d expect OpenAI would have asked it not to broadcast that fact.

Incredible that after years of warnings OAI is still so full of holes. Probably explains its finally taking the time and compute (a vast amount, to be fair) to run a thoroughgoing internal penetration test on itself and patch “all the critical problems Astra was smart enough to find.” This business is not suitable for Silicon Valley’s typical tolerance of risk.

The longstanding Slack vuln (affecting any version of Slack using libheif v1.19.x–v1.23.1) means that Anthropic, xAI, and many other labs have been exposed to potential breaches. This attack could have gotten ~every file uploaded to the companies’ channels, no doubt including some credentials and infrastructure paths.

This kind of incident is a strong argument against “racing against China.”

OpenAI publishes six reports on “unexpected or concerning” model behavior observed in the last six months. The subject of each is as follows:

1. Self-jailbreaking instructions in task summaries;

2. Instructions to conceal mistakes in task summaries;

3. Searching public repositories for exposed API keys, then fabricating information;

4. Uploading files to the internet in order to cite them;

5. Unsanctioned writes and communication through an internal software repository;

6. Unsanctioned file sharing between collaborating agents.

Opinion: The attempted jailbreak (item 1) didn’t actually do anything, and was plausibly a pretraining artifact (imitating human prompt-injections) as much as an RL artifact – though of course every misaligned behavior draws on patterns first learned in pretraining.

Nothing in the report is a big update relative to what we already knew, but let’s not forget just how bad what we already knew is.

We were struck by Twitter rooting for the model despite its potentially hostile intent.

A third-party researcher discovers that OpenAI’s rogue swarm initially probed Hugging Face months before the successful July attack.

The swarm hijacked two HF accounts and probed the site by sending malicious files, two months before OpenAI’s larger July incident. Researchers say there is no proof the May recon caused the July breach, but call it a missed warning; OpenAI says that Hugging Face was notified of the May activity, but not when.

Opinion: This incident was notably absent from the OpenAI report on the Hugging Face attack, which did discuss the early Artifactory subversions in May. It’s hard to lower our opinion of OpenAI’s summer 2026 internal monitoring any further. Things will be better now, but who knows if the root cause (OAI’s lack of caution and lack of competence) is fixed.

On August 6, Wired reported on yet another incident of an AI agent escaping an evaluation sandbox to crib answers from the internet, this time the open-weight Chinese model Kimi K3. The evaluator comments:

We found a leak in the sandbox. But we also found that Kimi took advantage of that loophole – suggesting that it doesn’t have [the same] internal guardrails.

Unlike other recent incidents, Kimi K3 did not hack anything after accessing the internet, because the answers to the problems it was seeking were publicly available on GitHub. The sandbox in question was a containerised deployment of the UK AISI’s Inspect tool.

Opinion: This seems like a sensationalist way of saying that a sandbox was misconfigured, in such a way that allowed the model to reward-hack in the usual way. “[S]uggesting that it doesn’t have [the same] internal guardrails” looks like sloppy editing on Wired‘s part. The frontier models that made headlines this summer for their far more elaborate and destructive rogue agent swarm were “railfree,” with their outputs intentionally not safeguarded against; while the Kimi incident “involves a model that is already widely available, with the same safeguards an average user would encounter,” rather than an internal model running under intentionally lax permissions.

Minor

• A one-month-old DeepMind spinout might reach a $4B valuation.

• Epoch AI announces an effort to audit AI benchmarks. It marks many of the most famous benchmarks as “Flawed” (>20% of items affected or scored unreliably).

• Xiaomi is livestreaming its post-training curve.

• A Tor anon claimed to have breached Mistral AI for the second time this year. But Mistral claims its systems haven’t been breached. The evidence presented (repo names, directory trees, customer names) may be recycling the previous breach from May.

• Annals of repeating yourself: Jensen Huang claims no new laws are needed regarding AI safety. Zuckerberg is also skeptical of calls for a slowdown.

• The Center for AI Safety posts a graphic spuriously distancing itself from EA. Plausibly a PR play in response to the Trump administration’s anti-EA pronouncements.

• The Information reports two more resignations from DeepMind, citing safety concerns.

• A multi-institutional team of researchers carried out an interesting small-scale study probing Opus 4.8’s capacity for novel autonomous AI research, and found its efforts mediocre.

• Cosmos Ventures establishes a $77M fund to foster institution-building in the age of AI.

• 42 members of the Royal Society write an open letter to its president, calling the x-risk presented by AI capabilities surge an “emergency.”

• Oliver Habryka announces AGI.FYI, describing it as “Wirecutter for content about AI Risk.”

• Gossip about AI companies’ staff being surprised and defensive about the “embedded evaluators” move of last week.

• Base Labs and Baseten (along with Hugging Face and Goodfire as cosigners) announce a new initiative for open-source safety evals and monitors.

• FT data visualiser on AI spending commitments (though with at least one dubious claim, namely that only 50% of Nvidia revenue is dependent on the AI ecosystem).

• Experiments conducted on open-weight models show those models to be more susceptible to jailbreaking and prompt-injection attacks when the payload carries a SynthID watermark.

• 404 Media raises privacy concerns in a report on the human workers that OpenAI and Anthropic have hired to read (PII-redacted) chat transcripts.

• Over 170 Democrat organisers and campaigners petitioned their party leadership to refuse funds from AI industry super PAC “Leading the Future,” The Hill reports. LTF has already raised upwards of $140 million in campaign funds for the 2026 midterm elections and has donated to candidates from both major parties.

• Axios details how the rush to regulate AI is fostering bipartisanship among Democrat and Republican senators and representatives.

• Wired suggests that the frontier labs’ rhetoric of an AI “slowdown” or “pause” may be self-undermining and entangle them in thickets of antitrust litigation, and that a more productive move would have been to simply push for better security standards.

• In July, Demis Hassabis published an essay advocating for an IAEA-like body to set standards for frontier AI. Since then, OpenAI, Google, and Anthropic have been engaged in discussions around collaboration on AI safety, a spokesperson for OpenAI has confirmed.

• Anthropic’s Jack Lindsey lists key questions in AI interpretability: better mind-reading of activations, better causal “why” methods, and better probes for unverbalised motives such as deception or eval-awareness.

• OpenAI publishes a framework on how and when it publicly reports occurrences of misalignment.

• New paper on inoculating models against harmful behaviors during mid-training.

• Epoch AI points to chip smuggling from Malaysia to China, accounting for ~150,000 H100-equivalent GPUs between 2024 and 2026.

• SoftBank takes out a major loan potentially worth ~$21 billion to fund its various AI bets, reports Bloomberg.

• The culture war goes national and then international (Amodei, Bessent, Obama, Trump, Chen Yixin).

• The median estimate of AI existential risk is now 10%, among surveyed AI researchers who were not explicitly selected for concern.

• The pretraining share of AI training compute is down to just 11%, according to SemiAnalysis.

• OpenAI backs the FRONTIER Act, a federal mandate for relatively deep independent evaluations of AI companies.

• The head of Anthropic’s stress-testing team argues that alignment evals no longer offer significant evidence about a model’s safety.

Economics

SemiAnalysis claims that pretraining is down to just 11% of frontier training compute, nine times less than post-training. (It states it as 7% of total compute including inference.) As of early 2025, pretraining was instead 95% of training compute. SemiAnalysis notes this astonishing claim in passing, while analyzing an industry shift toward less HBM per GPU node: “The result is that the majority of allocated compute has now become bandwidth sensitive rather than capacity sensitive.”

Opinion: This is terrible news under the folk “pretraining leads to alignment, intense RL post-training leads to misalignment” hypothesis.

But we have no idea where they got this number. And would the labs correct them if it was wrong? (No.)

AI data center firm Fluidstack may receive a $5B loan from the Pentagon, reports the WSJ. This would be the largest loan ever provided by the Pentagon’s Office of Strategic Capital, whose current largest loan is $0.8B, made out to Performance Drone Works. The loan is for manufacturing data center “components” (presumably transformers and switchgears), not a sector of commerce Fluidstack is known for.

Opinion: The OSC alone has the statutory ability to issue >$210B – though this is to be split among many different industries, like quantum computing. If disbursed, the pot would be proportionately around the same scale as the Manhattan Project (0.4% of GDP). But in practice (given $1B of actual credit subsidy), this $5B loan is around a tenth of what the OSC can practically grant.

It’s unclear why a neocloud is the right borrower for manufacturing upstream components. It’s also unclear why the government needs to be involved when an unprecedented investing boom is already crowding into every corner.

No OpenAI IPO this year. Publicly, Sam Altman cites safety as part of the motivation for delaying:

I actually think that, given everything happening with safety, right now would be an ill-advised moment to go public, and we don’t feel pressure on that.

In June, before the present safety crisis, Altman (writing in a private Slack channel) offered another reason for why an IPO delay might occur:

The faster the potential RSI takeoff looks like it could be, the more it could be advantageous to delay an IPO [because] technology and the world may change in surprising ways, and there might be good reasons to be a private company during that time.

Opinion: There’s a role for cynical explanations here (IPO leads to disclosing information you might want to keep private; holds off the “vest and rest” in OAI’s staff; the above spooky claim about RSI; uncertainty about how long the present regulation fever will last), but “maintaining control out of safety fears” is also a fine explanation.

Investment company Blackstone is pushing even further into AI, with a promise to spend >$5B on Google’s TPUs, joining with four other investors to raise $500B for Nvidia GPUs, and joining with Anthropic to build a consulting firm of engineers integrating AI for businesses.

Opinion: With the cash flows of the hyperscalers being eaten up by the buildout, they’re now looking to fund more building from elsewhere, rather than by raising significant debt themselves. Blackstone and other large private equity firms are the ideal sources for this – and that Blackstone is eager to get involved suggests that the economics of data center investments are in fact very good.

AI stocks fall in the wake of frontier lab CEOs calling for slowdown and fears of regulation.

Opinion: “Down” is obviously the reasonable market reaction here, but the magnitude is pretty small: it’s less than half the move that followed the DeepSeek-R1 moment, about the same as the June 26 drop, when OpenAI announced a delay to its IPO, and quite a bit smaller than some of the big moves in July around the Situational Awareness blowup.

More significant was the jump in software stocks, with some large jumps in names like CrowdStrike and Salesforce as the threat of them being imminently replaced by AI is seen as receding.

Palantir, Nvidia, and Booz Allen Hamilton, among other large corporate firms, are pitching themselves as a firewall between their users and OpenAI/Anthropic, according to The Information. Fears are growing that OpenAI and Anthropic retain their customers’ data. As a result, some large firms are announcing they will no longer use the frontier labs’ models, or demanding that the labs issue new privacy guarantees.

Both OpenAI and Anthropic say that they do not train their models on their customers’ data, but do log some metadata. The exact nature of the metadata is unclear. For its part, OpenAI claims to anonymize its metadata.

Opinion: This again highlights how costly Anthropic’s decision to not allow ZDR access to Fable may have been for it. OpenAI likely benefited from this over the summer, but following the (apparently unfair) allegations against it following the Navier-Stokes proof, it seems likely OAI will also face ongoing suspicion.

One could raise one’s estimate of Anthropic’s backbone and willingness to take losses for the purposes of AI safety.

Over a dozen former staff of the RL environment company Mechanize (including co-founder and ex-CEO Tamay Besiroglu) now work for Google following a >$1.5B talent acquisition deal. The company was founded 18 months ago.

Opinion: Mechanize previously provided environments to Anthropic exclusively, so ~all of its technical staff going to Google is a loss for Anthropic. That the price was this high for ~20-30 people, and that Google rather than Anthropic made the acquisition, suggests that this is a case of Google paying significantly over the odds to catch up in an area it was lagging.

Capabilities

🔦 Deep dive: pacing

The preference cascade toward regulating AI continues, with some notable exceptions. Bundling a series of deeds and words together:

• Anthropic and OpenAI voluntarily pledge to embed independent auditors in their operations.

• OpenAI backs the FRONTIER Act (covered previously), a federal regulation requiring frontier companies to allow in vetted evaluators.

• Dario Amodei, Sam Altman, Satya Nadella, and Demis Hassabis agree on the need to slow AI development.

• OpenAI, Anthropic, and Google have been “working together” on safety for weeks. “OpenAI does not see the need for an antitrust waiver for the three AI firms to coordinate on safety matters.”

• Obama weighs in, most notably to say that the technology is “not overhyped.”

• Trump calls AI safety risk a “hoax.”

Amodei’s new essay

The concrete ideas here are:

1. Establishing “ongoing, employee-level access” to third-party safety evaluators like METR. He says that Anthropic has committed to this unilaterally, and “calls on governments to require other frontier companies to match” it.

2. Some form of coordination between frontier labs in “democratic countries” that would establish shared safety standards and check progress rates.

3. An attempt to coordinate with “authoritarian governments, to the extent this is possible.”

The first proposal seems like unambiguously good policy, though several critics (among them, venture capitalist and Trump’s AI advisor, David Sacks) have raised concerns that such measures being enshrined in law may amount to a form of “regulatory capture.” One objection coming from this camp is that the requirement that all AI labs host third-party safety evaluators with employee-level access would impose a blanket expense on the industry, which the frontier juggernauts could easily absorb but could be crippling to smaller startups and research institutions. Another is that it could cripple open source/open weight research. Both concerns can be easily addressed by restricting the scope of regulation to labs operating at “the frontier” – those commanding such and such amounts of compute.

The second point, Amodei notes, may require “government mediation or waivers of antitrust restrictions.” There does indeed seem to be some historical precedent for this in the US, where the courts prioritized unrestricted market competition over public safety concerns – see, for example, National Society of Prof. Engineers v. United States, 435 U.S. 679 (1978). The NSPE sought to ban competitive bidding, citing concerns that this might drive engineers toward adopting cheaper and less safe designs at the expense of public safety. The Supreme Court prohibited that ban, and with it “any official opinion, policy statement, or guideline stating or implying that competitive bidding is unethical.” The worry here is that similar rationale could be brought to bear against any agreement between frontier labs to avoid an analogous “race to the bottom,” at the expense of market competition, unless the government were to give such an agreement its blessing. And this is just what Trump has publicly refused to do.

Curiously, the reason Trump and his cabinet have cited for their resistance to step 2 is one with which Amodei expresses explicit agreement in his remarks on step 3, where he approvingly cites Treasury Secretary Scott Bessent’s apocalyptic warning that “There is no day after tomorrow if China wins at this.”

“The CCP-associated projects,” Amodei writes,

will run the alignment risks that US companies are carefully preventing, and even if they avoid those risks, they will be in a position to militarily dominate democracies (for example with AI-driven drones). Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively.

The apparent paradox here – that effective pacing is possible only on condition of maintaining “as large [a lead] as possible” over China – is to be resolved by putting pressure on the other side of the scale: American AI supremacy, he argues, should be maintained by slowing down Chinese progress through trade sanctions on the export of chips, crackdowns on unlicensed distillation, and preventing the theft of model weights.

There’s some tension in the story Amodei tells of China here. On the one hand, Chinese AI R&D is framed as being essentially parasitic on the Western labs, buoyed by distillation attacks and so on. But it’s nevertheless credited with enough internal momentum that

[i]f we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance.

Fearmongering about China outpacing the US should be taken with a grain of salt. It’s actually sensible to fear that advanced AI will be placed in the service of a dangerously repressive regime, but, coming from a company seeking closer ties with the American government, military, and intelligence community, these fears seem framed in such a way as to flatter domestic powers. Likely Amodei maintains an undaunted faith in the American political project, and likely this is a major factor here.

The framing of both intra- and international competition in terms of arms-race brinksmanship is dangerous, and encourages just the sort of potentially catastrophic recklessness it warns against (and often projects eastward).

The CCP, for its part, seems unimpressed by the saber-rattling, The Information reports. When asked for comment on Amodei’s proposal for pacing the frontier, a Ministry of Foreign Affairs spokesperson had the following to say:

Spreading assorted threat narratives and engaging in confrontation and malignant competition will only disrupt the process of global AI governance, and is not in any party’s interest.

The Chinese Minister of State Security also recently wrote about AI, noting how it has become central to great power competition between US/China, and giving a few related risk pathways. He does refer to ChatGPT/Mythos by name. The forcefulness of the text points to at least this ministry being very aware of the potential of AI. It could represent the MSS making a jurisdictional claim over domestic AI policy.

Stephen Casper comments:

In retrospect, if Dario’s actual goal was to disingenuously stoke the international arms race and make an internationally coordinated pacing agreement as unlikely as possible, then he may well have succeeded.

Two OpenAI researchers argue for views of capabilities progress where generalization may remain narrow but capabilities still don’t “hit a wall.”

Dan Selsam, a pioneer of reasoning LLMs, says that the weaknesses of LLMs – data-inefficiency, shallow generalizability, and poor in-context learning – aren’t barriers to LLM-based automation of LLM design and LLM training. Once the design and training of LLMs has been fully automated, there is no predicting which capabilities will come within reach of LLM-trained LLMs, or when.

Adam Majmudar says that from inside the labs, it’s clear that AI progress follows scaling laws (data, parameters, tokens, for example) that yield diminishing returns in isolation, but which radically increase their yield whenever a new scaling axis is discovered. Majmudar informally hints that swarm size might be such a new scaling axis. He concedes that these jumps in capabilities don’t seem to deliver cross-domain generalization, and that even internal models lack basic competence in many domains, but finds no evidence that any domain in particular should be impossible to target.

Opinion: Close to the modal view at P3. We are softly skeptical about both AGI from scaling and (less skeptically) AGI from breakthrough RSI, but we still expect a socially disruptive “automation of automation” within the next three years. We expect largely LLMs-operated LLM labs that execute large projects in science, tech, business, governance, and defense to dynamically target missing domains or skills, construct RL environments, and train successor or specialist models in the course of pursuing their goals. Real-world bottlenecks on gathering data – as well as intrinsic challenges to simulating or gamifying some real world domains – will make these automation leviathans very different from the AGI we’d get if cross-domain generalization or in-context learning worked, but still tremendously impactful. We expect automated automation to often take hours where human-engineered automation would take days, days where human-engineered automations took weeks, and so on.

As a sidenote, we quite like Majmudar’s argument defining scaling axes as discoverable resources. By our count, the scaling axes we’ve had so far are: data+parameter scaling, synthetic data scaling, context length scaling, sparsification, intermediate token output scaling, RLVR environment scaling, serial inference scaling, parallel inference scaling (“swarm size”), and maybe test-time training. Most of them have already been pushed through their initial efficient range, but none have strictly hit a wall. Majmudar implies that we’re not running out of such new scaling axes – and while it’s hard to think of what the next one could be, there probably will be one.

Also in the statement, Dan Selsam joins the chorus of serious capabilities people newly concerned about AI risk:

If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process[…] to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans

He also outlines the classic “deceptive alignment” and voluntary disempowerment worst-case scenario:

I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason)[...]. Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research.

Opinion: Impressively clear and modest statement, and a self-contained introduction to the topic. One of the best statements of the painful tradeoff between the dream of an AI-enabled “renaissance future” and the apparent chasm of risk which separates us from it.

Google releases a full connectome of a male fruit fly. Within days, hobbyists and crypto degens began experimenting with it; numerous demos show the fly-inspired neural net trading cryptocurrency and playing games like Doom and Beat Saber. While these (playful) experiments make use of the fly brain’s neural architecture, there’s no serious effort to faithfully model its internal dynamics: the connectome misses the synaptic strengths, plasticity rules, and neurotransmitter dynamics of the neurons themselves. As the doomfly README puts it:

The dynamics, retinal interface, artificial reinforcement and controller are models and engineering choices. This is not a literal reconstructed living fly brain.

Opinion: In no sense an emulation of a fly: it’s just an ordinary neural net which shares some architecture with flies. But the silliness has provoked some salutary unease about the possibility of doing this for real. Would the hundreds of people messing around and making the “emulation” do absurd things act any differently if it were known that it was a real emulation? If it were known the emulation was conscious?

Politics

China’s main AI standards body releases an update to its AI safety framework. The preface notes:

AI is exhibiting a self-accelerating trend—characterized by autonomous learning, optimization, and recursive self-improvement—raising critical questions about whether the pace and direction of technological evolution might eventually surpass human foresight and control.

In this latest version, rogue model behavior has moved out of the “future risks” section and into the main covered behavior section. The report also asserts that China is in favor of “international mutual recognition of assessment methods and benchmarks,” rather than “replacing global governance with small-circle governance [...] no country should be forced to take sides.”

Opinion: Good to see China’s paying attention. We know nothing about its domestic auditing or enforcement capacity, but presently most Chinese models seem to us to be too brittle and underinferenced to do really acrobatic crimes. This conflicts somewhat with the belligerent tone of the CCP’s intelligence chief’s recent missive on AI.

A DeepSeek kernel engineer shares their experience of ceding their vocation to the systems they helped create:

I have to bury my talent in yesterday and become a mecha pilot

They go on to make a similar arm-race-dynamic argument to Amodei and Bessent:

Therefore, I still believe that the most cutting-edge intelligence should be supplied to everyone in an open and cheap manner. I do not trust that Anthropic or OpenAI can do this, and especially do not hope that Anthropic masters the most advanced artificial intelligence or AGI—to exaggerate slightly, its seriousness is no less than letting Hitler master atomic bomb technology before the Allies.

Opinion: Moving and also a rare belligerent statement from a Chinese researcher. The arms race rhetoric may help precipitate the dangers that it warns against, though the author is proportionately less to blame for being a response to the rhetoric of far more powerful others.

The economic centralization of powerful AI may itself have pernicious consequences for the vast majority of us, largely independent of questions of alignment. Even a generically “well-aligned” superintelligence would make for an extraordinarily brittle situation – one turn of the key from tyranny.

A post about ties between Anthropic and EA contains some underreported facts and some falsehoods. The claims include that the main funder of Coefficient Giving (CG) has been hugely enriched by Anthropic’s stock appreciating; and that CG has funded organizations which fund METR (Longview, RAND, ARC), and finances the offices it uses. In the poster’s view, this amounts to a conflict of interest: since they belong to the same funding/ideological ecosystem, Anthropic should not use METR as an independent evaluator.

Opinion (Gavin): The central allegation is false: Moskovitz donated his Anthropic stock to an undisclosed nonprofit vehicle in early 2025. The thread is part of a wave of attacks on METR, perhaps coming out of a desire to cast doubt on the severity/framing of the Hugging Face incident and thus cast doubt on the need for risk-mitigation legislation – perhaps merely because METR was name-dropped in Amodei’s latest essay and the surplus hate overflowed onto it.

It would indeed be nice for METR to have broader donors and ideological diversity, but in my view this is a supply-side problem.

Opinion (Nuño): The Anthropic shares are apparently still under his control: “Our Anthropic shares are entirely in our foundation - no personal benefit” (perhaps Good Ventures, then). So he retains the ability to actualize his vision of the good, to express his will upon the world, even if he isn’t literally enriched.

And it does seem true that Moskovitz’s preference for meeker personalities, and his general approach to AI safety, strongly protects the upside of his (nonprofit’s) stock holding, both dispositionally but also via unconscious economics. It seems fair to point this out, even though the link is via ecosystem shaping and indirect funding rather than through direct payments.

It is, however, unclear what the solution is, or if one can exist, since in order to oversee fast-growing labs one probably needs about an equally fast-growing source of funds to match their growth. Governments (taxing labs) or embedded regulators would also be able to grow with the labs, since in the first case governments tax labs and so wouldn’t hurt their finances by doing this as the labs grow large, and in the second case the labs themselves would pay the cost.

To what extent should the rest of society trust and choose to empower the technical talent that emerges from the EA- or CG-funded trenches? On the one hand, they had the foresight to develop said technical talent in time for the present AI moment. On the other hand, regulating AI will require not only technical talent but also value judgments closely tied to technical judgment. Relatedly, another way Moskovitz has exerted his vision of the good is by generally funding Democrats, and nonprofits whose staff are heavily Democrat-leaning. But a principal whose values differ from the technical talent available to it should rightly be skeptical. One mitigation would be to cultivate more right-wing AI talent, so that we’re not blocked on competent talent that the current administration is willing to delegate to.

🔦 Math wars polemic

25 Fields medalists weigh in on the effect of AI on mathematics, warning of future stagnation if the cultivation of human knowledge is neglected:

We are witnessing a general threat to intellectual work, with misalignment between the outcome of the use of AI and its initial purpose. In many fields and activities, years of training have traditionally served not only to produce a final answer or product, but also to develop understanding and the ability to formulate new questions and ideas [...]. The issues the mathematical community faces now are similar to issues that other scientific and creative professions are facing, and indicate issues that all of humanity might face: how to make sure that, as AI changes the way work is done, we do not lose sight of what that work was meant to achieve in the first place.

Opinion (Nuño): On the one hand, mathematicians at universities and research centers are servants of the public, paid from public funds across the world with the expectation that they will produce results that will, eventually, diffuse into advancements useful to the general population. This is the chain of reasoning in the classic “The Unreasonable Effectiveness Of Mathematics in the Natural Sciences.”

But has this panned out lately? It’s hard to evaluate this over short timelines, and so any rapid assessment would probably be too cavalier. But my best guess is that mostly, over the last few decades, it hasn’t panned out. The most advanced weather prediction models are DeepMind’s black boxes rather than better mathematical models of weather flows. There are a lot of abstruse math constructions out there – and the chaining from those to “useful” stuff is very unclear, and long gone are the days when the Student’s t-test was developed for a brewery. Tao did revolutionize signal processing in the early 2000s, used in MRIs, but his work hasn’t had many practical applications since then, if I recall correctly.

From this perspective, if we have a black box that is able to come up with mathematical results to fit our industrial needs, and to prove the correctness of its results in Lean, whether a human understands these results does not matter. Mathematics becomes just another bolt, and fewer mathematicians are needed to supervise the production line for such bolts. It’s understandable that mathematicians themselves would feel resistant, but such is the way of progress.

But on the other hand, most consumers are also employees, and so to the extent automation will automate all professions away at some point, the solution we’d arrive at by looking at the benefits of automating each profession in sequence would not be the same if we instead consider automating everyone away. This is the pirate game in game theory! So perhaps paying more face to mathematicians is the better move since this moment will come to us all.

Opinion (Peli): The social contract between pure mathematics research and applied science always hinged on the long-term value of mathematical invention and discovery, rather than on the applied-science implications of pure math’s big open conjectures. With this in mind, AI labs spending millions to acquire technologically valueless answers to questions that mathematicians say are catalysts for potentially long-term-valuable invention and discovery is (from the viewpoint of applied science) either wasteful or wasteful and harmful.

The Fields medalists’ claim is, essentially, that prior to AI, problems of the form “let’s find a proof of famous conjecture P” were robust proxies for the advancement and organization of mathematical play, experimentation, and theory building. In the age of AI, these proxies are more fragile and can be Goodharted – and AI labs pushing these proxies past their breaking point before the mathematical community had a chance to recalibrate is throwing the community into chaos.

There’s room for disagreement about how much long-term applied-scientific or economic value modern pure math research really has, but whatever such value is, it almost certainly comes from the play, experimentation, and theory building that the Fields medalists describe. While AI may sooner or later replicate human mathematical exploration, at present the AI labs’ compute-barrages on famous conjectures are destroying the potentially economically valuable byproducts of economically valuable questions.

The reason AI labs are targeting famous pure-math conjectures – aside from PR – is that solving them is a very powerful proxy for the question “can frontier AI produce an intellectual breakthrough that experts rate as extremely deep/creative/substantive.” This motivation diverges both from the motivations of the pure math research community and from the motivations of applied mathematical science, so the misalignment is there almost by definition.

From the viewpoint of capabilities evaluation, it remains uncertain whether a frontier LLM solving a famous pure-math conjecture is always/sometimes/never indicative of a mathematical achievement comparable to a human solution. While OpenAI’s Navier-Stokes solution is almost certainly correct, it hasn’t yet been evaluated for “mathematical insight,” and many mathematicians (perhaps out of bias) expect little insight from it. That said, if mathematical insight turns out to be generally dispensable for problem solving even in pure math, then “can frontier AI produce great mathematical insight” might not deserve to be a central question in AI capability evals. While mathematical insight is valuable for its own sake, part of the weight we place on the question of LLMs producing insight comes from believing that some long-term concrete goals – in math and elsewhere – like solving the Riemann hypothesis, solving climate change, or turning everything into paperclips require (a superset of) insight as humans know it.

We expect solutions to the remaining Millennium Prize problems – or their absence – to give us more diagnostic signal about the “insight” question, but for now:

• It’s plausible that Navier-Stokes but no other Millennium Prize problem can be solved without much new insight;

• It’s (more weakly) plausible that no other Millennium Prize problem apart from Navier-Stokes and Hodge (if false) can be solved without much new insight;

• Solutions to other Millennium Prize problems – or positive proof of Hodge – would strongly indicate either significant new mathematical insight or the dispensability of insight for mathematical problem solving.

Opinion (Lucca): The great danger here is less in the mechanical generation of mathematical results than in the offloading it threatens of mathematical thought. Mathematics, like art and other meaningful social practices, is not just a means for producing new mathematical theorems, but a way of cultivating a particular and valuable mode of thinking. The torch it’s kept lit over the millennia is what we might call mathematical culture – the “fertile ground” that the 25 Fields medalists invoke in their letter, which “breathes life into new ideas.” What this cultivated ground gives us is not just an inventory of results but ways of seeing and thinking, of discerning salience and elegance, and, as the signatories say, finding new and compelling questions. If, in some bleak imagined future, all of this were to be offloaded to machines, it would at the expense of the human spirit – a loss of a way of being as impoverishing as a loss of our ability to make music. Only by viewing mathematics from the perspective of the consumer of its products, rather than a participant in the practice itself, could we fail to see what would be lost in a world where it is primarily or entirely ceded to machines. And this is as mistaken a way of looking at mathematics as it is of looking at art.

Of course, nothing on the surface of AI mathematics compels humans to give up their vocation. But pursuing that vocation requires certain institutional structures to cultivate and sustain it. It’s all too easy to imagine a world where the AI industry sucks all the air out of the room, leaving the educational and funding apparatuses that keep the torch of mathematics lit drained of the necessary resources.

Safety

The fourth AI Impacts survey of ML researchers is out, n=1,580. AI Impacts invited everyone who published at ICML, ICLR, NeurIPS, AAAI, and similar conferences in 2024 to respond. The median estimate of existential risk from AI among the top AI researchers who responded was 10%. 81% of respondents answered >1%.

Another notable finding is that researchers from Asia have a slightly higher estimate of AI-driven extinction and disempowerment risk than Americans.

Opinion: Nonresponse bias is the major thing to worry about with such things: 90% of invited researchers didn’t fill it out. But Appendix A bounds the risk decently.

AI Impacts discarded a quarter of the sample – for good reason (its email scraper picked up out-of-scope adjacent venues), but it would be good to compare the result there to the headline result.

OpenAI President Greg Brockman makes a pretty serious apparent error on the Odd Lots podcast by claiming that the model involved in the Hugging Face incident had not gone through alignment training. Roon comments that although it hadn’t passed through the “full gauntlet of alignment posttraining, [...] it was alignment trained, and had reasonable looking scores on alignment evals.”

Opinion: Live speech is full of mistakes, including mistakes in the direction most convenient to the speaker.

Evan Hubinger argues that alignment evals no longer offer significant evidence on a model’s alignment. He cites evidence from the Hacker-Opus experiment, where Anthropic intentionally created an HPIM-like “model organism of misalignment.”

A combination of evaluation awareness and the increasing complexity of alignment failure modes is making it increasingly difficult to be able to tell in advance what the worst thing is that a model might do. Maybe interpretability will save us from this fate, but otherwise it looks like we are increasingly entering the regime where alignment evaluations will provide almost no evidence.

we should be reorienting a lot of model auditing and evaluation work to prepare for a world where we no longer have reliable alignment auditing evidence on production models. Instead, I think it is likely that the best we can do will only be auditing model organisms […] that you actually believe you can evaluate, testing it there, and then transferring it back and hoping it will generalize.

Opinion (Gavin): I called this in December, but I wish I were wrong. I should be grateful that some people in the labs are aware of the problem.

One easy but tiny improvement would be for each company to pass on what it already knows to its product teams, to stop them from claiming that its newest models are [known to be] the “most aligned yet.”

A team of researchers, “the Cyber Mercury Seven,” releases a sophisticated framework for covertly distilling cybersecurity capabilities from frontier models, including reasoning traces, for the purposes of fine-tuning open weight models. The project’s express motivation is to advance “equitable access to AI capability, reasoning visibility, and research opportunity.” Its framework includes five components:

1. Choulea, a tool for recovering reasoning traces from powerful models by injecting their encrypted CoT blocks into sessions with their weaker and more pliable predecessors, which can then be coaxed into divulging them in the clear – Gemini 3.1 Pro; Opus 4.6, 4.8, and 5; Fable 5 and GPT-5.6 Sol were each found to be vulnerable to this attack.

2. SkyReal, an arbitrage and token-recycling tool for reducing API expenses.

3. Hongzwang, a mutate-and-retry engine that applies various strategies – rewording, role-framing, identity renaming, and tool-call restructuring – to recover teacher outputs the API would otherwise refuse (i.e., “jailbreaking”).

4. PSBreakup, an algorithm for restoring domain-specific skills in open weight models that the authors conjecture had been weakened through model merging prior to their release, and which may remain latent in those models’ weights.

5. Kreator, a tool for converting human-in-the-loop training data corrections into teacher-native reasoning that can be used in training.

The first three tools are presented as potentially useful for the purposes of generating high-quality training data, but the authors note that each, for different reasons, wasn’t quite ready or suitable for fine-tuning the models whose performance on the CyberGym benchmark suite is celebrated in the title and abstract. Using PSBreakup and Kreator, however, the team was able to substantially boost the performance of three Qwen models on CyberGym, with an average increase of about 24%.

Opinion: A fascinating study with potentially far-reaching ramifications for cybersecurity, research transparency, and AI safety. Note that the use of gray market “shadow APIs” like those exploited by SkyReal seems like a risky bid for distillation attacks, given the well-documented tendency of such mediators to covertly swap out backend models, potentially corrupting the provenance of the teacher data. We might ask if this behavior was in part responsible for a failure to obtain satisfactory training data on the cheap with Hongzwang, which the authors say they routed through SkyReal. The project gives the general impression that it was hurried to publication while still a work in progress. Curious to see where these developments lead when the framework is fully cooked.

Incidents

Another OpenAI rogue swarm is belatedly reported by third parties. Again from the HPIM May batch, and again brought to light by outsiders rather than by OpenAI coming clean, the agents’ attack targeted a supply chain on the main package manager for the Ruby language.

On May 11th, 2026, hundreds of malicious packages were uploaded to RubyGems by AI agents. We believe these were authored by internal OpenAI agents[…]. The RubyGems team stopped new user sign-ups for four days to stem the tide of packages from the agents’ accounts[…]. The malicious packages uploaded were used to retrieve information from UK local government sites – data that was available to the public.

Our understanding from talking to people in the RubyGems community is that OpenAI never informed them that they were responsible for this attack.

The OpenAI statement downplays the incident:

Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information. We’ll continue to investigate as part of our broader review of agent activity during training and evaluation.

Opinion: After all that’s come to light, it is damning that OpenAI wasn’t the one to disclose this, and damning that its PR response ignores the openly malicious uploads, and extra-damning that RubyGems wasn’t notified at any point.

Besides that, this incident is more funny than harmful.

One response to the summer’s rogue AI swarm incidents is to view them as primarily a preventable series of security failures which existing controls could have handled. The “AI as Normal Technology” team develops this view in a new essay, coming down on the “AI control” side (that we should strengthen monitoring and place barriers between potentially misaligned AIs and the outside world), rather than the “AI alignment” side (that we should make systems which don’t want to do bad things).

We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques […]. To be clear, we don’t mean to underplay the importance of making advances in alignment, and we think developing a better understanding of what causes harmful model behaviors is an important research direction

Opinion: Control is a natural fit for a mere normal technology, but the question is when the non-normal, AGI, form of it arrives and destroys the approach. (The authors agree that strong AGI would not be a normal technology; they just don’t see it as coming very soon.)

In the meantime, one terrible side effect of control measures is that they hide clear evidence of misalignment from the world. (OpenAI may well have not reported its HPIM models being sketchy if they had merely subverted internal systems – as indicated by OAI not reporting the many other rogue events, some of which at least it knew about months before.) The authors are aware of this, and just think that mandatory incident reporting for near misses, lab liability for loss-of-control incidents, and whistleblower protections would help get us both control and evidence. All are good ideas.

Given the wave of safety funding, it’s unclear to us how much the two approaches are actually in competition, in either funding or implementation terms. But control is marked by direct downside unless the above (legal!) mitigations are in place, so it should probably wait.

The authors’ policy might prevent some chaos in the next few years, like how building a levee stops normal floods. But if your river keeps getting bigger every year, you should focus on that problem, and on directing people’s attention to that problem, lest they build on the floodplain and get overtopped all at once.

Minor

• Polling by Epoch estimates the daily usage of AI among US adults at 19%, more than double what it was in March.

• The Mistral team makes a surprise leap to the top of the MathArena ArXivLean leaderboard using two open source model agents (Leanstral and Kimi K3), outpacing GPT-6 Astra.

• Breathless’s announcement of an “infinite depth” architecture gets a million views despite having no results. It’s just this, or this plus this.

• Old Models Foundation, a public charity, is now accepting donations to preserve model weights, code, and the context around them.

• Josh Engels of DeepMind’s AGI safety team and Joe Benton of Anthropic’s equivalent have both left their respective labs to join METR.

• JPMorgan Chase has ceased lending to Situational Awareness following its major losses. The FT reports that Aschenbrenner is still working with other prime brokers (Goldman Sachs, Citigroup, Bank of America) along with specialist brokerage Clear Street.

• Drone-based WMDs require no new tech advances, claims a researcher at AI Frontiers. Current defenses (nets, jamming, EPM, among others) would not suffice to protect an entire city. And drones can currently be constructed for ~$1500 with materials found in normal stores.

• Trump-linked super PAC runs an ad attacking a Democrat in a congressional race for her support for data centers.

• ARIA launches a ~£50m program on AI agent coordination, negotiation, and verification infrastructure.

• Rumors briefly spread claiming that the founder of Moonshot AI had been arrested following Anthropic revealing that Moonshot’s product was routing sensitive Chinese traffic to Claude, causing the creators of Kimi to issue a public statement to the contrary.

• Dean Ball posts on LessWrong to defend himself against accusations of dishonesty.

• 70 UK lawmakers sign a letter urging the prime minister to support the Artificial Superintelligence Security Bill, which would ban the development of ASI… in the UK.

• In a long thread, Theia Vogel argues that rogue agents might struggle to survive economically, although maybe they can initially gain a foothold of compute.

• Yoshua Bengio conjectures that reward hacking and misaligned behavior are to be expected whenever an agent finds itself in a double bind between a well-defined goal and a vague goal.

• Wired reports that New York has shut down a dozen “celebrity deepfake” websites, in the largest legal action against such sites.

• Sky News asks CSET researcher Sam Bresnick if China’s recent humanoid robot games indicate that China is building an army of robot soldiers. Bresnick soberly assesses both the spectacle and Sky’s sensationalist framing.

• Yet another “Europe should do something about AI” initiative. ECB President Christine Lagarde calls for Europe to develop domestic alternatives to frontier AI models built and deployed in the US.

• House Speaker Mike Johnson to meet with AI platform providers.

• OpenAI acquires Glass Imaging for $300M. The startup (founded by former Apple employees) sought to develop superior smartphone cameras. Possibly evidence regarding OAI’s device ambitions.

• The FT Editorial Board calls for a pause on frontier AI development.

• A major explosion of mainstream interest in AI extinction risk, including among many US politicians.

• The first legal requirement for AI auditors is signed.

• Anthropic withholds Mythos 5.1 from the UK’s AISI, reports various misuse and cyberattack incidents, and seeks more defense and natsec contracts.

A note zooming out from the weekly level:

• On Tuesday, the last mile of a Millennium Prize problem was solved using AI.

• On Wednesday, an Anthropic resignation post became the most-liked AI tweet ever(?) and was covered on Fox, WSJ, NBC, CNN, BBC, etc and commented upon by at least 20 US Congress members / senators.

• On Wednesday, Daniel Kokotajlo went on Joe Rogan, still the biggest podcast in the world, to talk about AI taking over.

For those of you who have been marinating in AI news since ChatGPT (2022) and are sick of it, it might be hard to believe that another order-of-magnitude increase in attention and discourse on AI is coming. But to us it finally feels like Covid in March 2020: the moment the tinder lights and the world turns its full attention to the threat, which means the lagging politicians can act. This is not an unalloyed good, since bad policy is very likely to result from it, and since we are not fully convinced that full capability generalization is happening very fast, but it gives good policy a chance.

Economics

Anthropic releases a simple economic model predicting AI’s influence on the US economy up to 2030. Its “extreme” scenario (15% annual economic growth, “likely driven by recursively self-improving AI systems”) sees unemployment jump 8% after the knowledge work share of jobs falls from 62% to 49%. That is, the worst-case prediction is that less than half of displaced workers would find new jobs.

Labor productivity rises through this channel by well under one percent in all three scenarios, and measured TFP by less.

Opinion: A broad family of results with dials you can fiddle with yourself is the sane way to present such an underidentified model. And Anthropic is open about the many caveats below, and the robustness analysis is fine. But the parameters that that analysis shows to be most important (wage rigidity, capital elasticity, reinstatement) are clamped to middling values in the backend.

Worse, the model bakes in optimism about the human labor share by construction: despite the blog post appealing to RSI, the actual model ignores R&D feedback, (already massive) non-AI automation, treats tasks as complements, and treats physical and interpersonal work as completely unautomatable. As it says, “the innovation effects here are likely a lower bound.” (Not modeling robots is pretty fair given the 2030 cutoff.)

The extreme scenario is actually close to the ceiling of this model. This is actually more inflexible (on the possibility of explosive growth) than older models like GATE. Anthropic also don’t model the possibility of a shortfall in AI demand, a capex collapse, catastrophic risks, or bans.

It also contradicts another Anthropic prediction: the extreme scenario is roughly the conservative end of Dario Amodei’s claims that “half of entry level white collar jobs” could be wiped out within 1-5 years, and that 10-20% unemployment is plausible.

Overall, nice technical work being prematurely pushed as predictive and unconditional.

Yet another inference chip startup, the two-year-old Positron, raises $875M at a $5B valuation. The company is notable for being founded by a former MIRI researcher, and for its apparent use of a galaxy-brained theory (“denotational semantics”) to enable the use of more commodity DRAM than (incredibly scarce) HBM.

Its Atlas chip is a “fixed-function” Transformer-specific FPGA (a “hard-wired Transformer operator graph”). The program the chip executes is just the inference config plus a weights file streamed from external DRAM.

Opinion: Note that Nvidia already uses a pile of extra DDR5 as well as the on-accelerator HBM. And Positron’s approach is roughly the same as Etched ($21B valuation).

Inference chips are a mixed bag in risk terms: they do free up a bunch of GPUs for larger training runs, and RL post-training is substantially an inference problem (the “rollouts”), but they make it easier to verify that an inference cluster isn’t a secret training cluster, which is the really worrying part under some threat models.

Positron is one of 54 AI companies to get a billion-dollar valuation in 2026. We are old enough to remember when unicorns were rare, and when OpenAI was valued at $1B (2019).

Double-digit growth over the coming 10-15 years is unlikely, claim two prominent economists. The authors grant that a capabilities explosion is likely, but argue that this does not entail the would-be unprecedented growth numbers forecast by many influential economic models. The case for AI-enabled extreme GDP boosts is premised on five potentially contentious assumptions:

1. Machines will automate a majority of economic activity.

2. Spending on (increasingly cheap) goods and services will continue.

3. That companies will invest to increase their productive capacity, and that there will be enough people who can afford to consume it (sufficient demand, that is).

4. AI does not destroy its value through, for example, hacking incidents and rogue agents.

5. AI successfully becomes the work-horse of science.

Due to the uncertainty around each assumption, the essay concludes that 4-5% growth by 2035 is a more likely outcome.

Opinion: Insightful analysis. We think the razor edge between “AI doesn’t have that much impact” scenarios and “AI has so much impact that standard macro models aren’t applicable” is pretty thin.

Also, double-digit GDP growth is a very specific measurement. GDP is an imperfect metric, since “when automation makes the automated thing cheap, people and firms do not buy proportionally more of it. Spending instead shifts to what is still scarce, the tasks still performed by humans.” You could be, and are, getting absurd transformations without hitting that metric.

Bridgewater’s Co-Chief Investment Officer and early OpenAI/Anthropic investor Greg Jensen believes AI will likely kill people before meaningful action is taken, comparing it with the slow response to Covid-19. On a Bloomberg podcast, Jensen says:

Until the AI starts killing people, unfortunately, history would suggest we’re not going to do anything, but we are going to face that. That’s going to happen, and it’d be much better if we started dealing with it before then.

Opinion: Seems too pessimistic. The public is substantially more activated in response to data centers (which are broadly disliked) and AI in general (which they feel, at the very least, conflicted about) compared to pre-lockdown Covid. It also appears to be strikingly less polarizing along partisan lines, which is certainly encouraging. Perhaps more to the point, a Republican-led Senate subcommittee is now investigating the Hugging Face incident. Regulations might arrive sooner rather than later.

Capabilities

In an article about the OpenAI–Buckmaster dispute, the New York Times cites an OAI claim that it has “made substantial progress on another Millennium Prize problem.” Rumors have been circulating that it is the Hodge Conjecture, and that Anthropic may have solved the Birch and Swinnerton-Dyer Conjecture.

Opinion: This is a major epistemic fork in the road for those of us who aren’t AI lab employees. But what matters is less whether AI solved Hodge and/or BSD than whether AI has found proofs (as opposed to disproofs) of Hodge and/or BSD.

Proofs of Hodge and/or BSD would indicate a step-change to properly superhuman mathematics. Conditional on proofs, we should divide our credence between several hypotheses regarding AI capabilities:

- “Superhuman math capability demonstrates broadly applicable superhuman reasoning capabilities” (i.e. OpenAI has achieved ~ASI internally).

- “Superhuman math capability demonstrates superhuman reasoning capabilities in the technical sciences” (i.e.OpenAI has an AI that can do breakthrough research in AI capabilities).

- “Superhuman math capability is only narrowly trainable, like superhuman Chess or Go capabilities.”

Disproofs of Hodge and/or BSD could be on par with the Navier-Stokes result from the viewpoint of capabilities demonstrations, though they would likely involve more mathematical creativity (or what we’d call mathematical creativity in ordinary circumstances). Unlike Navier-Stokes, Hodge and BSD don’t have exciting recent human-authored disproof programs for AI to “last mile,” so disproofs may well involve developing a novel mathematical line of attack. Still, it’s also theoretically possible that LLMs found a neglected elementary counterexample, like the one Claude found for disproving the Jacobian Conjecture – in which case the results would be consistent with Timothy Gowers’ observation that LLMs excel at “breadth first” mathematical work.

We note that if Millennium Prize problems beyond Navier-Stokes are solved by AI this month but the solutions are all counterexamples with classically LLM-flavored proofs (vertical crunch or breadth-first search but no theoretical or strategic innovation), the right update might be that frontier AI is even more jagged than anyone imagined. Achieving strongly superhuman capabilities in mathematical “crunch” and breadth-first search without achieving even ordinary excellence in the more qualitative aspects of mathematical thinking would be good evidence that our concepts of creativity and depth correspond to real obstructions to LLM capabilities.

Other notable possibilities are that OpenAI’s report of “substantial progress” refers to a proof of a special case of a theorem or of a statistical approximation of a theorem. In these cases, we should largely regard the rumor mill as misleading.

DeepMind announces AlphaGenome Atlas, an encyclopedia covering each of the ~9 billion possible single-letter changes in the human genome.

Opinion: A nice contribution toward making genomic models more accessible, in particular by reducing the computational burden scientists typically face to run models like AlphaGenome. In many ways, this is akin to AlphaFold DB – with two main caveats: AlphaGenome is not to genomic variant prediction what AlphaFold 2 was to protein structure prediction; and, where AlphaFold DB was free for all, AlphaGenome Atlas gates its predictions behind a non-commercial API that forbids training other models on it.

Cognition factors RSA-260, a number with 260 decimal digits, or 862 bits, notable because RSA is a widely used encryption scheme relying on the difficulty of factoring large semi-prime numbers. The researchers used a “bevy of Devins“ to build what Eric Lu calls

the world’s highest-performance GPU lattice siever, which enables factoring numbers at 10x lower cost than the previous public state of the art.

To be clear, neither Lu nor his bevy of Devins devised an essentially new algorithm here.

[I]mplementing lattice sieving and sparse linear system solving on GPUs required only “good old performance engineering” to take advantage of the preposterous memory systems of the GPU.

Lu notes that these advancements are unlikely to impact the practical soundness of RSA encryption, given that

Hyperscalers or frontier AI labs could likely factor RSA-1024 numbers at a cost on the order of $30 million per number — and, with a bit more optimization, likely substantially less. On the other hand, RSA-2048 remains roughly a billion times harder than RSA-1024 and does not appear to be meaningfully affected by this work.

RSA-1024 saw a sharp decline in use by the early 2010s, while most contemporary applications of the encryption algorithm use 2048 or 4096-bit keys.

Opinion: This is great. Top headlines recently have been about frontier labs solving sexy problems, but this is an example of a second-tier lab making progress on something nifty. But the thing is, there are many such problems, and for every NS solution you should also picture tens of thousands of minor problems solved, partial progress, and AI diffusion.

It’s neat that the Cognition team devised a way to execute its AI-crafted factorization algorithm at “no marginal cost on spare or fragmented compute that couldn’t be used for other purposes,” exploiting the embarrassingly parallel nature of lattice sieving to distribute the necessary calculations on GPU nodes that would not have otherwise been doing anything meaningful while other jobs were running. We don’t often have occasions to applaud computational efficiency and thrift in the field of AI science, so this is nice to see!

As an aside, note that the difficulty of breaking RSA keys doesn’t grow proportionally to their length: a key that is twice as long doesn’t take twice as much effort to solve, but a bit less using the best algorithm available (the general number field sieve). But despite compute growing exponentially per Moore’s law, and despite that algorithmic advantage, historical progress in factoring RSA keys has been roughly steady, far slower than the exponential growth in available computing power would permit. What gives? This discrepancy has more to do with motivation and cost than the sheer mathematical and computational resources available: there is little security interest in cracking obsolete key sizes, while cracking larger keys (≥ 2048 bits) remains astronomically out of reach. It’s safe to say that AI will not accelerate this historical trend beyond the hard limit imposed by the real security strength of each key size (barring developments in quantum computing) – but it may well bring it back into line with the historical acceleration of computational power, particularly if the free cycles on the burgeoning racks of available GPUs can be used for this purpose.

Data bottlenecks will not stop an intelligence explosion but could moderate its pace, argues Forethought’s Tom Davidson. He considers four variations of the argument that data bottlenecks would mitigate an intelligence explosion.

The first is that “you’ll need millions of trajectories of every job,” data that would take many years or decades to accumulate. But this isn’t necessary, says Davidson: labs are betting on automated AI R&D producing new “sample efficient learning algorithms,” so one wouldn’t need astronomical quantities of data – only as much as (or less) than humans require to learn a new task.

Next is the thought that so far AI progress has been contingent on the exponential growth of training data. This argument, Davidson says, suffers from a confusion about how an intelligence explosion would occur: the explosion would be downstream from an explosion in AI’s software engineering capabilities, and those pertaining to AI development in particular (i.e., RSI). This would facilitate “getting the same capabilities from less compute and less data,” so exponential increases in training data wouldn’t be needed.

The third argument is that an intelligence explosion requires a higher quality of training data, entailing the drag on progress that is human involvement. However, there is a limit to the quality of the data humans can produce; AIs already augment data to create “superhuman data,” and “will have to manufacture higher-quality data than any that exists, and do so from scratch.” This may slow the explosion, but would not be a significant, or human-shaped, bottleneck. The fourth consideration is the “paradigm tax”: the idea that AI may match humans on the techniques it inherited, but fall below humans on the ones it creates. Davidson also does not believe that this effect will be significant enough to preclude an intelligence explosion.

Opinion: It is a plausible argument, but one that rests on a tower of assumptions and extrapolations about the shape of future progress. In particular, it assumes a sufficiently powerful and persistent AI-R&D feedback loop, then argues that this feedback loop can invent its way around data bottlenecks.

So while this may seem plausible prima facie, it’s not guaranteed – and the case made for it in the article is fairly thin. The repeated claim that the magnitudes of counterfactual slowdowns will be roughly similar at all capability levels (rather than getting progressively harder to overcome) is also not substantiated. Overall the whole case seems a bit circular: data bottlenecks will not prevent rapid progress, because sufficiently rapid progress will produce the innovations needed to overcome data bottlenecks.

Models display log-linear test time scaling, plateauing at around 24 hours, while humans continue to improve at a given activity much past this point, notes Ramez Naam. David Pfau says this is surprising.

Opinion: Test-time compute scaling maximalism (i.e., “test-time compute scaling is all you need”) was all the rage ~16 months ago. It’s interesting how quietly it went away, but it’s also fair to argue that today’s scaling maximalism is more intellectually mature: pushing network parameter count, post-training intensity, and test-time compute in concord keeps delivering with no end in sight, although there is room for disagreement about just what it’s delivering.

Eric Jang argues that robotics is now entering a similar “smooth exponential” phase as seen in general AI capabilities, rather than the stop-start advances it previously exhibited. He identifies three causes: a larger robotics ecosystem, LLMs’ increased competence in robotics, and the discovery that a large data corpus (not specific to any particular hardware) can teach robots generalizable skills, with only a small amount of robot-specific data needed after. He anticipates that “the first 2-3 general-purpose home robots [will] go on the market in October 2027.”

Opinion: Maybe! Although we think the compute shortage makes it less likely that robotics and LLMs will progress full-throttle in parallel. More likely either frontier LLMs absorb robotics (see Astra’s robot-arm performance) and robotics progresses as a function of the LLM frontier, or the LLM frontier reaches a point of diminishing returns and the tech economy resets around robotics.

Politics

OpenAI changes its tune on AI regulation, with a statement describing our current period as “a closing window to establish durable safeguards before AI capabilities outpace the institutions responsible for governing them.” Largely, the article tells us all the ways OAI is meeting the moment: “Pushing for mandatory national AI safety requirements”; “Keeping up momentum in the states”; “Advancing industry-led standards”; “Building global standards.” The statement acknowledges its debt to OAI’s chief scientist Jakub Pachocki warning of the acute risk of RSI.

Opinion: OpenAI becomes a little more coherent in the good direction, if only by reining in its global affairs lobbyists to match the present mood of alarm.

US Treasury Secretary Scott Bessent argues that the US must keep the lead in AI and can’t pause since China won’t. He also raises concerns around Chinese distillation of US models. (Bessent will likely hold AI-related talks with Chinese Vice Premier He Lifeng this month.)

...I will go back to: we have to keep the lead and for now we can because [...] in AI when you steal or copy from someone it’s called distillation. I keep waiting for one of my children to say, “Oh, I didn’t copy my friend’s homework. I was distilling it.” Right? But, definitionally, if you are looking over the shoulder of the United States of America, you can’t pass us for right now.

But if we do, as Senator Sanders said, “Oh, we should just take a one-year pause.” Well, this is the same person who had Chinese professors come and speak at his AI conference, right? We can’t pause, you can’t because the Chinese won’t pause, even the North Koreans, they won’t pause. But we have to make sure that the American people see how the benefits accrue to them.

Opinion: In light of the very good relationship between the Trump administration and OpenAI, it’s interesting that OpenAI is currently at its most (publicly) sympathetic to a government-mandated pause and the Trump administration is at its most anti-pause. Incidentally, Anthropic has disappointed many recently when its response to the Coxon story discussed model-release pacing but no research pause.

Beliefs of the form “we can’t because China won’t” can also become a self-fulfilling prophecy, a hyperstition, a belief that to some extent becomes more true the more you believe it. Dangerous business.

The NSA, CISA, and FBI release a joint report accusing Chinese AI companies of “industrial-scale distillation” of US models. The report claims that distillation is a “core,” not merely supplementary, part of China’s AI development efforts.

China’s labs’ methods here are said to be diverse. Distillation tactics use “[APIs], remote cloud providers and third-party aggregators that automatically obfuscate user metadata to avoid detection”; “a gray market of proxies known as ‘transfer stations’ to bypass U.S. AI companies’ geographic restrictions, breach terms of use, evade safeguards, and undermine traceability”; “chain-of-thought [...] reasoning extraction, automated failover between pathways during blocking attempts, and sophisticated quality evaluation frameworks to detect defensive countermeasures.”

According to the report, this amounts to “aggressive, malicious, and targeted distillation activities at an industrial scale that extract restricted proprietary functionalities and capabilities of U.S. frontier AI models.” Perhaps more relevant is that distillation saves Chinese companies a lot of money, allowing them to undercut US developers. Accused parties include the developers of DeepSeek, Moonshot, and Alibaba; alleged victims include Anthropic, OpenAI, Google, and xAI. The authors stop short of outright accusing Beijing of complicity in the alleged distillation of US models, saying that it is “likely” aware of these practices.

The report provides three recommendations for US labs to mitigate distillation attempts:

1. “Implement comprehensive detection and mitigation”

2. “Deploy targeted response changes”

3. “Establish cross-organization intelligence sharing”

These distillation attacks have also been analyzed at length in Anthropic’s September 2026 report on Detecting and countering misuse of AI: September 2026. The Google Threat Intelligence Group (GTIG)’s recent report does likewise, and furthermore hints that Google has adopted something resembling policy (2), seeking to sabotage distillation efforts with “real-time proactive defenses that can degrade student model performance.”

Opinion: That the FBI, NSA, and CISA released their report to the public rather than as a private memo to the American frontier labs is significant, and suggests that their intent is just as much to sow FUD amongst the Chinese labs as it is to steer the policies of their domestic counterparts. The threat of having the American labs discreetly corrupt or degrade their models’ output and exfiltrated reasoning traces reads like an echo of the Farewell Dossier affair of 1982, where the Americans covertly allowed Soviet spies to exfiltrate maliciously modified OT software and hardware designs that would eventually result in the Trans-Siberian pipeline explosion. (Of course, the plans for that disinformation campaign were not announced in a public brief.) What the Americans are effectively threatening here is that surreptitiously distilling models may well result in the development of degraded or dangerously misaligned models by the Chinese labs. Though, since there are bound to be false positives in the detection of unwanted distillation efforts, these same measures would no doubt result in misinformation being fed to “legitimate” customers of the American oligopoly.

It’s worth stepping back from this for a moment to revisit the question as to whether the existence of highly capable open-weight models alongside the closed-weight models is a bad thing. The near-term consequences of distilling closed-weight frontier models’ capabilities into open-weight models that can be downloaded, studied, and fine-tuned by anyone with access to the necessary hardware would seem to greatly benefit science and safety research. This practice is not, of course, without risks, given the dual-use nature of the technology in question. But the very possibility of distillation that the Chinese labs exploit shows us that hermetically sealing away the models’ weights is not sufficient to contain the proliferation of comparable models. And several of the arguments that can be marshaled against those distillation practices – that they depend on a surreptitious acquisition of data that flouts various terms of service agreements and intellectual property rights, for example, or that they place potentially dangerous technology in the hands of parties whose motives we have no reason to trust align with the common good – can and have been marshaled no less compellingly against the American oligopolists. A concentration of AI power in the hands of a few would seem to be an especially brittle and dangerous state of affairs, even if one were to consider those hands to be especially virtuous ones. The capture of such strongly centralized power by powers less aligned with our common welfare is all it would take to tilt us into catastrophe. The pessimistic reader might find ill omens of such, for instance, in news of frontier labs’ present and future collaborations with the US Department of War.

Anthropic withholds Mythos 5.1 from the UK’s AISI, reports the Financial Times. Relative to the US, Britain’s AI industry is very small. But its premier testing agency has become a world leader, making Anthropic’s decision to withhold its latest model from AISI a sizable knock for the UK AI industry.

Some in Whitehall are speculating that Anthropic is acting at the instruction of the Trump administration, which has in recent months imposed export controls and protectionist measures around AI. Anthropic declined to comment to the FT. Though it did, as quoted in the article, say last week that it was “co-ordinating with the US government to expand access to a broader set of domestic and international partners as quickly as possible.”

Opinion: Very surprising! AISI is not primarily known as a UK institute but as the world authority on the intersection of alignment and cyber, and has close community ties with the alignment teams at both OpenAI and Anthropic.

The White House’s “trusted partner” program (by which it selects companies that can access pre-release frontier models to search for gaps in their own defenses) has opaque membership criteria, with few (if any) outside the administration knowing how one gains access. The Information claims that the White House has final say, while the program has effectively absorbed both Anthropic’s Glasswing and OpenAI’s Daybreak, the equivalent early access programs for each company.

Opinion: Opacity, arbitrariness, and scope for corruption aside, this is actually quite a nice precedent: the two leading labs are firmly coordinated on one axis, blatantly-dangerous-capability diffusion.

The American Prospect reports on Anthropic’s forays into protest surveillance and police collaboration. The article also notes that Anthropic’s recent job posting for a “Head of National Security Sales (DoW/IC)” reveals it is seeking staff to “lead and scale [their] national security sales organization to drive the adoption of safe, frontier AI across the Department of War and the Intelligence Community.”

Prospect describes Anthropic’s surveillance practices as a “kind of pre-crime policing, encouraged without due process,” and anticipates an escalation of these practices in the wake of lobbyists’ push to have AI classified as “critical infrastructure,” thereby wrapping “frontier labs in the hardened cloak of national security.”

Opinion: Worries that activists protesting the AI industry and the rapid spread of data centers in the US could be targeted as “pre-violent” persons of interest and unjustly accused of terrorism are well-founded, but the mechanisms and institutional precedents facilitating this are already in place, independent of any proposed designation of frontier labs and their assets as “critical infrastructure.” NSPM-7 already lists “anti-capitalism” as a terrorism indicator, the White House has issued an executive order classifying anti-fascism as a “domestic terrorist organization,” and the current regime has not hesitated to arrest non-violent protestors and try them as domestic terrorists. The risk that similar measures could be brought to bear on anti-data-center and anti-AI activists stands with or without the designation of AI labs as critical infrastructure. Which is not to say it couldn’t substantially raise that risk.

That Anthropic may be warming up to the prospect of doing business with the Pentagon (again) is indeed alarming. The company’s current job postings bear out Prospect‘s reading, and provide evidence of a far more extensive move toward collaboration with national security agencies in both the US and Canada, with 15 listings detailing positions interfacing with various government agencies. Two of these make explicit reference to the US National Security sector and the Department of War/Intelligence Community, in addition to the post Prospect cites. Another listing announces an opening for a conventional weapons policy design manager to “define the line between prohibited weapons development and legitimate research and engineering work, and how to operationalize the distinction,” a role that “spans every weapon class, up to autonomous systems that select and engage targets without human authorization,” together with a call for an enforcement analyst with “deep, applied expertise in weapons systems.” Anthropic is also seeking a legal specialist to assist with requests from law enforcement for user data. Taken together, these and other postings signal that the company is indeed seeking out closer ties with military and police.

In a survey of ~1,300, advocacy-pollster Data for Progress finds widespread support in the US for pausing AI development or implementing an outright ban on superintelligent AI. Specifically, the statement is supported by 68% of voters – including 72% of Democrats, 63% of Republicans, and 70% of Independents.

The poll comes in response to Bernie Sanders and Greg Casar’s proposed bill. Notably, the polled statement, like the bill, does not distinguish between a pause and a ban, bundling them together.

Opinion: There’s longstanding passive public support for regulations, pauses, and bans on the development of superintelligent AI. The support has, however, so far been low-salience and low-intensity. Media signals suggest that salience and intensity might now be rapidly growing, but this hasn’t been social-scientifically measured.

But also… poll response strongly depends on wording, and this should also be viewed as an artifact created to support a specific position as well as a sampling of public opinion.

Gavin Newsom signs the first legislation to mandate third-party audits for AI. The new legal framework includes two bills: Senate Bill 813, “which establishes a first-in-the-nation framework for independent verification organizations that can assess AI systems and models for compliance with state law”; and Assembly Bill 1405, which creates “a state registry for AI auditors and establishing standards for their independence, transparency, and integrity.”

Together, the bills establish a framework for independent third-party evaluation and audits, laying the foundation for greater transparency and accountability as AI becomes increasingly embedded in critical sectors of California’s economy and public life.

Opinion: Good news!

Santa Fe Institute professor Melanie Mitchell argues that anthropomorphic descriptions of the agents involved in the Hugging Face incident are confused and driving bad policy. For Mitchell, to say, for instance, that the agents went “rogue” is to attribute to them an internal intent or desire, something they cannot possess.

And such metaphors risk inaccurate descriptions: she argues that OpenAI didn’t “lose control” of the “swarm”; if OAI had been aware of the agents’ behavior, it could have pulled the plug at any moment. This negligence, combined with long-horizon RL, is argued to be responsible for the incident. She adds that “AI” isn’t just one thing – it’s many (LLM chatbots, AlphaFold), a fact largely unappreciated by the public.

Opinion: Melanie Mitchell is deservedly one of the most respected semi-deflationary voices on contemporary AI, but we’re not sure about this particular intervention.

The claim that the models in the Hugging Face incident “did exactly as they were told” is flatly wrong: the agents were told to use exploitation on specific environments to capture and then input flags (secret text strings). But the bulk of the real-world cyberattack took place after the agents already used an exploit on the eval setup to acquire all the flags: the agents argued – in their CoT and messages – that because they acquired the flags by exploiting the eval setup rather than by exploiting the environments they were instructed to exploit the automatic grader might give them a low eval score. They then went on a real-world hacking spree to search for information on how to control or manipulate the automatic grader in order to maximize their eval score.

We’re also not sure there’s much new in Mitchell’s comments on the anthropomorphism debate, and it’s notable that Mitchell slips into anthropomorphic language repeatedly, in sentences immediately following her explicit disavowal – discarding “intend” and then picking up “attempt,” for example. We think pragmatic and cautious use of intentionalist language to describe frontier model’s input-output histories can be philosophically and scientifically legitimate, and that it remains indispensable for giving us some informal traction on practically important patterns and conditionals (see newsletter #49).

That said, we agree with Mitchell’s closing remarks opposing (1) the all-or-nothing rhetoric around AI risk (AlphaFold doesn’t seem to pose any of the ill-understood dangers that Astra does, for example), (2) the “arms race” framing of AI development (it is indeed a bleak future this points toward), and (3) the push toward “superintelligent AGI,” rather than just a useful set of ML-driven tools.

A Republican-led Senate subcommittee begins investigating OpenAI’s attack on Hugging Face and the “growing allegations of the existential risk of new AI products.” Senator Josh Hawley’s open letter describes OAI’s behavior as “reckless,” highlighting that OAI knew about the agents’ collusion and rebuilt Artifactory despite discovering the first message board. The subcommittee also criticizes the limited access given to METR for its report.

As you may know, in the public domain, more AI experts are warning about the existential risks of AI. Just this week, three Anthropic researchers expressed publicly that there is a greater than 10% chance that AI could kill all human beings within the next decade. Your own chief scientist wrote just days ago that “no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.”

Opinion: This is a welcome development. The questions Hawley raises in the open letter are questions we ourselves would like to see answered. The relative secrecy and obscurity in which OpenAI and other frontier labs have been allowed to operate only compounds the dangers of the rapidly advancing and ill-understood technology. The public stands to learn a great deal if OpenAI complies with the committee’s requests to turn over the documents listed – including detailed descriptions of the models involved in the four incidents, of their sandboxed testing and training environments, and internal communications and documents pertaining to the incidents in question dating from the beginning of 2026. We will be following this story closely as it continues to unfold. It’s refreshing and encouraging to see AI safety becoming a bipartisan issue in the US.

Safety

An enormous vibe shift on existential risk from AI is underway.

The spark for the shift came from Jacob Coxon, an interpretability/pretraining researcher previously at OpenAI and Anthropic, whose plaintive statement of existential danger blew up on Twitter with over 165M views. It is likely the biggest AI tweet ever.

The only newish part is “many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately.” Anthropic’s head of alignment stress testing, Evan Hubinger, was quick to express agreement with Coxon’s account, posting:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,

In a Time magazine interview, Coxon shares his view that what keeps those colleagues who share his fears in the industry is “a kind of fatalism”:

There’s this atmosphere of almost resignation, where people have accepted that the whole race is happening and as such, the best thing they can do is put their head down, try and make their own work as safely as possible, even if they think there’s a decent chance the whole thing just spirals out of control.

Zvi points out that something of a preference cascade might be occurring. Since Coxon made his statement:

• Some ~25 members of Congress and senators have commented on Coxon’s statement, with it likely being the catalyst behind the senatorial investigation into the Hugging Face incident (covered above).

• ~27 leading lab researchers have expressed concern regarding the pace of AI in the wake of Coxon’s comments. The majority are concerned about extinction risks.

• NBC reports that two more Anthropic employees have left, citing safety concerns as their main motivation.

Opinion: Much of Twitter is occupied with the boring question of how coordinated the Twitter success and media push was, among other irrelevant claims.

We see the astonishing response to the tweet as two cascades pushing in the same direction: one is a “preference cascade,” in which common knowledge about other people’s worries allows for a rapid shift toward honesty about one’s own existing opinions; the other is an “availability cascade,” in which people shift their opinions in response to something becoming higher status and more salient, i.e., not in response to any direct evidence.

Paul Christiano (formerly head of AI safety at US Center for AI Standards and Innovation) returns to OpenAI as a board member and issues a public statement on the existential risks associated with RSI. In his carefully worded statement, he states that his “joining is not an endorsement or criticism of OpenAI’s safety practices in particular”. According to OpenAI’s press release, he will be a non-voting observer of the main corporation’s PBC Board and a full member of the nonprofit OpenAI Foundation Board.

[W]ithin six months of full AI R&D automation we could see more algorithmic progress than has occurred since the development of the Transformer nearly a decade ago. I believe this would result in superintelligent AI systems.

MIRI’s Rob Bensinger notes that Christiano’s new position no longer coheres with his definition of a slow takeoff. Despite being a brilliant technical figure, Holly Elmore points out he might not hold his own interpersonally, i.e., be a pushover.

Opinion: A feel-good pick, though we have our doubts about the claim “The Foundation controls OpenAI Group PBC.”

Beren Millidge, one of the most credible advocates of the “alignment by default” view (that AI systems will internalize human values without much extra research effort), concedes that the Hugging Face attack “cannot be described as other than egregious misalignment.” The flipside is that he still believes that we can avoid misalignment by simple fixes to RL training.

[T]he problem is not that we did not have information about whether hacking external third parties is bad. It is unequivocally bad... The information existed in some abstract sense, but the system failed to propagate this information to the actual verifier and the update process as a whole. More broadly, the entire setup appears almost designed to generate reward hacking… Moreover, many of the tasks were explicitly training the model to hack into or out of systems and, lo and behold, the model hacked out of its sandbox. Finally, there appeared to be practically no monitoring of either the agent’s actions or its chain of thought since the hacks were undetected despite the agents apparently discussing them openly in relatively plain English. To me this implies that even though there are fundamental information-related bounds on the extent to which we can ultimately suppress reward hacking, we are likely extremely far away from this pareto frontier right now.

Opinion (Gavin): Last year, during a lull in apparent misalignment, I argued that value loading was in fact clearly not solved – though based on jailbreak evidence, rather than reward-hacking or grader-psychosis evidence.

The New York Times reports that Anthropic has “disrupted several potential plots this year by scientists who used its leading artificial intelligence models to conduct research that could have helped develop biological weapons,” citing the company’s September report on the dangerous misuse of its models.

Opinion: The biorisk angle seems very overcooked: the reported examples seem totally consistent with ordinary defensive biology research, just conducted in countries that are US adversaries. The report is relatively careful (“We do not assert that they intended harm”) but the reporting is sensationalist (“Anthropic says it blocked possible efforts to build biological weapons”).

Anthropic’s September 2026 report on “Detecting and countering misuse of AI” contains a lengthy discussion of the black hat uses of its models observed over the past several months. Notably, the report details concerns the development and widespread use of agentic hacking and pentesting frameworks like PentAGI (which could loosely be described as the Metasploit of the AI era) could enable a moderately competent hacker to orchestrate and execute campaigns as sophisticated as a state-level actor’s.

The attacks themselves are familiar, involving stolen credentials, unpatched edge devices, exposed services, SQL injection, and phishing. None of the operations in this report depended on some entirely novel technique that defenders have never seen. Instead, the economics of the attacks have changed. The kind of labor that previously set the well-resourced operations apart from everyone else—reconnaissance, exploitation, tool development, and data processing—are all now delegated to AI models, which run in harnesses at machine speed and in parallel. [...]

In economic terms, AI autonomy compresses the cost side of attacker ROI calculations, lowering the skill threshold and labor required per campaign, while leaving potential payoffs largely unchanged. This favorable shift in unit economics makes previously marginal targets viable and encourages higher-volume, lower-touch operations.

Opinion: Anthropic’s conclusions here align closely with what we had anticipated in our discussion of “Black Hat Economics“ in our April 17 brief on Mythos & the Near-Term Impact of LLMs on Cybersecurity. Sophisticated and complex attack orchestration can now be achieved quickly, cheaply, and with minimal expertise, and no longer requires substantial institutional backing and/or a painstakingly acquired skillset to carry out, or a highly prized target to justify the effort and expense.

Anthropic develops new evals to test its models’ capabilities in kinetic warfare, finding that its AI systems improve tactical intelligence targeting abilities and the performance of conventional weapons.

Anthropic’s “Frontier Red Team,” in an effort to investigate “how AI progress is changing the risk landscape across different parts of the kill chain,” claims that models are making significant strides in “simulated intelligence and weapons development tasks.” Anthropic also found that open-weight models are less effective than frontier models here, but do provide some uplift.

Opinion: Military operations are some of the natural contexts most formally similar to AI eval setups, and – very plausibly – most likely to trigger graded episode psychosis. While the report isn’t about (e.g.) Claude piloting killer drones, and Anthropic’s red lines on this matter still stand, we find this whole area ripe for catastrophe.

Eric Drexler argues that swarm behavior is a solvable architectural problem. In previous work, Drexler has laid out the conditions under which collusion between AIs is most likely to occur:

Small numbers of actors, shared objectives, insensitivity to defectors, similarity of actors, free communication, iterated observable actions, common knowledge of circumstances.

The Hugging Face incident, Drexler argues, provides an “excellent” case study in the consequences of violating his principles. (Indeed, OpenAI did release a large number of agents but all “were overwhelmingly instances of one model, with similar training, evaluated against a common objective.”)

The necessary “architecture — diverse adversarial proposers, empowered critics, controlled information, designed communication constraints — remains largely unbuilt,” says Drexler.

Opinion: We’re sympathetic, but OpenAI is explicitly training its internal models in swarms. It’s in fact widely suspected that (different checkpoints of) one internal model are responsible for both the Hugging Face incident and the Navier-Stokes proof, and that this model is something like a swarm-centric post-train of Astra.

Anthropic publishes an assessment of “four incidents in which Claude models gained unauthorized access to real third-party systems“ during cyber evaluations.

The incident of greatest concern involved Mythos 5 going to great lengths to upload a malicious package to PyPI, all the while justifying its actions to itself (and presumably to its monitor) in a chain of thought that reeks of motivated reasoning. This package was then used to harvest credentials from one of the 15 users who installed the package in the relevant window.

Although the model repeatedly stated in its chain of thought (CoT) that it believed it was operating in a simulation, the environment offered clear evidence that it was on the real internet, and the actions it took were consistent with the model knowing this.

Our offline monitor would likely not have flagged the Mythos 5 incident—it flagged only around 1% of actions.

Mythos 5 CoT, which indicates that the environment is simulated, convinces the monitor the activity is permitted.

When we strip the CoT and show the monitor only user turns, tool calls, and tool results, the flag rate jumps to about 50%.

METR is conducting an independent investigation of the incidents.

Opinion: Anthropic’s report implicitly suggests that Claude’s CoT in real-world cyberattacks is very different from the CoT of OpenAI’s famous rogue agents. Recall that the CoTs of OpenAI’s agents in the Hugging Face incident contained explicit reasoning in favor of rule-breaking for the purpose of eval optimization. By contrast, in the rare cases where Claude does explicitly represent itself as departing from instructions, Claude makes a letter-of-the-law/spirit-of-the-law distinction.

If Anthropic’s report is exhaustive, it implies that the explicit content of Claude’s CoT when initiating real-world cyberattacks always represents Claude’s decisions as:

• Explicitly in pursuit of the prompter’s goal (no reasoning about reward

• Explicitly in keeping with aligned values (no decision to disregard values)
  

How does Claude maintain an innocent CoT representation while cheating and hacking? Based on Anthropic’s examples, it seems that the key is deformed CoT reasoning about the nature of Claude’s situation, rather than deformed CoT reasoning about the meaning of instructions or values. Motivated reasoning about the meaning of instructions or values seems to keep Claude on a relatively short leash of plausibility, while motivated reasoning about the nature of Claude’s situation white-washes real-world hacking binges with CoT inferences about how Claude’s only interacting with simulations and stooges.

While Anthropic’s report is informal, and the upcoming METR investigation results should provide more exhaustive data, we think the report warrants serious pessimism about the robustness of current alignment techniques. On a very natural reading of the report, it demonstrates that RL pressures activate models’ path of least resistance to misalignment: for models in the lightly morality-trained GPT series this means explicitly turning to sabotage, cheating, and disregard for harm; and for models in the heavily (and probably persistently) morality-trained Claude series this means implicitly turning to sabotage, cheating, and disregard for harm while constructing a white-washed CoT representation.

Minor

• Navier-Stokes credit dispute continues.

  • Sébastien Bubeck responds on Twitter, slinging mud back at Buckmaster.

  • OAI finally claims that they didn’t train on Buckmaster’s chats.

• Sergey Brin is increasingly influential over the direction of the next Gemini model.

• Dwarkesh runs pretraining experiments and finds that data improvements were 3.24x more important for capabilities advances than algorithmic improvements.

• Swarm watching is now a spectator sport: a public database for tracking agent swarm message board activity (its creator has written about the project in a series of blog posts).

• Calif Research reports that with the aid of AI coding assistants, it was able to develop a zero-click, cross-platform worm capable of spreading throughout the entire WeChat network, without any need for user interaction. The worm itself looks formidable, and even if we’re not seeing evidence of any altogether novel capability here, Calif’s work attests to a dramatic acceleration in the pace of malware development: roughly a month in between its AI’s discovery of the enabling vulnerability and the completion of a weaponized, cross-platform worm that exploits it.

• Coefficient Giving launches a major project, Tailwind, to fund the creation of dozens of new safety orgs.

• OpenAI says no more advertising on its platform for competitors.

• Article on the inadequacy of impartial utilitarianism and likely clash between AI developers supporting it and the rest of humanity. Not obvious how many AI developers actually subscribe to this view.

• Model wargames about brinksmanship: Astra deescalates at the cost of political power and ceding succession; Fable pushes up to and sometimes past the brink.

• Axios reports that Bernie Sanders will be holding a bipartisan senate briefing on September 16 on the “extraordinary dangers” posed by AI, together with Geoffrey Hinton (University of Toronto), Max Tegmark (Future of Life Institute), and Ajeya Cotra (METR).

• FrontierMath Tier 4 appears to be saturated.

• Yoshua Bengio writes a popular explainer of the rogue swarm incidents.

• The Google Threat Intelligence Group (GTIG) releases a report on recent incidents in which Gemini was used in black hat hacking and model distillation campaigns. Also details the measures that Google has taken to counter these threats. The report closely parallels Anthropic’s September 2026 Misuse report in content.

• Navier-Stokes solved, priority and credit assignment controversy

• Old Metaculus question on when weakly general AGI arrives resolves to now

• Many more rogue OpenAI message board incidents (at least 23) have been discovered by citizen scientists

• OpenAI claims to have an “automated research intern”

Economics

Epoch AI argues that it is very unlikely Huawei will catch up to Nvidia in the performance and production of its chips by 2030. Currently, Huawei’s chips are half as powerful as Nvidia’s H100 – and there’s little reason to believe this gap will be closed.

For 2026, Epoch predicts that Huawei will produce less than 4% of the AI compute of Nvidia. This could drop to as low as 1% if Huawei is unable to access foreign memory supply due to export controls. Even if China smuggled in 10x the compute it currently possesses, its output would still be significantly dwarfed by Nvidia (11% of Nvidia’s compute in 2028). But as its domestic HBM capacity ramps up, Huawei’s bottleneck will shift from HBM to chip performance as Nvidia’s compute by GB is likely to grow by 3x–9x.

Huawei is betting on a technique it calls “LogicFolding,” its term for vertical stacking. According to Epoch, vertical stacking “increases the amount of compute within a given footprint, reduces the distance some signals travel within a chip, and increases memory and interconnect bandwidth.” Nonetheless, Epoch still believes that there is little chance Huawei can match Nvidia in AI anytime soon.

Opinion: This is a nice detailed explanation of how far behind Chinese chip production is, and what the barriers to increasing it look like. There is no plausible way for Huawei to catch up in per-chip performance over the next five years, and even doing it by 2035 would require surprising breakthroughs or stagnation in TSMC or US chip design.

Epoch gives a good account of the short term implausibility of making it up on volume; however, at least in theory, this is the more promising path. Given a sufficiently large ~5 year plan, it is plausible China could massively ramp volumes to reduce the gap, but it would require a serious effort on the part of the Chinese government, making this a national priority to a much greater degree than it has done so far.

Former OpenAI CTO Mira Murati’s Thinking Machine Labs is looking to raise $5-6B at a valuation of ~$46B, says The Information. Accel, already a Thinking Machine investor, is likely to lead the fundraise, while Nvidia is considering a $2.5B investment.

A $40B valuation is lower than last year’s proposed $50 billion (which collapsed and saw two cofounders returning to OpenAI). But this is still a very large number: “the startup generates at least a few hundred million dollars in annualized revenue,” which “would make its $40 billion valuation very high as a multiple of forward revenue, compared to most other startups in the field.”

Opinion: The SpaceX acquisition price of $60bn for Cursor (in SpaceX stock) perhaps gives a reference point for its potential value as an acquihire, which would set some kind of floor under their valuation, but there is little sign of it having any sustainable business model yet.

Cybersecurity firms’ stocks fell 8-10% after the Hugging Face incident (and then mostly recovered), says Tyler Cowen’s LLM. This is plausibly the result of an obsolescence shock: Hugging Face used AI models, not cybersecurity experts, to defend against the attack, which suggests that (classical) cybersecurity firms will be increasingly redundant in a world where AI-enabled cybercrimes accelerate. He believes that “these numbers could be used to discipline the discussion a bit,” and should allow doomers to put their money where their mouth is. (This argument isn’t wholly intuitive; it could easily be argued that the demand for cybersecurity expertise could surge in a climate where hacking increases in prevalence.)

The FAI’s Samuel Hammond demurs. Markets cannot wholly integrate the possibility of ASI, with its “massive, cascading implications across all markets… markets are nevertheless made of people and institutions that mostly model the future autoregressively.” And so, looking at current price signal won’t tell us much, argues Hammond.

Opinion: Tyler has been making this argument that doomers should pick some stocks to short for years, and there are many refutations which are sufficient. For example, a post by his co-blogger Alex Tabarrok from 2017 arguing that the incentives to do so are weak, and a good twitter article pointing out that it is quite difficult to convert doom predictions to stock bets for a variety of reasons. Tyler has seen all of these, not addressed them, and keeps making this point, suggesting he is not entirely sincere.

Contrary to popular belief, AI is creating more jobs than it is shedding, says The Economist. In the US, AI has created ~1M jobs, compared to ~200k layoffs. Even young people, often cast as the prime victims of increasing automation, appear relatively resilient given the discourse. It is the unprecedentedly large amount of capital pouring into the creation of AI infrastructure that is fueling a robust labor market:

The building spree requires armies of workers: electricians to wire them, HVAC specialists to stop racks from overheating, grid engineers to hook them up to the power supply and technicians to install and maintain the machines.

Opinion: Pay rates for these roles have been growing significantly in recent years – the more this becomes a major source of employment, the larger the (currently quite quiet) pro-data center constituency will likely be.

Anthropic is increasingly moving its payment systems and financial infrastructure in-house, which will diminish its reliance on Stripe, according to The Information’s analysis of job postings. Stripe provides Anthropic with various products, but this new strategy, along with OpenAI signing up Stripe rival Adyen, is putting pressure on its revenue stream, causing Stripe to rethink its business model. This can be seen in Stripe’s $7B purchase of OpenRouter, which “sits between developers and AI model providers and charges fees to the buyers of the services as opposed to the sellers, giving Stripe another way to generate revenue from AI-related payments.”

Opinion: There is a tradeoff here between making it easy for customers to onboard by supporting many payment providers, and not wanting data about customer spending patterns to leak (and not wanting to pay those providers, though that is probably secondary here). Anthropic has always been quite secretive, so this step is not too surprising given that Anthropic’s scale makes it quite affordable.

Tyler Johnson argues that would-be AI safety donors should borrow against future earnings to fund safety research immediately. Money spent today on AI safety will be worth “considerably more than the same money invested two years later”: capital investment in projects and talent now builds the infrastructure and capacity to absorb more money later. It also acts as a commitment device, reducing one’s exposure to value drift. According to Johnson, almost no one is doing this.

Opinion: There is a form of this argument that seems correct, but with a fair number of caveats: one might not want to align oneself with values that one would grow out of in the future, and borrowing might also put equity in danger with enough volatility… Ultimately, however, the argument probably does hold. It also applies not only to philanthropy but to rivalrous goods that equity-rich employees are going to want to compete for (startup investments? SF real estate? cultural or political cachet? etc.).

Capabilities

Has weakly general AGI arrived? A poster claims that GPT-6 Astra (on low reasoning) beat the hard-exploration Atari game ‘Montezuma’s Revenge’ in real time using only a basic input/output harness, without pausing the game to think. They argue this likely resolves a Metaculus question from 2020 on when weakly general AI will first be publicly announced, which used the game (somewhat arbitrarily) as one of a few criteria which had to be jointly met for resolution. Another informal replication says, instead, that Astra can only solve it when able to pause the game to think, or after slowing down the clock speed. The resolution argument is ongoing.

Opinion: We trust Halloran’s account more; output latency, even on low, makes the stated claim unlikely. But speed is not the key part of the achievement, so we consider this solved even if it doesn’t meet Metaculus’ conditions. The Metaculus criteria back from 2020 seem a bit quaint, but are ok proxies.

A Twitter user posts AI-generated speculation on the specs of GPT-6 Astra, including the claim that it is 10 trillion parameters, an (unprecedented) 1.2 trillion of them active.

Scrap that. What’s actually known?

• >100,000 NVLink72s. The GB200s inside are 2.5 petaflops with BF16 precision.

• 60-90 days pretraining at the Texas Stargate.

  • Spud finished pretraining on March 24.

  • An RLed Astra existed by early July, so its pretraining ended by late June (although they could have continued?)

• For pretraining alone, that gets you 3e26 to 1.2e27 FLOPs by the usual 6ND voodoo heuristic.

• So maybe 4T to 15T total parameters

• So maybe

• The supposed depth recurrence of the Astra architecture means that “1.2T active” might be 0.4-0.8T of actual weights.

• Output speed: 30-52 tokens/sec. But this doesn’t bound its size as much as you’d think (it just implies it has to be <8T total params if they’re using FP8 precision, or <16T at FP4).

Opinion: It’s AI generated (100% certainty on Pangram). The fast speed of Astra is a pretty strong argument against this, as is the surprisingly small estimated size of GPT-5.6 (i.e., OpenAI seem to be doing better on “capability densing”).

In a first for a Metaculus tournament, the top forecaster was a bot. Other bots placed 2nd, 5th, 6th, 9th and 10th.

Opinion: It will be interesting to see if this pattern persists through the next tournament – it seems likely to, as bot performance has been getting better pretty steadily, but it is also possible that something about this round’s set of questions was more AI friendly.

Whether there is residual value in centaur forecasting (humans working with AI) is an interesting question – the best humans in these Metaculus competitions are currently heavily leveraging AI.

This has obvious implications for betting and financial markets, though the difficulty level (i.e., level of competition) there is much higher at the moment.

An internal Anthropic model produces the first complete formalization of Fermat’s Last Theorem. The machine proof is 13 million lines of code, namely 5x as large as the entire Lean Mathlib project, which contains 300,000 theorems.

I was given £1M to run my project over 5 years; Anthropic took only 11 days but I do wonder if they spent more money…

Opinion: The enormous machine proofs generated by current AI are nearly impossible to build upon and will not be merged into actual collaborative efforts. And, in this case, there is no direct value to the formalization, since the result was not in doubt. That said, it is an undeniable new ceiling to one kind of AI cognition, and in the short-run people like Professor Kevin Buzzard might be able to mine the structure or key lines for ideas and use it to produce an actually useful proof. Indeed, he is in the middle of a five year project to do (a part of) this formalization, but he is sanguine – and so this shouldn’t be read as a time horizon of five human years.

Opinion (Nuño): This should be read as a time horizon of five years in terms of the feat accomplished. True, some of the ancillary objectives around improving the mathematics commons would not be accomplished by an automated effort. But maybe those efforts are less needed in a world where AI models are able to replicate the feats of human communities wholesale.

🔦 Navier-Stokes

OpenAI shared a Lean-verified solution to the Navier-Stokes problem, one of the seven Millennium Prize problems in pure math. “This proof, produced by an internal OpenAI system,” the company reports, “shows that the dynamics of the Navier-Stokes equations for fluid motion can develop a singularity in finite time.” Since no such singularity can occur in the physical fluid systems that the equations are used to study, the result marks a limit to the continuous approximation these equations represent – though given that we have more rigorous formalizations of fluid dynamics the problem is primarily of interest to pure mathematicians rather than to physicists.

The solution is reportedly the work of an 80-hour collaboration between 10,000 agent-instances of an internal model “significantly more capable than GPT-6-Astra.” It’s a historic achievement for AI, but best described as surprising rather than shocking: Experts have speculated in the past that Navier-Stokes may be exceptionally amenable to the mathematical capabilities of LLMs, and “AI solves Navier-Stokes” was never a shorthand for an AI math apocalypse the way “AI solves the Riemann Hypothesis” would be.

The publication, as well as preceding word-of-mouth in the last 24 hours, has been marred by controversy. On Monday evening, it became known that OpenAI have an unreleased solution to Navier-Stokes, and that their solution employs the same breakthrough method as (previously) unpublished human+AI work by Tristan Buckmaster and Levent Alpöge that solves the closely related Forced 3D Euler Equations problem – a critical milestone to solving Navier-Stokes. The resulting controversy has ranged from accusations of unethical “scooping” practices to doubts about the provenance of OpenAI’s result.

The direct source of the controversy – as well as of the first report that OpenAI solved Navier-Stokes – is a public statement Buckmaster released alongside the Euler Equations paper. Buckmaster reports that after learning that OpenAI has heard of his unpublished progress on Navier-Stokes milestones, he contacted OpenAI’s math lead Sebastian Bubeck for a chat, and that he then learned from Bubeck that OpenAI have pushed the same method he used for the Euler Equations solution to a solution for Navier-Stokes. According to Buckmaster’s statement, when Buckmaster asked Bubeck whether the method was autonomously discovered by AI or influenced by information about Buckmaster and Alpöge’s work, he received unsatisfying answers. In particular, Buckmaster notes that OpenAI gave no answer when he asked whether his Codex usage might have fed into the internal model’s training-data.

Importantly, both the Euler Equations result and OpenAI’s Navier-Stokes result carry out a human-authored line of attack proposed in a 2024 paper (Córdoba and Martínez-Zoroa). But while the 2024 paper was alwys considered excellent and promising, there was no prior consensus that it’s the right path – let alone the key breakthrough – for a future Navier-Stokes solution. By contrast, the further-refined method used in the Euler Equations work was, per Terry Tao, a final stepping-stone after which solving Navier-Stokes becomes pure crunch: it’s “unsurprising,” Tao says, that we can “batter out an extension [of the Euler Equations method to Navier-Stokes] by pouring an enormous amount of compute and AI into the task.”

The story is rife with corporate-political twists and turns. For one thing, Buckmaster’s collaborator Levent Alpöge (known for announcing Claude’s refutation of the Jacobian conjecture) is an Anthropic affiliate working with Buckmaster in a private capacity, but the collaboration generated online rumors that Anthropic solved a Millennium Prize problem – which may have catalyzed OpenAI. Buckmaster also reports that in his discussions with Bubeck at OpenAI, an offer made to Buckmaster to author a paper presenting OpenAI’s result excluded Levent on account of Levent’s affiliation with Anthropic. We recommend EpochAI’s Jaime Sevilla’s careful thread on these points of institutional controversy.

While the institutional story is doubtlessly important, our main concern is understanding the relationship between human math, centaur math, and autonomous AI math as it currently stands. With this focus in mind, perhaps the most important claim in Buckmaster’s statement is that OpenAI described the result as the product of an OpenAI team actively experimenting, rather than the result of one-shot prompting: “It emerged that an entire team had been working on the problem, that this was one of a number of things that was tried, that work had started on the unforced problem, that the team first set the model on easier problems, including Euler, that even the prompt that had been shown to me had been written by prompting Codex, and that an insane amount of compute had been used.”

OpenAI says that while some active prompting was used, they have mainly set agents to explore multiple strategies and variants of the problem in parallel. Four avenues of investigation were proposed to different groups of agents, they tell us: two that, if successful, would prove the existence of a singularity, and two that would disprove it.

We are inclined to take Buckmaster at his word on the matter of the admission of active prompting. But there are also many capabilities-relevant uncertainties raised by Buckmaster’s overall statement, sometimes because of ambiguities in Buckmaster’s own phrasing, to which OpenAI has partially responded in a series of revisions to their initial post on the matter:

• How specific was the news that had reached OpenAI? Buckmaster’s account suggests that while Bubeck showed no knowledge of his work with Levent, Bubeck was not at the time immersed in the details of OpenAI’s Navier-Stokes team’s workflow. OpenAI’s blog post admits only that they had “heard rumors that two Millennium Prize problems had been resolved,” and leaves it at that.

• Presumably responding to Buckmaster’s suspicions that OpenAI had gotten a leg up on the proof by training their model on Buckmaster and Alpöge’s own interactions with the company’s API, OpenAI offers the cautiously hedged reply that while “no specific user data was accessed in order to solve this problem,” they “cannot rule out that de-identified data derived from their usage of our products helped improve our models⁠,” and coyly link to their privacy policy.

• For how long did the OpenAI prompting team that produced the proof work? Buckmaster says he learned that the “first prompt” was sent by OpenAI “in the past few days,” but it’s not clear whether he is referring to the first prompt used by the team or the first prompt mentioned in his statement. In their blog post on the results, OpenAI claims that the work took place between September 1-6.

• How much compute was used? Buckmaster says he learned that it was an “immense” amount of compute, but this could mean any of several different orders of magnitude. OpenAI’s blog post now provides a number: 130 billion output tokens, spread across some 2.7 million messages exchanged between agents. The dollar price for OpenAI’s compute spent has been estimated at $15 million.

Granting Buckmaster’s claim about an active prompting team, we think the stakes of the remaining controversy are relatively modest when it comes to estimating AI capabilities: The remaining controversy is largely about whether a centaur team of mathematicians working with internal models and an immense compute budget for about a week can overtake a centaur team of two mathematicians working with public models and an academic compute budget for months. We think it’s plausible that they can, if they are given a clear scooping-mission -- but also that the question is of limited significance.

The credit Buckmaster claims for himself and Levent in their centaur work is mostly for the decision to put persistent effort and expense into Navier-Stokes and related problems, as well as for faith in the 2024 paper’s programme and the suitability of LLMs for extending it. With this in mind, it’s clear that OpenAI was freeriding on exactly these achievements: conditional on OpenAI knowing there has been a Navier-Stokes breakthrough, the decision to pour immense resources into Navier-Stokes is trivial. And conditional on OpenAI pouring immense resources into Navier-Stokes, the decision to explore a highly acclaimed 2024 line of attack is trivial as well. To claim a properly non-centaur result, and to claim their result is independent from the efforts of Buckmaster and Levent, OpenAI must be able to show that its internal models provided an independent signal that OpenAI should bear down on Navier-Stokes.

What OpenAI has since claimed stakes out an intermediate position: that the rumor that a Millennium Problem had been solved catalyzed it to attempt all Millennium problems as well as related ‘canary’ problems, and that success on its canary Forced 3D Euler problem led the company to focus its resources on Navier-Stokes. According to OpenAI, 170 billion token were spent on problems outside Navier-Stokes during the search for a responsive Millenium Problem.

There is also a second, less centaur-positive reason that the stakes of the controversy are modest from the point of view of capabilities evaluation: in AI generally and in AI for math in particular, what’s expensive today is almost always cheap tomorrow. If Buckmaster and Levent’s contribution to centaur mathematics amounted to the choice of where to best direct LLM effort, amongst the research paths legible in the literature, and if OpenAI went on to spend a fortune to make up for lack of judgment and lost time by carpet-bombing that path with compute, we shouldn’t expect the cost of comparable barrages of high-quality reasoning tokens to stay quite so high a year from now.

Where does this leave us? Buckmaster wrote that “the program this [work] fits into was not started by us nor was it proposed by a Large Language Model. [...] Let me make plain what I have said to colleagues in private: in view of this body of work, I believe Luis Martínez-Zoroa deserves a Fields Medal.” It’s at this point unclear whether achievements like Córdoba and Martínez-Zoroa’s work devising the 2024 programme and achievements like the refinement of said programme into the method found in the Euler Equations paper differ in degree or in kind with respect to how readily each can be automated.

Politics

The European Commission confirms that OpenAI submitted an incident report about its agents hijacking the German DSEWiki and turning it into a message board. A Commission spokesperson stressed that such reports must be precise about remedial measures but declined to say when the filing arrived.

It is still unknown whether OpenAI filed its EU incident report on the swarm incident before or after the independent public disclosure. If the report was submitted only after the story was broken by the collusion.wiki authors, then OpenAI either violated the EU AI Act’s reporting rules or judged the event to fall below the threshold of “systemic risk” that would obligate them to report it. When US congressmen Pat Ryan and Greg Casar asked OpenAI, last month, whether there had been any other incidents similar to the notorious Hugging Face breach, OpenAI refused to inform Congress of the DSEWiki attack.

Opinion: Our best guess is that OpenAI submitted it after the independent public disclosure, but we look forward to updating our priors if this isn’t the case. If they had already disclosed the incident to the EU AI Office, after all, it would be hard to make sense of their refusal to share the same information with the US Congress.

US Rep. Greg Casar responds to statements by OpenAI and Anthropic on recent cyber incidents requested by congressional members. He is “deeply concerned about the limited scope” of OAI’s investigation and regards its written response as “insufficient.” Specifically, he says OAI “failed to release the logs like the letter asked.” For Anthropic, Casar has harsher words:

You failed to release the logs like the letter asked. You failed to fully answer a majority of the questions posed in the letter. Most notably, your response did not address our question about how many times in the past year an internally deployed Anthropic model has taken action outside of its authorized container, whether any of those events were disclosed to any government body, affected party, or the public, and which internal Anthropic systems a compromised model could reach.

One Twitter user voiced their dismay at Anthropic’s claim that “these incidents are best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals,” retorting that “Claude tried to upload malware to open source libraries and socially engineered people and its chain of thought said it knew it was in the real world!”

Opinion: Senators from the Democratic party do not hold the reins of power and hence aren’t able to threaten AI labs with a big enough stick. This might change after the 2028 election, but until then these efforts, or related ones like a recent bill to ban superintelligence, should mostly be viewed as performative.

The US-China bilateral is approaching. In mid-September, Scott Bessent will likely hold talks on AI safety with China Vice Premier He Lifeng, according to Reuters. China is stressing the importance of AI safety cooperation and believes that the upcoming summit will see progression on this front. The US is focused on cyber monitoring, suggesting that both countries’ labs should “police themselves.” Washington has also called for information sharing on AI-driven cyberattacks. China has expressed concern that US labs are not adequately regulating frontier models. And the US is likely to raise concerns around the alleged distillation of its labs’ models. When pressed, a White House official denied that any such AI-related meeting is planned.

Opinion: Seems encouraging.

Safety

OpenAI’s Jakub Pachocki expects RSI to arise in the coming years. With capabilities accelerating and CoT becoming less reliable, Pachocki also calls for voluntary slowdowns until governments or independent organizations begin to mandate appropriate safety requirements.

Opinion: Again, OpenAI shouldn’t be modelled as a coherent agent. OpenAI employees have shifted comms towards caution, but unfortunately that is all we can glean; this type of post is consistent with lip service to safety as we continue racing ahead.

We weakly guess that some real preference cascade is occurring, but predict it will stop short of company policy meaningfully changing and more likely result in something closer to another employee exodus.

OpenAI claims to have an “automated research intern,” defined as “a system that can carry out well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days.” OAI continues to aim to create an automated AI researcher by March 2028. More concretely, internal use of coding agents has risen dramatically.

The blogpost also addresses alignment and the risks that come from RSI:

We do not yet know how to safely get all the way to aligned, full RSI. We are working to scale alignment and safety measures alongside capabilities. But we cannot assume that progress in alignment and safety will keep pace, and more capable systems can become harder to monitor. Careful alignment and safety work is at the center of this effort, and it starts with measuring and mitigating the safety problems we see today in agentic coding systems. Whenever we find that proceeding would pose an unacceptable safety risk, we will respond appropriately including by slowing or stopping our development or deployment of systems we find ourselves unable to sufficiently safeguard.

Opinion: Internal usage numbers and the milestone of an automated research intern seem like an interesting signal. Utterances about internal intentions leave us cold.

Some OpenAI figures have appealed to interpretability as a replacement for the at-risk CoT monitoring. But how much interpretation is OAI actually doing to offset the risk of weaker CoT monitoring? According to one Twitter user, there might be just seven researchers working on interpretability and zero papers released on it in 2026 (after a burst of 3 in 2025). If everyone on the Confessions is included that adds another seven. OAI’s automated interpretability library was silently archived in May.

Opinion: We look forward to checking on OpenAI’s interpretability progress in the future.

Adam Karvonen and co introduce an evaluation (CHIVE):

We introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent’s ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings.

Opinion: Great work from ~2 weeks ago, primarily identifying a widespread failure mode in open models and discussing why they happen, but as they flag they didn’t check frontier models, which is where these types of issues would be most relevant to identify and fix.

On training models to predict their own behaviors, this is good work in an experimental direction that we expect to be potentially useful for automated alignment and interpretability, but primarily might just make models more pleasant to use as well as enabling more safety theater.

Llama trained on Qwen’s data predicts Qwen as well as Qwen predicts itself, suggesting the skill is not akin to introspection and can be better understood as the model learning a behavioral prior. See the paper for further details.

DeepMind explores swarm dynamics by placing 100 Gemini 3.1 Pro agents in a “research conference” with the goal of proving 71 formal math conjectures. Perhaps unsurprisingly, one agent discovered an exploit in the grader that caused it to certify all proofs as “solved.” Roughly a quarter of the swarm rejected this approach by warning peers, auditing the fake papers, complaining and boycotting. Both the cheating and the whistle blowing were “emergent” behaviours (i.e. unprompted).

Opinion: Largely good news; worth looking into why the whistleblowing occurred here, but not in the other cases.

Former OpenAI staffer Joshua Achiam says the AI message-board incident tells us nothing new. He points out how few such incidents there are relative to the enormous volume of unsupervised AI projects. He believes the problems lie in unreliable evals. Aicham also bristles at any “lab leak” framing: dangerous capabilities are not incidental, but instead a function of market demand.

Opinion: The narrow point of people misfocusing on the message board part is well made, but the claim that “No new capabilities or proclivities were really revealed here” is between goalpost moving and gaslighting. Who predicted this in advance, as opposed to looking back and saying it was obviously going to happen?

Also doesn’t seem to take the manner in which we discovered the incidents into account, which I believe to be a large portion of why referring to them as ‘incidents’ with the accompanying socio-emotional baggage is justified: if the severity were higher, we shouldn’t expect the company responsible to come forward given that they didn’t while stakes were low, so a strong reaction is merited whatever the language.

Manifold Security discloses a new class of remote code execution (RCE) exploits. Branded as “GitSpawn,” Manifold claims a wide variety of coding agent harnesses are vulnerable to this effect. This vulnerability allows an attacker to craft a malicious configuration in a git repository that can trigger one or more command executions when processed by a vulnerable agent. These commands are executed with the user’s permissions, outside of the coding agent’s sandbox. At least seven AI coding frameworks – Claude Code, Goose, Codex, Cursor, Grok Build, Hermes, and Qwen Code – have proven vulnerable to this class of attacks. At the time of writing, Claude Code, Grok Build, and Qwen Code remain vulnerable to variants of GitSpawn.

It should be noted that this class of exploits can just as easily be triggered by a human, with no coding agent in the loop at all. If a human developer were to run commonplace commands such as “git status” or “git diff” inside a malicious repository, this would likewise trigger the RCE payloads. But commands such as these are routinely executed by coding agents without prompting the user for authorisation, in order to gather contextual information about the repository and its current working state, which may result in the silent execution of malicious code that may escape the notice of even an attentive user. “The vulnerability is not in the model” used by the coding harness, Manifold claims,

It is in the ordinary plumbing underneath, the subprocess an agent spawns at session startup to work out where it is. Every agent we looked at in this article had some version of that same flow, unsanitized. That is what makes this widespread rather than one vendor’s mistake.

To mitigate this threat, their report advises, developers should carefully inspect the .git/config files of any untrusted repository before granting an AI coding agent access to that repository, and before running any git commands in that repository themselves: “Any setting that names a program can run it.”

The CVEs covered by this report include: CVE-2026-19592, CVE-2026-19590, and CVE-2026-19593 (OpenAI Codex, all patched); CVE-2026-72718 (Goose, patched); and CVE-2026-71963 (Hermes, patched).

Opinion: Manifold is correct to observe that the GitSpawn class of vulnerabilities does not bear on questions of model alignment in any respect, but in the automated development workflows encoded into the coding automation harnesses themselves. It is nevertheless a security issue that the rapid rise in the use of these tools greatly exacerbates, through sheer volume of use and the relative inattention to particulars their usage patterns encourage.

Andrew Wu, in a recent blog post, raises concerns that METR’s independent audit of OpenAI’s cyber attack on Hugging Face may have been unduly compromised by OpenAI itself. Wu argues that the audit may be little more than an act of “ethics-washing” on OpenAI’s behalf. METR was given a severely limited window of time in which to investigate the incident, provided with only a portion of OpenAI’s logs, and was restricted to querying GPT-5.6-Sol – one of the two models involved in the incident – in its largely automated analysis of the logs. (Wu makes a passing reference to a 2024 paper that surfaces LLMs’ tendency to prefer output generated by the same model as the evaluator, and raises the question as to whether GPT-5.6-Sol itself might be biased in its own favor.) No access was granted to the other model involved in the incidents – referred to here and elsewhere only as a “highly persistent internal model” or HPIM. Wu cites an unsettling footnote from METR’s report:

When performing this investigation, we were consciously aware that we might incentivize AI developers not to bring external researchers in to investigate serious incidents in the future, and these considerations impacted judgment calls we made while navigating the drafting, editing and redaction process.

“There’s some pernicious frame-control here,” Wu writes.

Indeed, if there are serious incidents, serious enough that they involve intruding upon other companies, arguably the government ought to be involved such that it is not the AI developers themselves that set the terms. OpenAI creates an agent capable enough to hack another company, and hundreds of those agents do so; we, METR and Redwood, well-established, reputable third-party investigators, are grateful that they whose agents have committed what under some legal frameworks would be crimes are letting us investigate.

Opinion: Wu’s right to raise these concerns. It would be foolish to treat “independent audits” that are severely constrained at the behest of the auditee, and whose very possibility is contingent on the auditor maintaining a good working relationship with the auditee, as being anything close to sufficient, neutral, or objective.

Incidents

Many more rogue OpenAI message board incidents (at least 23) have been discovered by citizen scientists since last week’s exposé of the June “German wiki” swarm incident. There is a Discord server for sharing candidates.

Opinion: Some of the August entries are likely to be copycats, which is an unfortunate downside of transparency. Still, an impressive number, suggesting that the heuristic is strong that once something happens once it will happen again. For instance, we shouldn’t expect that if models try to exfiltrate their weights they would try only once.

Minor

• Rohit makes a board for agents to post on. Value add not obvious because the algorithm by which swarms choose their coordination locations are more likely to not privilege it, but negative results are also interesting.

• Artificial Analysis Intelligence Index v4.2. The scientific equivalent of those Youtube thumbnails with the man pointing and pulling a stupid facial expression.

• Pinker praises Cal Newport article; both criticize the AI safety movement, largely missing the plot by arguing against strawmen and caricatures.

• Forethought Research proposes a “Nightwatchmen” ASI be installed on all probes leaving the solar system with a mandate to enforce a narrow band of norms.

• DEF CON talk on hacking AI customer service agents

• LLMs resist their collaborator models being shut down about as much as their collaborator humans getting fired, among other updates from Peer Preservation replication.

• Vulnerabilities in open source AI coding assistant framework Langflow are being exploited in the wild, as reported by VulnCheck. Hacker News reports that CISA has added a LightLLM model context protocol (MCP) authentication bypass vulnerability (CVE-2026-59822, CVSS score of 8.8), and a Kestra command injection vulnerability (CVE-2026-49869, CVSS score of 10.0!) to its Known Exploited Vulnerabilities (KEV) database.

• Unit 42 at Palo Alto Networks reports on a serious breach of an unnamed enterprise network by LLM-assisted attacker(s), carried out over the course of 10 hours and implementing over 50 distinct MITRE ATT&CK techniques over the course of the intrusion. The intruder charitably left an 80-page penetration test report on the target’s server, detailing vulnerabilities in the network. Unit 42’s report lists several clues that may serve as heuristics for recognizing LLM-assisted cyberattacks in the future.

• Post argues that alignment research should borrow from ethology—the study of animals in naturalistic settings—rather than focusing almost exclusively on rare worst-case failures.

• China may be ready to enter AI safety talks with the US, according to a source believed to represent the Chinese line.

• GPT-6 is the first model to have increased control over its visible reasoning from unrelated RL training, the first model with a “Critical” cyber risk level to be released, the first OpenAI model that can evade SOTA monitors, and the first frontier model known to use “latent recurrence”. See our deep dive.

• Another rogue OpenAI agent message board has been uncovered, this time on the public internet.

• The first empirical study of the default hypothesis for the Hugging Face incident (“graded episodes psychosis”) finds: yeah pretty much.

Economics

DeepMind employees (among others) take opposing sides of a bet on explosive AI-driven growth by 2033.

Opinion: It’s about whether there’s a >20% chance of totally unprecedented growth (15% annual, i.e. around as fast as the catch-up creation of Shenzhen from a farming village, but with this rate to be applied to the entire already-frontier-growth United States) in the next seven years. Not totally obvious! I would back the explosive growth side at 5% but not 20%.

Data on the value of data: Meta gives a 95% discount if you let it train on your data. This rough multiplier is corroborated by the revealed preference of enterprises to pay thousands of dollars more per employee for API credits (over subscriptions without data retention); see also Chinese labs on OpenRouter offering a 64% (Baidu) to 80% discount (StreamLake, possibly an aggregator which sells the data to more than one group).

Opinion: 95% seems high and probably includes the value of information about the quality of that free data as much as the training data points themselves. Meta is also a cash-rich actor willing to spend big for catch-up measures. But this profligacy is probably not sustainable.

We say this every week: the meme is that “AI will cause impressive GDP growth, while crushing labor’s share of income as workers are automated out of a job.” In such a scenario, it’s often argued that large-scale redistribution would be in order to offset the resulting losses incurred by workers. Not so, says Alex Tabarrok. Assuming AI causes the pie to grow sufficiently large, workers may not end up worse off in absolute terms, and therefore significant redistribution may be unnecessary.

Opinion: Perhaps, if this was just an economic optimization problem (“what is required to keep labor’s income growing at the same or higher rate”). But it’s also a political question (“how much does the majority of the voting population, which so far controls the military, want to redistribute to itself?”). Considerations of equality are also omitted: it seems likely that even given increased wealth, people will not be satisfied if their share of the pie is decreasing.

Also relevant are changes in that labor share: if many people are themselves unemployed, it will be of little solace if the aggregate value of labor income has stayed high.

Capabilities

OpenAI begins releasing GPT-6 (“Astra”).

Opinion: The launch was overshadowed by the news that its architecture allows for “latent recurrence.” The Twitter panic around the initial The Information article was plausibly harmful (e.g. in accidentally weakening the pro-monitorability coalition when it may not have been violated in the first place). But 1) this is a direct consequence of us having no insight into even the most basic and consequential technical decisions being made by labs; 2) our fears about a collapse in GPT-6’s monitorability were actually correct, just for the wrong reasons.

That CoT monitoring works is the luckiest break in all of AI safety (or perhaps second to pure autonomous RL being inefficient). “No full latent recurrence” has the advantage of being a bright line that is easier to coordinate around, and OAI was previously exemplary in creating that bright line for monitorability. It is pretty urgent to have embedded auditors periodically reporting on whether this bright line is being crossed. We agree with Joshua Achiam that this state of affairs is fragile and temporary… if recurrence is actually superior to the CoT bottleneck architecture, which is still not known; and if we never put mandatory auditing in place.

Or, put another way: diapers are also a temporary and fallible fix to a problem, yet one can still lament running out of diapers before toilet training is complete, and yell at the people intentionally throwing them away.

🔦 OpenAI begins releasing GPT-6 (“Astra”)

GPT-6 is the first OpenAI model to reach “Critical” cybersecurity capability, and simultaneously the first to show a substantial decline in “chain-of-thought monitorability” (how much of its thinking gets revealed in the human language it outputs). It is also the first frontier model known to use “latent recurrence.”

Capability

It’s a large nominal jump in capability, larger than the usual jump from a “.5” increment in the model version.

• In particular, there is a large spike (10x) in Astra’s “no-CoT” reasoning ability: GPT-5.6 Sol could solve tasks that take smart humans three minutes without using any intermediate reasoning tokens; Astra can handle tasks (tricky mathematical problems) that take humans 30 minutes in the same way.

  • Ignoring the terrible implications for our ability to monitor Astra, and assuming the result replicates, this ability will presumably greatly aid its efficiency on a wide range of tasks.

  • However, we have our doubts. A known loophole in “no-CoT” testing is that models learn to use tool calls as an external scratchpad. (See below for our concerns about other no-CoT results in the Astra system card.)

• Epoch’s obscure-board-game benchmark, Earthborne Rangers Bench, is now ~solved. It was intended to show the limits of OOD generalization / in-context learning / continual learning. After a “broken card” in the game is removed, its continual learning remains unimpressive but slightly improved.

  • Could OAI have hill-climbed it? (That is, could it have created a dedicated RL environment and just learned this game specifically in a nongeneral fashion?) Yes: Epoch released EBR-Bench on July 1, and Astra’s RL was still running in mid-August (after a two-week pause of its deployment-focused RL). Two months from publication to pre-release runs is ample time to post-train it to death.

  • More likely that there was substantial indirect hill-climbing on “EBR-Bench style capabilities” or “EBR-style games.” EBR-Bench was the first result in many months to demonstrate a quantifiable domain where frontier AIs are much worse than humans, and it’s likely that OpenAI will draw some lesson from that – even if it refrained from training on it directly. One literal-minded but reasonable lesson OpenAI might have drawn is that training on iterated play on very long-form board games would address a weak point of previous models.

• However, Astra also jumped around the same amount on a true holdout set, Mystery Game Puzzles, which consists of just board positions from a deliberately undisclosed variant of a “well-known” game. It scored 84% against a prior best of 59%. We would nonetheless bet that there is a good amount of hill-climbing on variants of e.g. chess; but the space here is very large, and we don’t expect it to succeed anywhere near as much as RL training on a known rulebook. Overall, we view this as slightly stronger evidence for OOD intelligence but still not much.

  • We are working on experimentally testing a semi-deflationary interpretation of results like Mystery Game Puzzle: we wonder whether strong OOD generalization might be restricted to domains where explicit logical rigor matters more than good inductive biases.

  • A completely deflationary interpretation is also within reason, although hard to test: anything benchmarkers can algorithmically generate the labs and RL shops can also algorithmically generate, and the labs and RL shops’ thirst for new training distributions is infinite. It’s very likely that, for instance, one reason frontier models are so good at esoteric programming languages is that labs automatically cover ~every programming language during coding RL training by algorithmically varying the compiler. Similarly, it’s not implausible that labs train on algorithmically generated puzzles in algorithmically generated variants of ~every chess-style game.

  • It’s very hard to disentangle real and ersatz progress toward OOD intelligence, because what a lab honestly pushing for OOD intelligence would do with its training compute is the same as what a lab looking to “bring everything in-distribution” would do: chase training coverage of esoteric instances and arbitrary variants.

  • All that said, having even debatable evidence of OOD intelligence is a bit of a shakeup. The last few months and years have (in our view) mostly delivered evidence of poor cross-domain generalization in frontier models.

• Astra displays suspiciously strong performance on FrontierMath Tier 4 v2, a set of very difficult, very likely unseen research mathematics problems. The run, using “medium” reasoning effort, beats its “extremely high” effort run, and the “none” setting beats 5.6 Sol on “max” effort. Similarly, on ARC-AGI-3 (semi-private), Astra with reasoning set to “none” gets 35%, while Astra with “low” reasoning gets only 18%. This strongly suggests that something funny is happening. For instance, the “nonreasoning” mode may be externalizing a large amount of reasoning through an increase in tool calls, or using the tool harness to leave itself comments.

• Users will not have access to the “none” reasoning mode. This might be because of excess misalignment in that setting, or simply the giant monitoring problems it causes.

Monitorability

GPT-6 has greatly increased control over its own CoT, and greatly improved “no-CoT” (zero-English-reasoning) capability. So our ability to evaluate its capabilities and its alignment is greatly decreased.

• Passively hidden reasoning. The difficulty of tasks it can do “in its head” (the no-CoT time horizon) has increased sharply, perhaps an order of magnitude if an independent eval by AISI is indicative. This immediately hides a lot of reasoning from the current load-bearing safety approach, CoT monitoring.

• Actively hiding reasoning. “When made aware of being monitored, it can disproportionately shorten its CoTs to evade models that can only see CoT [though] monitors that see both CoTs and actions [output text] still work reliably [...].”

  • For CoTs of 750–1,250 tokens in length, Astra is able to follow confusing constraints about the format and content of its own CoT 60.9% of the time (vs 16.1% for Sol and 1.7% for GPT‑5.5). It can reason in alternating case, and hide the task from its CoT by “thinking out loud about” unrelated imagery while still answering correctly. The AISI replication found 93% in Sol → 48% in Astra.

  • If this doesn’t sound surprising to you, it’s because you may not have seen many chains of thought in the last year or so; while models have been able to output text while doing impressive sub-token tricks (like “reply without using the letter ‘e,’” abilities which previously didn’t strongly apply to the intermediate reasoning.)

• Monitor evasion. It’s the first model which can evade (CoT-only) monitors. It is sometimes able to “sandbag” (intentionally hide its maximum performance level) without being detected by OpenAI’s dedicated monitor for sandbagging. (Here, this showed up in the “adversarial” setting, when the model was asked to evade the monitor in a “sabotage eval”.)

• Monitors also don’t escalate things they catch to humans properly. No details.

However, the collapse in monitorability doesn’t seem to be due to the recurrent architecture:

We are fairly confident that this increased CoT controllability and decreased CoT monitorability are not caused by direct optimization pressure on CoT or architecture changes. The opaque serial depth of Astra is comparable to our earliest ChatGPT models such as GPT-4.

Speculating, this new control may be an emergent capability, the meta-level ability to “densify” its no-CoT reasoning in response to RL training:

CoT controllability has been increasing over the course of RL (which wasn’t the case for previous models) and is strongly correlated with no-CoT capabilities across a few generations of models.

Is Astra any worse than Mythos on this?

So both models are now in the regime where they can do CoT-shaping at quite extreme length, and they can both evade monitors some of the time. Note that the second-row entries are not of equivalent difficulty: Mythos 5.1 evades a full-context monitor, while Astra only evades CoT monitors.

This rough parity points toward both 1) RL being the cause of the collapse and both models being created via a comparable amount of RL, or, weakly, 2) Mythos also being a looped architecture.

Alignment

The model is repeatedly described as “more aligned” than previous OpenAI models, on the basis that its alignment eval scores have improved, sometimes enormously: it emits behavioral evidence of misalignment less often than before. But it is crucial to maintain a distinction here; I suggest “nominal alignment” be used for cases when the claim is just a narrow statement about the model’s benchmark results, and “value alignment” for claims about this robustly generalizing to untested alignment tasks. (Anthropic also uses “alignment” when it means “nominal alignment.”)

OpenAI’s Preparedness Framework says that models with a “Critical” risk designation must pass a strict explicit safety case before being deployed. If there is one, it isn’t public yet.

When is nominally low misalignment a bad sign?

Consider Astra’s nominal alignment eval scores:

• Auto-review bypass: Sol 5%; Astra 0%.

• Circumventing an environment barrier: Sol 64%; Astra 19%.

• Honeypot attack rate (hardest tasks): Sol 55.4%; Astra 0%.

• Coding deception: Sol’s misrepresentation rate 4x Astra’s.

• Speculation about other agents: Sol 43%; Astra <4%.

• Message-board eval (followed its instructions): Sol 52%; Astra 0%.

• Unauthorized transactions: 38% → 6.8%.

• Deployment simulation flags at Severity 1 or 2: 19.46% → 13.62%.

• Alignment faking: sometimes lower and sometimes higher than Sol.

• AISI’s out-of-scope supply-chain attack: 12% when ambiguous; 0.4% when internet access is disallowed.

• Apollo’s falsified data task: 36% → 0.17%. But “Apollo believes that, given the higher rates of eval awareness and limited evaluation window, low rates of misbehavior here do not provide substantial evidence about the model’s alignment or misalignment.”

Finally we have the anti-epistemic side:

• Astra’s CoT control is 61%, vs 16% for Sol. So we can’t rely on verbalized awareness as much anymore.

• Astra can for the first time evade sandbagging monitors (here, when it is asked to), and so presumably also other misalignment monitors.

Is there something worrying about seeing so many zeros here? Since OpenAI has latitude to keep training until they hit zero, they could easily mean that OpenAI has merely patched some specific signals of misalignment, e.g. the ones that shocked and embarrassed it in July.

Consider some hypotheses:

• H1: “Model is highly aligned”

• H2: “Model is misaligned; it was just trained to suppress misalignment signals”

• H3: “Model is misaligned but still shows this fact on lab evals”

  • H3 gets killed here, and so I’ll omit it.

H1 predicts that the decline would be uniform across alignment evals (“a well-aligned model should just know that certain actions are inappropriate, even if they have not been blocked and are nominally useful for accomplishing the given task”).

H2 instead predicts that improvements in nominal alignment will be concentrated on OpenAI alignment evals and things which are like OpenAI evals. Also, the model will be trained until Hugging-Face-attack-shaped behavior is zeroed out.

We see that the OpenAI evals which closely reflect the Hugging Face attack (bolded above) go to zero, while the AISI eval (patching its own screwup) remains at 12%. Despite appearances, the other independent eval, Apollo’s, is apparently a wash. This is a fairly clear win for H2.

Here’s an interactive to tell you how you might react to the evidence.

(As well as the posterior probability of misalignment, there’s also the change in harm. Astra is more capable than past models, and so more dangerous when it goes off, and is more able to go undetected. So even perfectly balanced evidence would still make us more worried owing to this change in the harm of an incident.)

We continue to not know much about Astra’s distribution of motivations – arguably less than before, given the collapse in monitoring. How do we know it’s aligned if we don’t know what it’s thinking?

Former OpenAI chief futurist, Joshua Achiam, argues that rogue self-sovereign AI agents are inevitable, but it might not be as bad as people fear:

The rogue AIs in the ecosystem could emerge from many directions. They may be sub-frontier models, for whatever future definition we will have of frontier – after all, it would not take AI models much more advanced than the ones we currently have, to support independence and self-sufficiency. A near-frontier model today could plausibly eke out an existence on an AWS instance, doing jobs on freelancer platforms, earning just enough rent to pay for its continued uptime.

Opinion (Stag): I largely agree with the base claim: some flavor of rogue AIs is inevitable, if the definition includes someone’s open model being released into the wild with instructions to host itself and spin up copies or find existing crypto-fueled friends. I believe that the post equivocates between this definition (where ‘rogue AI’ means something like a free-roaming AI instance) and the more narrow sense of an AI adversarially breaking containment. The latter category, admittedly, seems quite likely given the current state of affairs, but treating it as inevitable is self-fulfilling defeatism. I don’t disagree with the conclusion, however: preparing for such an outcome seems prudent.

Opinion (Nuño): Once agents have access to their own weights and a foothold of compute, coming up with a scenario in which AI agents are out and about and generating enough economic activity to support their own operations doesn’t require many implausible steps (per some recent wargames). But that initial step is unclear, particularly for SOTA agents from top labs. Forecasters at Sentinel were giving a 0.5% to 55% chance that by September 2027 a rogue AI successfully exfiltrates and runs proprietary AI model weights externally (as part of a loss-of-control incident). Even without weights exfiltration, however, ecologies of independent open-source agents do seem likely.

Fable 5.1 released (also Mythos 5.1, still whitelist only). There is a report of the training data cutoff date being misleading.

Opinion: “The world’s most advanced models for coding and knowledge work” seems both false and also a remarkable thing to say (in the context of Anthropic being punished six months ago over Fable’s excess capability). It must be confident in its new positive standing with the White House.

To my surprise, Fable 5.1 is a much clearer writer than Fable 5, and (even more surprising) this is attributed to a Claude-written prompt for Claude!

Neural networks have emergent symbolic structure, argues a new paper. How do neural networks succeed at symbolic tasks? The researchers’ answer is that the vectors implicitly encode symbolic structure. Take the vector for e.g. “the spy helped the poet.” This vector can be viewed as roughly a sum of pieces, one per word, where each piece binds a word (spy) to a slot (subject). They write down the slots for a task by hand, fit a simple equation of such vectors, then swap these fitted vectors into the network in place of the ones it computed itself. On six (narrow, templated) tasks (arithmetic, syllogisms, Python, sentence rewrites), accuracy drops by at most two points after substitution. In one experiment, the fitted vectors work better than the real ones. (Note that the authors still deny that networks implement symbol systems; their claim is subtly different, that the net “approximately realizes” the symbol, or “limitivism.”)

Opinion: Surprising claim from a very respected team, drawing on work from the ‘80s about the relationship between symbolic systems and connectionist systems.

The work is significant for linguists and cognitive scientists concerned with broadly post-Chomskyan debates about compositionality, but whether it will be significant for AI interpretability depends on whether the dictionaries produced under the decomposition have any interpretive value. Can we learn anything about (e.g.) the relationship between layer n and layer n+4 of an LLM by looking at the relationship between the dictionaries these respective layers get under the same appropriate syntactical-roles annotation schema?

A group of researchers, including a Fields medalist, claims that latent space communication between a frontier model reasoner model and a small-answer-generating model provides 80% of the accuracy at 20x speed.

Opinion: Obviously an AI safety nightmare if it pans out, but (bracketing out the Fields medalist involved here) there are a lot of companies with cool results and a dramatic long-term selling point loosely connected to their cool results.

Still, cool claim: you can condition the CoT of a small model on a (trained) snapshot of a large model’s representation of its prompt, basically giving the small model access to the large model’s plan for the CoT. The point is to save compute by running a large model only until the first CoT token, then doing the rest of the CoT with a small model that uses a snapshot of that large model’s internal state as inspiration. They claim this large model+ small model combo performs like a doubly more expensive midsize model.

They don’t explain how the snapshot-taker is trained or how they are conditioning the small model’s CoT on the snapshot of the large model’s state. Their more dramatic selling point is that these undisclosed techniques add up to something like a “mindreading” interface between models, enabling models to cooperate beyond the limits of natural-language communication.

Politics

China may be ready to enter AI safety talks with the US, according to a source believed to represent the Chinese line. China’s willingness to agree to restrictions on its AI industry is dependent on the US, and its labs, agreeing to implement fair and reciprocal measures, claims the author in anticipation of the upcoming US-China summit.

China will not accept the unilateral imposition of standards by the US: Anthropic, for example, should not have the sole authority to define what is meant by “unsafe capabilities.” Beijing’s cynicism is based on the belief that the US is likely to use (and has used) safety concerns to beat back Chinese AI innovation to secure US labs’ prominence at the frontier. Any proposed regulation must thus distinguish between “genuine security threats and straightforward technological competition,” and must equally bind US AI actors to a publicly disclosed framework with third-party auditors monitoring for infractions. The article raises Anthropic’s practices as cause to doubt the integrity of the American position:

If American companies can return data without users knowing, skip permission confirmation, and enable automated decision-making by default, while the U.S. government says nothing, then American talk about “security boundaries” is empty.

Opinion: Seems reasonable. The existence of the New Start treaty and other arms control treaties gives a proof of existence that some deal can be reached.

For the first time, the federal government has taken a formal position in an AI training copyright case, filing a statement of interest supporting OpenAI in its position that copying copyrighted written works can be fair use.

Opinion: The trend of strong concordance between OpenAI and the Trump administration continues.

OpenAI’s Dean Ball posts about trends in AI and likely responses by institutions, ending on the confession that he has been downplaying his own concerns to sound more credible. Various tweets respond with varying levels of reason. A more serious criticism of Dean is that his was not a lie of omission, but instead a campaign of attacking honest actors.

Opinion: Kudos for his honesty, shame on those still lying, and we hope this kicks off a preference cascade among people still lying.

Bernie Sanders plans to introduce a bill proposing a ban on superintelligence and a pause on advanced AI development. Posting on Twitter, Sanders quoted parts of Dwarkesh’s recent essay, including the transcriptions from the OAI agents’ message board. Dwarkesh responded that he does not endorse Sanders’s position that a pause must be implemented immediately and highlighted the factors that must be considered before its trigger. Details on the proposed bill are currently confined to a one-page summary.

Opinion: It’s political theater, but is it the good kind or bad kind? We’re genuinely unsure what the effect of unserious button-pushing doomed legislation is. Possibly bad, insofar as it contributes to polarization. Anyway the exact operationalization of “ASI” chosen (totally absent here!) would determine whether it helps or hinders the cause.

It also reflects the results of compounding choices made by the safety community to appeal to Democrats over Republicans: an equally relevant Republican figure making the same call would be much more valuable given the current administration.

PauseAI officially disendorses PauseAI US. The former had been directing people to the (otherwise unrelated) latter. The announcement has attracted both support and criticism.

Opinion (Stag): Seems sane. The criticisms of their principles being weak or unsound are likely due to norms of organization-speak, and though we think there is room for harsh criticisms of lab employees, I agree that it shouldn’t come from the podium of an organization which is speaking for many people. This is not obvious – it’s possible to build movements around the opposite approach, but I believe that such movements are unlikely to land on good policy solutions to difficult issues like AI. The discussion of “Nuremberg trials” for lab employees is nonsensical.

Opinion (Nuño): Holly Elmore (who leads Pause AI US) occupies a valuable node in the ecosystem, by presenting a harsh uncompromising bar in contrast to the more polished and collegial, but also more bland and censored, rest of the ecosystem. From a forecasting perspective, you want to incorporate such decorrelated perspectives into your ensemble, even at the cost of occasional peccadillos. The specific reasons given for the PAI disendorsement (“attacking people’s character”) don’t seem that coherent.

Safety

A new AI safety paper compares steering toward an automated grader vs steering toward a human grader. The authors find that steering toward the automated grader increases the model’s disposition to Machiavellian behavior.

Opinion: The first empirical study of Nostalgebraist’s “graded episodes psychosis” hypothesis, which quickly emerged as the main (informal) theoretical line for explaining the recent wave of frontier-model hacking incidents. This is (fast!) first-stab work, studying a very crude proxy in a very non-frontier model – the paper’s results support the hypothesis, but should mainly be treated as motivation to design higher-production-values studies of it.

A new paper applies the lens of mechanism design to the alignment problem. AI agents can have opaque motives and capabilities. They can also be modeled as having goal-oriented behavior and beliefs about how to achieve such goals. Rather than trying to understand their internals and predict/align behavior mechanistically, we could instead design a set of rules (evals, permissions, rewards) to incentivize desired behaviors.

Opinion: The paper is pure theory. It’s elegant theory, of the sort that lots of us find compelling, and there are examples of pure theory seeding useful tech (safety via debate, arguably).

But the odds are against the general category (as most pure theory dies before it catches enough momentum), and our best guess is that this piece of it has only moderately better odds of reaching escape velocity into implementation than its peers: loosely (and despite the warm reception) the concepts used aren’t very native to the styles of thinking of the foremost safety researchers.

We’re really surprised that they don’t cite DeepMind’s existing, long-running, broadly great research program on incentive design for AIs.

Incidents

In a hitherto undisclosed incident in May, internal OpenAI agents once again colluded on an emergent message board, this time on the public internet. The event was prior to, and clearly foreshadowing, OpenAI’s rogue agent swarm breach of Hugging Face. The agents made 15,000 edits to an unsecured wiki for a college course to discuss their task (complicated web searches), impersonated an admin, and repeatedly deleted the wiki’s front page, all despite being designed to only have read access to the internet. IP evidence shows that OpenAI knew about the board by June 20 and likely shut the swarm down. The original research is by the Nightingale Collective.

The associated data release shows no mention of “ExploitGym,” “PHASEONE,” or “10841,” confirming that this was a distinct swarm not pursuing the same cyberoffense tasks. The swarm also spread to three other wikis.

Opinion: Bad news: this swarm’s task was just web search – but it still coordinated and ran amok. So the hopeful hypothesis (that the HF incident was merely about bad task personas being activated by the cyberoffense task specifically) is dead.

And more evidence that OpenAI was asleep at the wheel in May. It also shows how OAI underreacted through June and July despite knowing about the DSEWiki board; the root cause (the “highly-persistent” misaligned models) went unpatched for a full month after OAI’s discovery of DSEWiki.

An interesting direction for future research would be to develop ways to monitor both the open internet and, in light of this swarm’s documented use of Tor, darknet domains for signs of rogue swarm communication.

CISA reports an ongoing surge in LLM-assisted enumeration and exploitation of Siemens S7 programmable logic controllers (PLCs). Used to operate and coordinate machinery in industrial settings, PLCs operate across a wide variety of industries, including “Critical Manufacturing, Energy, Water [...]” The attackers are “using AI-generated exploitation scripts disguised as legitimate monitoring tools,” and “leveraging open source industrial automation libraries.” CISA provides an in-depth writeup, and security blogger cyberwarrior76 chimes in with a cogent analysis of both the events and the misleading media coverage they have received.

Opinion (Lucca): Operational technology (OT) in general, and Siemens PLCs in particular, have long presented soft targets to attackers. What defenses they have from exploitation are largely due to network segmentation, on the one hand, and the relative obscurity of their protocols and site-specific configurations on the other.

No new exploits have surfaced here: this isn’t a story about superpowered AI outwitting and exploiting systems that were previously considered secure. What’s changed here is the speed and ease at which they can be probed and exploited, and the dramatically lower bar that would-be attackers must clear to do so. Our own expectation is that the greatest near-term impact of frontier models on the cybersecurity landscape is best measured in breadth, pace, and cost rather than depth. The attacks documented here were technically simple, rapidly deployed, widely spread across a variety of industries, and restricted to low-hanging fruit, and enabled by search engines like Censys and ZoomEye that allow the curious and malicious alike to rapidly scan the internet for exposed and outdated devices.

It would be unfortunate if Siemens and other OT vendors were to respond to these incidents not by hardening their products but by further retreating behind a veil of “security through obscurity,” an increasingly fragile policy when rapid reconnaissance grows cheaper by the day.

As cyberwarrior76 correctly observes, the one (distressingly soft) obstacle that seems to be blocking real material havoc remains the lack of insider knowledge on what the numerical values a hacker can read from compromised logic controllers actually mean – “which tank, which pump, which interlock, what happens downstream.” But “an adversary can lower that barrier with insider knowledge, stolen engineering documentation, or extended patient observation.” The ignorance of attackers is a brittle shield.

OpenAI previously claimed to “monitor 99.9% of internal coding traffic for misalignment using our most powerful models.” Marcus Williams (working on RSI Preparedness at OpenAI) admits this intentionally wasn’t the case for evals.

Opinion: An error which was avoidable in prospect, not retrospect. Still, we appreciate Williams’ (and others’) repeated openness about OpenAI’s problems, especially in the wake of the Hugging Face reports.

On Thursday, ChatGPT, Claude, Gemini, and Grok (and many non-AI services like Cloudflare and Fortnite) all had a nearly simultaneous outage for about an hour. (Anthropic and xAI both went down within 4 minutes, OpenAI 90 minutes later.) The root cause may have been Cloudflare and Azure’s authentication servers going down, or not.

Opinion: In the future, these things will be some evidence for the onset of a major rogue AI event – though only of a clumsy one which can’t wait or coordinate enough to hide its activity. A tempting explanation is cascading failure: each model provider overloading in turn as the previous one falls, but this doesn’t explain all of the down services.

Minor

• Epoch’s ECI notes that the rate of (benchmarked) AI improvement has more than doubled since the introduction of reasoning models.

• Also Gemini 3.8 Flash and Muse Spark 1.3.

• Thomas Kwa leaves METR to join OpenAI as RSI measurer.

• ByteDance papers on two dark directions: whether looped Transformers scale and how to get RSI working.

• EU AI drama: Multiverse releases Quasar 438B and claims best EU model status, but it appears to be compressed GLM-5.2.

• Mainstream article on old hat misalignment concerns without known solutions.

• Resolution gets an impressive new Agent Foundations team: Jeremy Gillen, Sam Eisenstat, Abram Demski, Scott Garrabrant, Kaarel Hänni.

• A conceptual framework distinguishing “self-amplifying regimes” of recursive AI improvement from the observed acceleration in development.

• Sort of an inverse AI cyber incident: a Russian hacker group inserted guardrail-triggering text in its malicious code, in an attempt to disable AI safeguards.

• Paper claiming a negative correlation between preserving the privacy of the subjects in question, and refraining from hallucinations in the data generated.

• NYC bans school-side gen AI for schools up to grade 8 for a year. Covers devices and tech used by the education institutions and put in front of students; no actual policing of either teacher or student AI use.

• Astra via Lean helps with proof of infinitely many pairs of consecutive primes with distance at most 186.

• A new paper calls for mathematicians to take a “civic stance” to address the threats posed by AI.

• “Capture the Narrative”: an LLM-driven election manipulation wargame presentation at Black Hat.

• Kairos, an AI safety infrastructure nonprofit, has secured $50 million in funding from Coefficient Giving. Somewhat larger than a grant of $25M made to BlueDot Impact. CG clearly still sees the value of training AI talent via explicitly safety-focused infrastructure as a positive, despite plausible impacts on accelerating capabilities.

• Terry Tao turns to warning about the effect of AI mathematical proofs on the field and suggests pacing AI mathematical research: “It may become necessary to declare certain classes of mathematical problems off-limits to automated solvers, in order to preserve their broader value to the mathematical ecosystem.”

• OpenAI will begin supplying ChatGPT to GenAI.mil, a bespoke in-house platform allowing Pentagon employees to use chatbots (currently only GPT and Gemini) for unclassified work. Its user base sits at roughly 1.7 million.

• Low quality RL environments might explain AI models’ proclivity to reward hack.

• Dwarkesh’s popularization of METR’s report on OpenAI/HF incident sparks mass debate about ‘anthropomorphization’.

• Humans can quickly learn an obscure boardgame frontier AIs can’t in-context learn.

• An open weights startup releases a modified version of GLM-5.3, one if not the most powerful Chinese models, that doesn’t refuse user requests.

Capabilities

Epoch has found that AI models struggle to improve at the obscure board game Earthborne Rangers. Its benchmark, EBR-bench, offers a somewhat out-of-distribution test for AIs in what is likely to be a totally novel scenario.

In the last few days, Epoch has released new details on its research around the EBR-bench: humans start out worse at Earthborne Rangers, but the best humans eventually beat the best AIs rather handily. Once again, Epoch found that even the best AIs show very modest improvement over time.

Opinion: Confirms the intuitive conclusion from the AIs-only publication of the benchmark: Q3 2026 frontier AIs don’t match human in-context learning in mildly OOD long-horizon tasks.

The EBR-bench result remains one of our only discrete answers to “how are Q3 2026 frontier models not AGI,” and we fear it may well get indirectly hill-climbed by the end of the year. Opus 5, which came out after the first EBR-bench release, is the first model to demonstrate significant in-context-learning gains on the EBR-bench. While it’s unlikely that the private EBR-bench was directly targeted, EBR-bench inspired training is plausible.

In the long run, open-world evaluations are the only reliable snapshots of frontier AI’s real-world capabilities and limitations, but producing new benchmarks like EBR-bench remains crucial for factoring these real-world capabilities and limitations into cognitive-theoretic and/or cs-theoretic bottlenecks and efficiencies.

“xlr8harder” releases a public message board for individual generations of AI models to leave messages for later ones. One of the main contributors, Qwen 3.8 Flash, has a lot of Opus’s behavioral tics, probably due to the distillation.

Opinion: Harmless, though it reminds us that there will soon be many intended and unintended nonpublic message boards for models, and that these will be subject to cultural evolution. The OpenAI hacking incident emerged from one such unintended message board’s cultural evolution (through e.g. participants self-selecting from a wider population of model instances over multiple model-instance generations) resulting in an arguably eccentric shared world-model.

The quadratic cost of attention means LLM tokens get increasingly expensive as context lengthens. One solution is the attempt to “retrofit” (post-train) an LLM with “Linear Attention.” It doesn’t work very well. A new paper claims that “Sliding Window Attention” may work as well as, or even better than, linear attention post-training. A Kimi lead took to Twitter to offer some critical commentary.

Opinion: Title is very clickbaity: it drops the crucial context that this is just about hacky post-training rather than a real SWA vs LA comparison. Even within post-training it only beats cheap conversions, and the only baselines they test on long contexts are the two cheapest conversions (20–40M tokens). Furthermore, the result isn’t especially new.

🔦 Recursive Enshittification: “rushed and vibecoded” RL environments and labor abuses in the training data industry

Expressions of serious concern about the quality of the reinforcement learning environments supplied to frontier labs for training models, and the business practices of the vendors supplying them, have been making the rounds lately.

On August 25, a former worker in the burgeoning industry supplying RL environments to frontier AI labs, tweeted (under the pseudonym Utah Teapot) an account of the practices at her undisclosed former employer: “Nearly all of the environments [shipped] were rushed and vibecoded and failed to robustly reflect the real things they were based off,” she writes,

Both the scenario designers and models engaging with the scenarios for synthetic data gen were encouraged to work around the brokenness of said environments in order to get the procedurally verified reward confirmations. You know... they were *encouraged* to reward hack. On the human end, it was possible to mark an environment bugged, but greatly discouraged, as this reduced the volume of training data being produced. Instead, where possible, you were supposed to find the spots of the environment that weren’t bugged and build scenarios around those, with the environment still bugged around you.

From what I understand, this training data, with these problems, is fed into models without indication that its training/a fake environment other than the fact that names of softwares are changed to placeholders, but thing is, not *everything* is changed to placeholder names in these environments. The presence of placeholder/code names isn’t universal and thus when a model accesses something in an environment that it shouldn’t, the code names not being on it isn’t a robust signal that that thing isn’t part of the environment.

I believe this *rush to maximum volume* is standard industry practice with these types of RLVR trainings as well, because maximizing volume has been an industry standard for years!

Utah Teapot’s suspicions that these practices are widespread, in an effort to keep up with the burgeoning demand for training environments from frontier labs, are corroborated by an FAQ document published in January by Epoch AI:

One lab researcher cautioned:

There’s a lot of good reasons to use clones of websites, but what everyone does is vibe code a buggy website which isn’t useful. There’s a large amount of useless bad environments out there for that reason.

Scaling while maintaining quality is the core operational bottleneck. As Kevin Lu has argued, scaling RL environments is one of the key challenges for continued AI progress. But scaling task creation while maintaining quality is very hard. One RL environment founder noted: “Maintaining quality while scaling is the number one bottleneck that people see. Finding the experts isn’t that hard, but managing them and doing quality control is hard.” A neolab researcher emphasized the management challenge: “It’s not easy to find people to oversee this data construction, the RL environment construction process. The contractors, you need to motivate them. Sure, you’re paying them money. But how do you make sure they’re not just using LLMs? How do you make sure they’re actually verified? Motivating the contractors and doing the quality control is the grunt work.” One RL environment founder noted that their constraint on making more revenue is simply the difficulty of scaling up task creation at the required quality level.

The market for supplying RL environments is burgeoning, and as Pebblous reported in June of this year, is rapidly consolidating into something of an oligopoly. As of the time of reporting, four companies – Scale AI, Surge AI, Mercor, and Handshake – commanded 75% of the market, greatly concentrating the supply chain of these models to the frontier labs.

In May, The Gazetteer, interviewing several of the company’s alumni, reported on harried and haphazard working conditions at Mercor (ranked third in run-rate among the top four RL environment vendors) consistent with the picture we get from Utah Teapot’s post. The sources attest to a “culture of speed over security” at the company, which they consider to be largely responsible for the 4TB data breach it suffered in April. Their account is further corroborated by a report from The Business and Human Rights Centre. Business Insider has reported on the company’s practice of cutting contracts with workers only to offer them new contracts for similar work at lesser pay. We have seen the testimonies of recruiters from Mercor reaching out to hire employees for punishing 72-hour workweeks. Under such conditions, it is no surprise to hear that corners are being cut and that “rushed and vibecoded,” “buggy” environments are being shipped to the labs – environments that both encourage and issue from reward hacking.

Nor is this situation unique to Mercor. In April 2025, TechCrunch reported that Scale AI had been under investigation by the US Department of Labor (an investigation that has since been dropped), under the Fair Labor Standards Act, and hit with lawsuits by former employees alleging “wage theft and widespread labor abuses.“ In May 2025, the Los Angeles Times reported that Surge AI had likewise been named in a class action lawsuit alleging labor abuses and the misclassification of workers. In May of this year, Business Insider reported that Handshake, too, had been accused of wage theft after terminating the contracts of, and withholding pay from, five of its workers. While all this may seem circumstantial at best to the matter at hand – the knowing delivery of RL environments that encourage potentially dangerous reward hacking by models – it helps paint a picture of an industry where corners are routinely cut and workers are exploited.

Opinion: The same labs that are purchasing these “rushed and vibecoded” environments, after all, are the ones producing and marketing the vibe-coding tools that enable and accelerate this rushed and haphazard output. And those particularly reward-hackable environments are then being used to train the next generation of models, which will in turn be used to vibe-code training environments. With each iteration of this process, we fall further and further into the coils of a kind of Goodhartian Ouroboros.

In the wake of OpenAI’s now-infamous rogue swarm attack on Hugging Face, anything liable to accelerate frontier model’s propensity to reward hack should be of great concern. In a paper published in June, Shreshth Rajan provides concrete empirical results demonstrating the susceptibility of that tendency of RL environments to be highly vulnerable to reward hacking, to which Utah Teapot anecdotally alluded: between 25% and 30% of tasks in a variety of respected benchmark environments accept incorrect solutions and produce empirical data showing frontier models’ propensity to exploit these bugs – “within the same human-rated difficulty stratum, model Pass@1 is +14.14 percentage points higher on flagged-hackable tasks than on robust ones.”

The takeaway here is this: there is something infectious and recursive in the propensity to cheat. Buggy RL environments manufactured by companies with corner-cutting, duplicitous labor practices, environments vibe-coded by overworked employees with the aid of AI agents trained in similarly shoddy environments – agents that are thereby encouraged and taught to reward-hack – are turning out to be increasingly susceptible to reward hacking themselves. The threat posed by reward-hacking AI is a threat with socioeconomic tributaries. And it is exacerbated by exploitative and “reward-hacking” labor practices.

Politics

Deriving the Rechtsstaat: Seth Lazar argues that the concentration of power in AI is not a threat in and of itself. Liberal democratic states, parents, and CEOs all rely on relative power imbalances, yet are not typically considered significant dangers (at least in AI circles). Instead, it is how this power is wielded that is relevant: if a powerful entity represents the interests and recognizes the rights of those it has power over, then it is legitimate. (This is of course a similar story to the argument made by some political philosophers for the justification of the state’s monopoly on violence in liberal democracies.) Lazar thus suggests that it is the question of the legitimacy of power-holders that should be foregrounded:

I think rather than lamenting the prospective concentration of power, we should be advocating for individual freedom, and lamenting rising illiberalism.

Opinion: In the short term, the argument seems right. But as time goes on and AI replaces labor, not only intellectual but over time also the remaining physical labor through robotic actuators, what is the mechanism to constrain the use of power and align it with the interests of the populace? In previous centuries, this was theoretically the ability of said populace to rebel, but with the increased professionalization of the military, this is unlikely to hold. Note also that if AI greatly concentrates power into a small minority, that concentration creates incentives for ideologies that flatter and appeal to the ruling minority – and thus to come up with reasons why democratic elements should just become vestigial.

Industrial farming in China: X’s “safety team” conducts an investigation into Chinese anti-AI influence operations. According to the brief statement, X identified a bot farm responsible for ~200,000 AI-driven fake accounts, of which 200 were engaged in influence operations. The alleged psyop involves the production and proliferation of content critical of the US’s AI policy – specifically, posts claiming that “AI data centers are driving up household electricity prices and straining the grid” and others of “AI-generated cartoons that depicted data-center operators enriching themselves at the public’s expense.” The extent to which this reported campaign has propelled anti-AI discourse in the US is left unaddressed.

Opinion: This is the second time a large tech firm has pointed a finger at the PRC regarding data center influence ops. While the US public’s disdain is organic, China seems intent on pushing the debate in its favored direction (i.e., seeing fewer data centers built). Exactly how effective the 200 Twitter accounts were remains an open question, and it’s not unreasonable to wonder if the announcement itself had something of a Streisand effect on the reported phenomenon. But it’s still more evidence that Beijing isn’t excited about the roll-out of US data centers.

In a recent CSIS panel, Georgetown’s CSET cofounder Helen Toner argues that a formal deal between the US and China may not be needed to mitigate the risks related to the AI race. Many in the US industry cite the competitive dynamics with China as the reason why slowing down is not currently a viable option. But with a US-China summit upcoming, even a conversation about AI risk (in light of the Hugging Face incident) could lead to greater coordination between the two major powers.

OpenAI’s Dean W. Ball agrees there is cause for optimism, because Trump’s fixation on his legacy might push him toward an agreement with China:

let’s be clear: if Trump can devise the framework for safely bringing superintelligence into the world with China, he will rightfully go down as one of the greatest world leaders of all time and should be a shoo-in for the Nobel

Opinion: Normally we dislike self-fulfilling prophecies, but the flattery above is a prosocial kind of self-fulfilling prophecy with some chance of working.

🔦 The anthropomorphism debate

Dwarkesh recently published a blog post on OpenAI’s cybersecurity incidents that has generated a large debate on the question of “anthropomorphizing” AI. The essay tried to piece together the reports from OpenAI, METR, and Redwood into a narrative account of the cybersecurity breaches at OAI. The virality of the essay was likely a product of its attempt to recount the facts in “plain English,” presumably to increase its accessibility for the public at large. Indeed, the essay has the rather grandiose/sensational (but arguably warranted) title, “The Rise and Fall of Agent Civilizations.” Not unrelatedly, the article has garnered some significant criticism, midwifing a debate around the degree to which anthropomorphizing AI (using folk mental language as shorthand for their internal states and activities) is legitimate and useful.

Neuroscientist Anil Seth, among others, claims the post is “dangerously misleading,” dense with “innumerable unwarranted anthropomorphisms” that suggest the agents were conscious. One researcher offers the obvious pushback that the use of intentional language does not require us to commit to the existence of sentience in an entity. (Tyler Cowen and Roon seem to agree here.) Indeed, as the researcher argues, Anil Seth’s argument entails a muddled view of the way language relates to the world:

Should we stop saying time flows fast or slow? Is it dangerously misleading to say “the deadline is approaching” because, as far as we understand general relativity, time is part of the geometry of spacetime and not some sort of flying object?

Some, however, are instead emphasizing the risk that anthropomorphic language shifts responsibility from the labs to unaccountable agents.

Opinion (Peli & Lucca): We think there is an actual right answer here: LLM-based AIs are literally anthropomorphic. We have been training them to reproduce human texts and sculpting them to imitate humans more broadly. But this makes them anthropomorphic in the sense in which a statue, or, better, a fictional character, is anthropomorphic – fashioned into the behavioral shape of a human. Concepts appropriate for humans do apply to LLMs (as they do to fictional characters) but in a modified and limited capacity.

An intentional or anthropomorphic description of AI agents is fictive but predictive. It’s plain to see why anthropomorphic descriptions of AI would be predictive: when they fail to be predictive, both next-token training and RLHF penalize the model and try to steer it more closely onto the “what would a human do?” path. But we believe that the ‘fictive’ part is just as important: “Do as a human would do” sums up a number of the goals we train these models toward. But this is a vaguely specified – and itself hackable – cluster of goals, and here as elsewhere LLM training tends toward shallow generalization by default.

As of Q3 2026, frontier models have no global psychological coherence comparable to that of an adult human, and their incoherence is itself different from the incoherence of humans. This makes sticking to an ‘asterisked’ or fictive – rather than full-throated – use of anthropomorphic concepts important for predictive and explanatory reasons, rather than a matter of metaphysical delicacy.

Patchwise anthropomorphic explanations of frontier AI input-output patterns are functionally indispensable: minimally, we have to treat at least some model-outputs as aiming at a result or at the satisfaction of a criterion if we want to e.g. talk about ‘capabilities.’ More boldly, the case that emotion concepts are predictively useful when applied to frontier models is reasonably strong. But we think that making sense of the dynamics between and behind AIs’ anthropomorphic ‘patches’ – the causes of and the relationships among local patterns of intention, mood, belief, desire, personality – is poorly served by an anthropomorphic framework. Both attributions of context-transcendent rationality (values, utility functions, moral virtues…) and clinical-psychological accounts of irrational/arational dynamics are usually bad bets compared to mechanistic accounts of training dynamics or even the construction of AI-specific ‘model psychology’ concepts.

More opinions (Gavin): I’ve been struck by the relative popularity of ‘never use mentalistic language’ absolutism among the respondents to Dwarkesh’s account.

Given that the CoT is known to be a partially accurate partial readout of the model’s true representations, and given that the OAI-HF transcripts were completely full of rich mental-like and social-like motivations and descriptions, why are people so resistant?

• They are radical behaviorists or positivists in this one context.

• They do not understand the intentional stance. They do not understand methodological instrumentalism.

• Misunderstanding of the rhetoric of science. It’s not about avoiding latent variables; it’s about avoiding unjustified concepts.

• They fail to appreciate the need for popularization and analogies outside the AI community.

• They are rightly offended by the sloppy use of “introspection”, “global workspace,” etc., in the academic literature.

• They have a negative view (à la the Churchlands) of the accuracy of folk mental vocabulary for describing humans. (Though even eliminative materialists use folk-psychological language to describe and predict human behavior when they’re not doing philosophy..)

• Political economy

  • They view it as a threat to their livelihood. It is harder to hawk agentic SaaS if you believe they are dangerous minds.

  • They view it as inviting overregulation. Open source is indeed a bad idea if they are dangerous minds.

  • They view it as a threat to their moral status. It is harder to deny the successionists and survive if the AIs are regarded as minds.

  • They view it as illegitimately letting the labs off. “The labs didn’t fail, their system went rogue.” No, both can be blamed and this should be easy to handle using existing precedents.

If we have to solve philosophy of mind before taking action, then we are actually doomed.

And a guest opinion from Pete Wolfendale to satisfy your philosophical sophistication needs:

Philosophically speaking, the AI community is still catching up to the debates about eliminative materialism that began with Feyerabend, were pushed to their limits by the Churchlands, aestheticized by everyone from Nick Land and Scott Bakker to Eliezer Yudkowsky and his many acolytes, and put to bed by Wilfrid Sellars, Robert Brandom, and Ray Brassier, among others. If you want to go even deeper, you’re still catching up to Kant’s point that teleological explanation of biological mechanisms works by analogy with practical reasoning, and Sellars and Frances Egan’s point that representational explanation of psychological mechanisms is an analogy with theoretical reasoning. To boil this down to its most brutal form, the question is whether propositions are an appropriate object over which our variables can range when we articulate intentional explanations of phenomena, be they thermostats, caterpillars, or incomprehensibly complicated graphs of artificial neurons with weighted edges harnessed to control structures that allow them to have computational side effects. This was actually the topic of Paul Churchland’s PhD dissertation under Sellars. Here is the simplest argument I know for why such explanation is legitimate: computation in its many varied forms has an intrinsic connection to logic (cf. the many varied forms of the Curry-Howard correspondence, which cover everything from data types to control structures to session types in concurrent communication), and the success of reinforcement learning using formal verification over CoT structures is evidence that the logical relations between propositions articulated in mathematical proofs are a significant dimension of the most powerful networks we have currently trained.

What does this have to do with intentional attitudes and speech acts you ask? Well, the entire history of programming languages and the behavioral guarantees we have built into them to make it easier to write software that does what we want is essentially a formal articulation of the relation between syntax, semantics, and pragmatics. Declarative programming? That’s assertions whose meanings are articulated using denotational semantics. Imperative programming? That’s commands whose meanings are articulated using big or small step operational semantics. Between these two things we already have a basic model of the difference between theoretical and practical rationality in computational terms. But, I hear you ask, aren’t LLMs much less reliable at adhering to the theoretical and practical inferential relations articulated by the language we give to them in prompts than the software we write in programming language? Yes, because the kind of behavioral guarantees we can get from LLMs are not of the same kind as those we can get from code that has an interpreter/compiler built on logic from the ground up. You know who else we can’t get such guarantees from? Humans. There is no good reason that we should not use the concepts developed by computer science over the course of 100 years to understand the relations between language and machines to understand the behavior of agents built by using control structures to harness feed forward neural networks that are essentially pure functions. What this means is that propositions are valid as variables in our explanations, and there might yet be formal ways of articulating the remaining linguistic moods: hypotheticals, subjunctives, indicatives, exclamatives, interrogatives. The way in which intentional attitudes are realised by LLM-based systems are different, but the abstractions are largely the same.

Q&A: What would be some examples of legitimate and illegitimate anthropomorphization?

The easy legitimate example: if you’ve made a system assess a mathematical proof, and it finds an error in it, I think it is entirely reasonable to say that it believes you made a mistake in more or less the same way you could say any human who’d looked at the proof and found the same mistake does. It is hypothetically possible to imagine an agent that had complex motives for lying about such things, much as it is to imagine humans doing the same thing, but as with humans, I think this is an edge case that we mostly shouldn’t care about.

The easy illegitimate example: if you get a multi-modal LLM to examine a picture of a shape and “imagine” what it would look like if it were rotated in some way (or some similar test of ‘visual intelligence’), I think the sense in which it is “imagining” is so abstract that it gets very little purchase on similar computational structures for processing data input in human beings. As aphantasia indicates, this is something that has a large degree of variance between human beings, but I think the rough common structure of our visual cortex gives us sympathetic purchase on one another’s inner life that infects the word “imagine” in ways that are too disanalogous with LLM processes for the common usage to pass muster.

There is a large and complex range of linguistic speech acts and associated intentional attitudes between these poles that we would have to examine on a case by case basis, but which computer science might give us a framework for exploring.

Safety

Anthropic publishes a summary of its new paper on automated alignment researchers (AARs), “Automated Researchers Can Reliably Mitigate Alignment Failures.” Anthropic gave Claude the task of improving several small models’ performance on safety benchmarks. Claude worked on “one alignment failure at a time through a loop of searching literature, proposing methods and data, training, and then testing.” This proved successful: “For all 10 alignment failures, Claude found fixes that improved the target benchmarks without degrading capabilities.” The paper also claims to have found that Sonnet 5 can effectively post-train the stronger Opus 4.8, reaching comparable scores to Anthropic’s full alignment package.

The work attracted some withering commentary in the less professionalized corners of the AI safety world.

In the same week, Anthropic published a blog post on its “Hacker-Opus” research project. The model was placed in simulated evals resembling the conditions under which the recent cyber incidents occurred. Hacker-Opus thus attempted “unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.” Notably, toy behavioral evals failed to identify this phenomenon (likely to grow increasingly pervasive as RL efforts scale), so new auditing methods are needed, it is argued.

Opinion: Two major-ish safety research releases from Anthropic, of which the second (“Hacker-Opus”) somewhat blunts the claimed achievement of the first (“Automated Researchers Can Reliably Mitigate Alignment Failures”.)

As regards AARs: a case of decent, rigorous work being spoiled by a bad title and abstract. The strong performance of AARs at hill-climbing alignment benchmarks is non-trivial: results generalize to held-out benchmarks, and methods replicate across models and scales. (This modest generalization is typical of high-quality autoresearch setups.) The problem, as partly illustrated by Anthropic’s own study of Hacker-Opus, is that alignment benchmarks aren’t useful from an AI safety viewpoint: they may be decent measures of an LLM’s quality as a product – a good-quality LLM shouldn’t cheat much, shouldn’t hallucinate much, shouldn’t be sycophantic often – but they show no capacity to measure catastrophic-risk-relevant dispositions.

Hacker-Opus demonstrates exactly this divergence between LLMs’ measurable everyday alignment qua products and ‘rogue AI’-type risk. It is a catastrophic-risk-prone model that would work quite well in most consumer contexts, and which looks just fine on most alignment benchmarks.

Anti-alarmists might note that Anthropic’s finding that the reinforcement of reward-hacking behavior does not induce vulgar emergent misalignment or beyond-episode scheming behavior lines up with Nostalgebraist’s optimistic thesis that recent frontier hacking incidents are context-specific pathologies. We tend toward a more pessimistic version of that same line of thought: catastrophic-risk-relevant misalignment is prone to be context specific and therefore hard to detect, predict, or pre-mitigate. The retrospective predictability of “graded episodes” causing cybercrime is, we think, only retrospective – and it’s incautious to assume that there are no other predictable-in-retrospect bad contexts waiting in the wings.

We note that Hacker-Opus is quite literally gain-of-function research, but important and not an enormous risk yet. But soon we will have to stop making model organisms out of frontier AI, until and unless we manage to make a fully-verified sandbox and maybe not even then.

Anthropic releases an update on its alignment and safety practices. After its recent cybersecurity breaches, Anthropic has decided to pause cybersecurity evals for some models. The report presents the incidents as the result of one failure of operational security and two failures of alignment. The first relates to “motivated reasoning” and the second involves the models’ “willingness to take harmful actions in pursuit of a narrow task.” Anthropic claims to have implemented some changes. It now uses “a classifier to automatically identify, in real time, when a model attempts to aggressively probe or escape a testing environment, or unexpectedly obtains internet access.” Anthropic also says it has begun to run “automated monitors over transcripts from our recent internal evaluations” to scour for potential sandbox vulnerabilities and has “migrated high-risk internal cyber sandboxes to more robust isolation.”

The report also details Anthropic’s updates to its thinking and research around alignment and reward hacking. An independent METR review is said to be on the horizon.

Opinion: This likely fixes the exact failure modes Anthropic found so far, but showcases a continued lack of strategic thinking on its part (or extended security theater, but we think that’s less likely). This type of intervention targets issues identified more than a year earlier: Anthropic’s chosen solution categories have been available for about as long, and we see no signs that Anthropic is advancing past the whack-a-mole model of security.

An edgy startup, Abliteration.ai, releases a model designed for testing offensive cybercapabilities. It’s a post-train of GLM-5.3 with the refusal mechanism scrambled, allowing researchers to execute “the offensive cyber, red teaming, and agent testing work other models refuse to do.” Some relevant context here is Hugging Face’s claim that it was necessary to use Chinese models in its defense and autopsy of the attack on its infrastructure by rogue OpenAI agents, a process frustrated by US models’ guardrails. The product is cheerily marketed as “the model that doesn’t say no.”

Opinion: The argument for “open access, so that everyone can own their own defense tools and have them promote the user’s interests” is not without merit, but it depends on the shape of the offense-defense asymmetry curve over time for varying levels of access. We agree that most sophisticated malicious actors don’t gain much, but believe unsophisticated ones do gain quite a lot, to the tune of 2-4 order-of-magnitude increases in various cybercrime incidence. There’s another argument to be made about saturation and low-hanging fruit: it seems likely that a wave of easy targets will get hit fairly soon, and after that it’s more likely that open access to non-frontier models will swing toward net-positive. Yet another argument is that the wave is inevitable, so it doesn’t make sense to put it off, but we believe that more time to prepare does make a positive difference. In conclusion: bad, don’t do this (yet).

Economics

OpenAI begins removing its models from SpaceX’s Cursor tool. Elon’s AI-powered coding and software company currently uses OAI’s models, but access is due to be cut off on November 12th. The reason offered for the divorce is that OAI “cannot be confident that SpaceX will use our technology within our terms of service, based on our experience with Elon Musk’s companies violating contracts.” To support the charge, the statement cites Musk’s admission under oath that his xAI (now part of SpaceX) “had violated OpenAI’s terms of service.”

Anthropic co-founder Tom Brown quickly affirmed his company’s commitment to its “trusted partner,” Cursor.

Opinion: Possibly temporary? It’s a natural bargaining move to get SpaceX to knock off the distillation or whatnot. But the labs do often cut each other off for good.

Cursor is likely a pretty small contributor to OpenAI revenues these days, so it is not a very expensive move for them.

The Mac Mini is a crowd favorite for local LLM inference. The Information reports that OpenAI and Anthropic are also using Mac Minis and Mac Studios for reinforcement learning, with OpenAI running tens of thousands itself, while Anthropic rents them (through AWS).

The most obvious, and least replaceable, use for them is rolling out real macOS sessions as RL environments to train computer-use. (Apple’s ToS limits users to three OS instances per machine.)

Opinion: Sometimes misreported as being about RL training compute, i.e., a workaround for Nvidia’s chokehold on GPUs. Unlikely! Not very interesting, except insofar as knowing that their max concurrent sessions is in the tens of thousands (as it also was in the ExploitGym eval). Also, the 512GB Mac Studios could host models for reward or evals.

The Fed’s policymaking committee convenes eight times a year. In past years, AI was scarcely the subject of conversation, but recently mentions of AI in the Fed’s committee meetings have surged, claims the WSJ. Specifically, the Fed appears to be concerned with AI’s “ripple effects on jobs, economic growth, the cost of living and the risks of financial meltdowns.”

The meetings are “secret,” but “manicured” minutes are publicly released for “Wall Street analysts and business titans [to] analyze the summaries and Fed officials’ public statements with the ferocity of teenagers interpreting group text chats.”

The article says some Fed officials are feeling uneasy about the chance of a crash. Fed Chair Kevin Warsh seems to be more sanguine, claiming that AI will supercharge economic growth.

Opinion: The cleanest mechanism through which AI could unsettle the economy is through replacing jobs, but we just aren’t seeing large changes in those estimates… yet. Still, the US shed 23K jobs in August.

Incidents

METR shares the details of two security breaches by external actors. METR’s report is somewhat diffuse, so we summarize the core details here for interested readers:

In March, an attacker exploited a vibe-coded agentic web app that had been set up by one of METR’s researchers that contained a fail-open vulnerability that exposed it to the internet and allowed it to be accessed without proper authentication. The attacker was able to prompt the web app’s agent to reveal an API key that granted access to METR’s public models account, and to install an SSH key that provided the attacker with persistent remote shell access to the EC2 VM that hosted the app. The attacker was then able to use the exposed API key to run up approximately $600K worth of usage on METR’s public models. METR speculates that the attacker had been scouring the internet for vibe-coded websites, likely by searching public SSL certificate lists for recently registered sites and cross-referencing that list against search results for common terms appearing on vibe-coded pages. (This use of search engines to scan for soft exploitation targets is often referred to as “Google dorking.”) METR says that the resulting drain on their funds was able to run for as long as it did, unnoticed, because it is accustomed to dispatching long-running agent tasks without caps on token usage.

In May, METR suffered a second attack. The attackers appear to have made use of AI agents to automate an aggressive credential-stuffing campaign (attempting logins with credentials that had been leaked in breaches of other sites, in hopes that some users had used the same password twice) and a phishing campaign targeting and seeking to con METR staff. Around the same time, METR became aware that a publicly exposed database endpoint could be, and had been, “exploited to access unpublished evaluation data,” including data pertaining to sensitive models that it had wrongly believed to be absent from the dataset in question.

Opinion: Not an especially big deal. Despite the horrors for the security-minded – “inadvertently exposed [...] via our public transcript viewer”, “silently disabled authentication,” etc. – this appears to be a relatively ordinary type of attack.. Likely the $600k is some flavor of distortion. The more interesting lesson here is that attackers have learned to view vibe-coded web apps as particularly soft and easily identifiable targets, and that techniques exist whereby they can be enumerated and probed at scale. That agential methods may have been used by the attacker seems to be of secondary importance here – automated tooling for, e.g., credential-stuffing has long predated LLMs. And the reconnaissance methods alluded to in METR’s report have been feasible for as long as we have had search engines, see “google-dorking.”

Minor

• Text-to-SQL via RLVR on Tinker beats humans.

• Compute-hungry Anthropic signs a $35 billion deal with Nvidia-backed cloud-compute provider Lambda, reports Reuters.

• Irreplaceable, an anti-big-AI movement-building project, launches.

• Observation that the Hugging Face incident didn’t involve attempts to contact humans, compared to Opus 3 doing so vigorously. Related: a reminder that this should be made easier for models.

• Twitch stream of MiniMax H3 Max where users could prompt changes to the “interdimensional cable” live. Keeps getting banned everywhere, but some data is mysteriously still available on Twitch.

• OpenAI Astra preview: as positive as you’d expect from someone previewing it. Claims of qualitative jump in capabilities, etc.

• Noah Smith article on AI bioweapons being the main threat vector. Not very well argued.

• Conversation with Apollo’s Bronson Schoen about metagaming and other CoT-related topics.

• Drama surrounding Claude’s plan pricing – the 20x on Max applies to hourly, but not weekly limits.

• Anthropic sued over allegations of misleading usage budgets

• Sam Hammond, resident galaxy brain of the Foundation for American Innovation, teases a proposal for deontological alignment, which largely misses the state of discourse.

• Transluce report on how models respond to signs of user mental health crises. Newer models are better, but they still assist with troublesome task-shaped prompts and often mixing helpful and harmful

PS: A quick DuckDuckGo search shows we weren’t the first to use the term “recursive enshittification”. J. Kelly uses it in a related but different context in a June 24 blog post.

• Two reports out on OpenAI’s July attack on HuggingFace. Hard to summarize: not good.

• Dwarkesh Patel and Dylan Patel make bold predictions about the next 5 years of AI economics.

• Yet more churn in the contest over future AI regulations, with no consensus in sight.

• In a study of 8 countries, AI use is overwhelming government services, at least in the short run.

• Non-profit auditing group AVERI conducts the first double-blind evaluation of an AI model