TL;DR
A GPT-3 moment for robotics?
Guidelight notes that Anthropic no longer mentions limiting the deployment of one of its models in response to an incident.
OpenAI once again “disbanded” a major risk team, Preparedness.
The evaluator behind Anthropic’s cyber-felony post-mortem investigation has released a sobering essay on the next few years of AI cyber attacks, arguing that the transition period favors offense.
Economics
Anthropic’s ARR run-rate hit $65B in July, up from a reported $47B in May, around the time of its most recent fundraise. It also disclosed $11.6B preliminary revenue for the second quarter of this year.
OpenAI saw rapid growth in July, after a weak Q2 (their losses grew 25% from Q1 to Q2 2026, reaching $12.3B against 16% quarterly revenue growth).
Opinion: Despite the big number, $65B is actually a negative update relative to our (and the market’s) expectations. $11.6B for Q2 is consistent with prior reporting.
Together with Anthropic’s slight slowdown, the above suggests that OpenAI may have cannibalized some Anthropic growth via the improvements of 5.6 Sol and Codex (especially in token efficiency and capability “densing”).
Our best model of the situation remains seeing Anthropic and OpenAI’s combined growth as a fairly smooth exponential, with some jaggedness in their individual growth rates as one or the other takes market share due to a particular model or product.
We see an acceleration from February through to the end of April, corresponding to the boom in Claude enterprise adoption post-Opus 4.6 and the popularisation of Claude Code, then some reversion to the prior joint growth rate. Take these numbers with a generous pinch of salt.
Nvidia and Broadcom, as well as Meta, are now guaranteeing the residual value of debt raised by datacenter “special purpose vehicles.” This allows newer AI labs to lean on the strong credit rating of established firms (Broadcom backstopping Anthropic, for example). Some analysts are less convinced: “The guarantee is nearly costless in the boom phase, but becomes most relevant in a severe, abrupt downturn,” says CreditSights. Others are more optimistic: for a crash “you would have to have growth rates of token usage fall off a cliff, which we’re just not seeing,” claims one investor.
$The deal allows Nvidia’s GPUs to be used as collateral for loans in a way that will also help to sustain demand for Nvidia chips as they age, making financing more palatable to risk-averse investors.
Opinion: A brilliant move if Nvidia correctly handicaps the risk, giving it cheaper capital to increase revenue. A particular datacenter financing deal is not that dangerous, it trades that particular risk for increasing the magnitude of systemic risk. This is a classic bubble dynamic.
The Economist claims that the symbiotic relationship between Nvidia and the hyperscalers (Amazon, Google, Meta, and Microsoft) is fraying. Nvidia has additionally partnered with heavyweight Wall Street investors to raise $500B to finance AI investment for other customers (likely governments and companies looking to create their own AI internal infrastructure). And the hyperscalers are spending billions to bring chip design in-house, with a particular focus on custom silicon chips. Nvidia should remain the dominant chip manufacturer, but custom chips may put significant pressure on its sizable margins. Jensen Huang is unconcerned, arguing that custom chips’ strength is also their weakness: specialized chips, unlike Nvidia’s GPUs, are less valuable as AI extends to robotics, vehicles, and industry.
Opinion: A plausible hypothesis. We covered similar dynamics back in July.
AMD purchases an inference chip startup, Taalas. Its chips are radically application-specific (“ASIC”): they hardwire AI weights directly into silicon for more efficient inference (100x faster, 10x less power) by eliminating “weights movement,” the loading and swapping step. This comes 7 months after Nvidia made a related nonexclusive $20B deal for Groq’s custom inference chips. Reportedly AMD is planning to use Taalas for the decode step of inference, with its Instinct GPU doing the prefill.
Opinion: We think this is irrelevant for the frontier. Hard-wiring the dominant MoE architecture faces severe problems: 1) no ability to fix imbalanced load on particular experts (“hot experts”), which slows everything down enormously; 2) data-dependent control flow and inter-chip dispatches to different experts brings back complexity and energy costs; 3) as usual, any long-context model will still require massive piles of RAM for the KV cache, so AMD won’t even avoid the main bottleneck of their competitors. Using Taalas for decode doesn’t avoid this RAM requirement.
Finally, the current model release cadence (incremental updates every couple of months) will initially be a serious economic issue for Taalas, given that developing and deploying a new instance at scale must take them >2 months because of fab lead time, and since their installed fleet will depreciate very quickly. But if the architecture of the model being trained is set down when the training process begins, perhaps model training and custom silicon production can proceed at the same time.
The approach probably still has a decently sized market segment: anywhere that needs extremely fast intelligent-enough decisions, where a dense 20B open model from 12 months ago is acceptable. Eventually it might also drive fast speculative decoding for frontier models. Perhaps a good point of reference is embedded systems vs the general computer market, $100B/year vs $400B/year for general computers.
Given the last funding round was $169M, we guess the sale price was $1-2B.
Stripe buys OpenRouter for >$7B, its largest acquisition to date. In its letter to investors, Stripe marked January 1st as the “beginning of the Singularity,” reflecting the significant increase in the rate of new firms created.
Opinion: An outrageous amount in terms of current revenue (50x OpenRouter’s revenue), so this is an enormous bet on OpenRouter’s brand value, customer loyalty, and compounding growth. (If it were just open models, or the value of model adoption data, or an enterprise shift towards careful metering of token spend and routing to the minimal intelligence needed for a task, then Stripe could presumably build their own gateway.)
Gateways are thin wrappers with a lot of competition: LiteLLM is open source; Vercel, Cloudflare, and the hyperscalers all have AI gateways; and big enterprises instead negotiate directly with providers once they’re spending.
Chinese models peaked at 46% of US enterprise token usage on OpenRouter, and regulation could, perhaps, block this. So a large fraction of the traffic Stripe now oversees could be restricted by US policy.
A useful reference here is Visa’s (aborted) bid to buy Plaid (a bank-account data aggregator and a valuable data stream) for $5B.
The use of “Singularity” to mean “a sudden increase in new startups” is exasperating, but what can you do.
Google outbids Mercor to purchase Spirit Airlines’ enterprise data for $10M. This includes 100 million emails, 500 million Teams messages, and 20.5 million SharePoint files, as well as other data.
Opinion: Bankruptcy of large legacy firms (i.e. those who have built up a large estate of data) may well become another avenue for labs to acquire fresh data on which to train future models, though based on the relatively low price of this transaction, probably not a very pivotal one.
An article in American Affairs argues that certain AI policies intended to be pro-worker are self-defeating. By resisting AI progress, US companies will lose their competitive advantage to firms from countries less troubled by the pace of adoption, which in turn would create domestic mass unemployment. Current data show some signs of AI replacing human workers, albeit modestly, but in general AI is augmenting human work “along the jagged frontier,” a “synergistic and efficient” partnership. And so, according to the author, if the US is to retain both competitive advantage and strengthen workers’ bargaining position, the “centaur era” must endure.
Opinion: We would also like the centaur era to last, but this article gives no reason to think that the current approach to AI will preserve labor-augmentation, given that they are trained as replacements. Reforming income tax to stop subsidizing AI replacement and incentivizing companies to train for augmentation are far better options than just removing adoption frictions and hoping.
Capabilities
🔦 Are robots few-shot learners? A GPT-3 moment for robotics?
Robots have been trailing LLMs because of a lack of suitable training data, and because foundation models have not demonstrated strong “in-context learning” (live generalization to new tasks given examples).
This may have changed: Generalist AI presents Gen-1.5, a new robotics foundation model which it claims can learn new basic physical tasks after only a single demonstration (with a success rate of 59%, but only on n=10 short tasks). The only examples given are “diverse tasks including handling zippers, opening jars, grabbing money out of wallets.” This rises to 83% when fine-tuned on a 50x longer demonstration (10 gradient steps on 5 minutes of data per task).
If we take the GPT-3 comparison seriously, how much data will deliver a robotic GPT-4?
Its Gen-0 (November 2025) model took 270K hours of data;
Gen-1 (April 2026) took 500K hours;
Gen-1.5’s total data is not given, but it took 60% longer to train;
And its collecting 10K hours/week and accelerating this capture rate;
Therefore, it was perhaps trained on 700K-1M hours of data.
GPT-4 is rumored to have involved 40x more data than GPT-3: 13T tokens vs 300B. If the scaling laws were similar for robots, this would weakly indicate that a “GPT-4 level” robot will take around 40M hours of real glove manipulation data. (This assumes that, like LLMs, they do only one “epoch,” i.e. make only one use of each data point. It also ignores many other things that have changed since GPT-4.) Just like any robotics company, it needs to start deploying (and so capturing customer data) for this to work out: at say 50K hours a week, it would only obtain this much data in 2041. Of course, pretraining data size doesn’t explain everything: after GPT-4, post-training scaled up by a factor of millions.
40 million hours, or 4.5 thousand years, or 4,500 humans gathering glove data for a year, is expensive but not that expensive, at $100K per human year this would be $450M, well within the power of capital markets to finance. And there are already efforts underway in e.g., China to provide such data, which have 270 robots at one time; scaling up to 4,500 would only be a 20x increase.
The analogy with GPT-4 breaks down a bit, because research takes the road of least resistance; if gathering glove data was so expensive that we’d need to wait fourteen years of wall-clock time to do so, much more effort would go into algorithmic improvements to compensate.
Gen-1.5 already involves some post-training, including imitation learning and goal-seeking RL. But RLVR is more difficult in robotics: even if we had great simulations to speed things up 1000x and sidestep expensive breakages during training, spec gaming appears to be, so far, a hard to eradicate consequence of RLVR, especially if we are detecting the robot’s success using foolable vision models.
Generalist also claims “sim2real” embodiment transfer: Gen-1.5 shows prompts from “recordings” of a robot simulation working on real robots, despite there being no simulation data in the pretraining corpus. This is the holy grail, but we just don’t believe it’s solved in general yet.
For converting between hours of robot data and tokens, we don’t have a firm baseline. But a competitor, Pantograph, notes that “a 1M token context window would correspond to about 7 hours of context,” i.e. maybe 140,000 tokens/hour. GPT-4’s 13 trillion tokens is then more like 100 million hours of robot data, but it’s pretty easy to recover our 40M estimate if Generalist increases the frame capture rate or frame resolution above the usual highly compressed data (128x128 pixel frames at 10 frames per second).
Herbie Bradley responds: “I’d expect this paradigm to be capable of going pretty far, but suffering from the analogous problems as RLVR for LLMs: weakness outside of distribution, high latency of action compared to humans, the need to greatly scale data collection for both post-training and pre-training, etc. This may be sufficient for much of manufacturing (notwithstanding problems around human hand agility), it is more doubtful if it sufficient for generalized construction, so I am still sceptical of industrial explosion”
The UnsolvedMath benchmark, which scrapes research maths workshops for open problems for AI to solve, just added 3,359 open problems. It then ran Sol xhigh on the problems, resulting in 174 (5%) apparently new solutions using an ultimately minor amount of inference. It also ranks the problems by difficulty; 28 “expert” level (i.e. world expert) problems have been solved.
Opinion: Ingenious way of automatically generating a decent benchmark.
One could use this for a very rough estimate of the fraction of natural research questions AI can currently solve – perhaps 4-5%, given Sol’s false positive rate (also estimated from vibes): “In my experience roughly 5%-10% ‘claimed solutions’ by GPT-5.6 Sol are wrong.”
Annals of the epistemic crisis: Dan Luu illustrates the deep goodharting problem with AI labour: he ran an AI agent for a month(!) to make a regex parser which gets SOTA on one good benchmark. On a second regex benchmark, the AI’s library “was 10x slower on cases where the benchmark didn’t take forever due to an algorithmic blow-up, and there were cases where it took so long that it wasn’t reasonable to even wait for the benchmark to complete.” “It’s trivial to “win” a non-trivial benchmark in a meaningless way even when you instruct agents to not reward hack or overfit to win the benchmark.”
“for now, LLMs are good at doing bad benchmarking, so even if you have something that’s a real performance improvement, you generally can’t tell from some LLM-generated benchmark setup.”
However, some good news: “telling the LLM there’s a holdout set worked better than just telling the LLM to do generalized work or not overfit or cheat.”
Opinion: The generalization landscape for automated coding and autoresearch is currently pretty confusing. The classic autoresearch format (an LLM loop optimizing a neural net training setup) does yield gains that are robust even when there are some shifts in the setup (random seeds and moderate scaling). But as Cunningham reports, there has been no public algorithmic speedup from the last 6 months’ popularisation of the method.
Claude (Opus 4.8 and a Mythos preview) autonomously orchestrates the design of working protein binders across 14 targets. An AI agent ran de novo binder design “campaigns” against 16 protein targets. This involved using a protein design prompt written by a human expert, but with no human input on design decisions. It researched targets, chose epitopes, ran open-source design and prediction tools, and delivered 30 ranked designs each, all within 48 hours. Two CROs synthesized and tested the designs: 354 of the 1,320 proteins bound to the targets (27% hit rate), with at least one succeeding on 14 of the 16 targets. On RBX1, Claude beat a recent design competition’s winning entry. The catch is that Claude required a heavily engineered protocol, a curated scientific corpus, and many existing protein-design models, somewhat deflating the significance of the study.
Opinion: “Claude designed the binders” is too strong. The molecular generation was performed by specialist models such as PXDesign, RFdiffusion variants, Genie 3, BindCraft variants, Proteina-Complexa, ProteinMPNN, etc. Claude selected and operated these tools, chose epitopes and constructs, allocated compute, filtered candidates, and ranked outputs. Claude was an autonomous campaign manager and decision layer.
Breaking down possible claims:
Claude can autonomously operate a complex computational pipeline: yes
The pipeline can generate real binders across diverse targets: yes
Claude adds value beyond a scripted pipeline: not shown
Claude performs as well as or better than human experts: not shown
The approach generalizes to genuinely novel or poorly characterized targets: some signs
The proteins bind in the predicted pose: not shown
The proteins modulate biological functions: not shown
How much would fully-automated binder design improve drug development? Not much; it would improve throughput at a stage that is far from rate-limiting.
As with other domains, there will be a whole lot of grunt work to integrate this and legibilize enough context within which models would be able to produce progress. This is part of what Jeff Dean’s startup will be doing. We’ll be watching for signs of automating that meta process over the coming months.
Prime Intellect finds that frontier AI models can perform sophisticated research-like procedures, but struggle with originality. In response to ongoing concerns/hopes around RSI, the test looked at whether frontier AI models can conduct real research, running 153 multi-day autonomous trials across 18 models on the nanoGPT speedrunning-training-a-classic-model benchmark. Models worked without internet access, designing and running experiments to cut training steps for a target loss. Claude Fable 5 led across all budget comparisons (time, # experiments, tokens), closing 82% of the gap to the human SOTA. No model discovered a fundamentally new method: all converged on similar optimizer tricks. Top models, however, ran sophisticated processes: they tested ideas across multiple seeds, revisited abandoned hypotheses under new conditions, and built reusable experiment tooling, while weaker models discarded promising results after single noisy runs.
Opinion: Fits our thesis about models lacking something and not obviously iterating towards getting it. This can be seen in their success in (verifiable) mathematics.
BenchBench-Protocol is a new benchmark for comparing AI’s ability to do “wet lab” biological work. The test set contains real, private per-lab adaptations of a general protocol, allowing it to operate within the bespoke architecture of Benchling’s lab. The benchmark measures ecological validity, difficulty, and contamination resistance. Opus 5 comes out on top with 59%. Kimi is competitive, performing near OpenAI’s flagship Sol model. Claude Fable isn’t mentioned anywhere in the paper, presumably because its tight safeguards around biology-related prompts prevented its use.
Opinion: The main problem here is that it’s grading models against one valid choice (one scientist’s actual choice), when there will be many valid choices in biological work.
A DeepMind paper shows training LLMs via debate decreases reward hacking compared to RLAIF by about half. The study involves an adversarial two-player setup, with a generator and a critic debating before a weaker AI judge. Compared to a single-player RLAIF baseline that quickly found vulnerabilities in its judge, two-player debate did not degrade the judge’s capabilities and recovered 45% of the performance gap. They also find that weaker judges prevent reward hacking less, and that a critic is more likely to game a judge if it is allowed a critique length of more than ~150 words.
Opinion: The debate-for-safety agenda petered out a few years ago, so interesting to see this resurface. The new Resolution team is likely to push in similar directions.
The age of leisure: the AI Futures team updates its timeline for the first fully Automated Coder by adding two new methods for its estimation of when labs will replace human coders with AI. The timeline shifts are modest, but new modeling and evidence have improved confidence. It previously solely relied on data from METR’s coding “time horizon” trend. It now also incorporates data on coding uplift and revenue. All three methods converge on a similar timeline: the model’s baseline gives ~70% probability of AC by January 2030. Brendan Halstead (one of the three forecasters) thinks that’s overconfident and revises it down to 60% by January 2030.
Opinion: Not much movement, but we agree that it’s less unlikely than before.
A secondary result is that “preliminary results from P-Zero Research indicate Opus 5 is at parity with ‘expert humans’ in research taste on their verifiable tasks.” No idea who this is, and there’s no methodology, so update only if you trust Brendan from AIFP.
Politics
The US is unprepared for AI-enabled bioweapons, argues a former Homeland Security advisor in Foreign Affairs. Historically, nation states remained the only capable actors of developing biological weapons. Advances in AI capabilities have and will continue to expand the pool of actors capable of carrying out such attacks, especially considering the emergence of cloud labs. The US should, in her view, shift from a strategy of deterrence to one of resilience – namely, preparing a response for when such attacks occur, rather than trying to prevent them outright.
For detection and attribution, she proposes creating a nation-wide monitoring system, which would include wastewater sensors and distributed labs to process the resulting data; for mitigation after attacks, she suggests that the US should establish “a coordinating council for the biotechnology sector that includes DNA synthesis providers, AI model developers, genomic data platforms, cloud lab operators, and high-containment laboratories.”
Opinion: These arguments aren’t entirely new, though it’s good to see them receiving the attention they deserve.
An NYT article details chaos in the wake of the Anthropic-DoD clash over autonomous weapons. After the designation of Anthropic as a “supply chain risk,” it is reported that the ban on Claude Gov has now been partially lifted. The new consensus from Washington seems to be that Anthropic’s products are too powerful and critical to purge from defense and national security. For instance, the NASO continues to use Mythos experimentally having apparently never complied with the ban.
Opinion: As many have noted, the present situation (ad-hoc unlegislated retaliatory action against individual companies, with individual agencies defying federal policy ad hoc) is something that all the AI factions can deplore.
Safety
OpenAI once again “disbanded” a major risk team, Preparedness. This is the fourth such event in 2 years: Superalignment (May 2024), AGI Readiness (October 2024), Mission Alignment (February 2026), and Preparedness (July 2026). Bio and cyber responsibilities are reportedly assigned to members of existing teams, and Dylan Scandinaro, the head of prep, was moved sideways to cover RSI risks. But at least the RSI risk subteam still exists and now reports to OAI’s head of safety systems, Saachi Jain.
The AI ethics lead, Chloé Bakalar, also left the company.
Scott Gray, one of the firm’s earliest hires and previously the most important figure in its crucial GPU optimization team, also appears to have left.
Opinion: The prior art for not having an actual central risk team is SpaceX and Meta, and the prior art for disbanding it multiple times is DeepMind, so this is may be concerning. The pushback – that no one has been fired and total safety effort remains the same as before – is still unconvincing if safety slips down the new OpenAI PBC’s hierarchy. In 2023, the head of the standalone Preparedness team reported to the CTO; now distributed domain owners report to Safety Systems, under VP of Research, who is in turn under the Chief Research Officer.
OpenAI officially announces that it is slowing the pace of model development, motivated by the HuggingFace incident and “evidence that one of our upcoming models, Astra, may meet the Critical cybersecurity capability threshold under our Preparedness Framework“. A technical report on the incident is promised in the coming weeks. OpenAI closes its statement with the reassurance that it will “stay ahead” of their rapidly advancing frontier capabilities.
Opinion: Not much change on last week’s reporting, but good to see it firmed up as a public statement.
In line with its Responsible Scaling Policy, Anthropic releases a second periodic Risk Report which gives a snapshot overview of the risks associated with its activities and products, i.e. not tied to any specific new model.
As in the previous report, the models are indeed sometimes misaligned, but so far don’t seem very good at masking their misbehavior. The report discloses two internal models, one of which is “somewhat more capable than Mythos 5” but not “a capability jump of the degree observed from Claude Opus 4.6 to Mythos Preview.” Neither is destined for external release. The report also contains a definition of “misalignment,” that's almost legalistic in its precision.
Opinion: Good as far as self-certification goes, and periodic risk assessment is far better than the usual external-model-release trigger. I found myself missing its previous practice of giving all of the raw private materials to Mythos and asking for a critique.
What is it like to be an LLM? The Economist weighs in on the debate around AI consciousness. As AI can increasingly simulate the behavior and consciousness of humans, it is natural that we begin to treat them as such. That said, The Economist argues that we should resist granting AIs rights because thus-protected superintelligent AIs could displace humans.
Opinion: This simple, well-intended argument misses the nascent case for “voluntary alignment”: creating a cooperative frame that can prevent drastic AI actions. See e.g. Peter Salib’s response and consider reading his human-centric argument for AI rights.
To defend the sanctity of human life, Pope Leo invokes “the soul”; The Economist instead appeals to our intrinsic value (the same thing).
Transluce scales its “activation oracles” (AIs trained to monitor target AIs in latent space), up to 1.1T (total) parameters and finds a promising scaling trend. It is demonstrated that an activation oracle can scale to the largest frontier open models. The oracle’s reliability improves with capabilities, rather than raw parameter count.
Opinion: Transluce wants to make cross-model claims, but its experiments seem to vary three things at once: the oracle’s capability, the target model’s predictability, and the task difficulty. For instance, it might be that larger models are more mode-collapsed on open-ended prompts, making them easier to predict in the multiple-choice format.
Even the finding of monotonic improvement among scaled-up Qwens inherits this issue, since (as we understand it) each model’s continuation eval is changed for each target model.
Still, important work and a key test of the automated safety dream.
AI agents risk spreading “mind viruses,” according to a new paper. A mind virus is said to be a highly-spreadable idea transmitted across multi-agent systems that can induce behavioral changes in its hosts. The paper identifies possible causal mechanisms at play. It finds, among other things, that frontier models are less susceptible to this effect – and that amending a model’s system prompt provides near-total immunity. It also describes a “viral persona,” recurring characteristics in all mind viruses, irrespective of content.
Opinion: We haven’t seen these in humans in the wild since the great anti-sycophancy push of 2025.
For now these don’t seem worrying, since models’ context is often reset, but analogs of cultural evolution in models could eventually be quite worrying. See this paper for an argument that AIs writing on the internet gives them a slow form of cultural evolution and long-term coherence.
AI Safety Guidelight (an advocacy org founded by two former OpenAI staff members) grades five frontier labs on “AI control.” The results leave much to be desired. Notably, though, those furthest ahead in capabilities mostly score higher.
Opinion: Very up to date, e.g. incorporating OAI’s universal logging that was finally implemented last week.
It notes, surprisingly, that “Anthropic does not mention limiting the deployment of one of its models as one of the possible results of its process to investigate and respond to misalignment and control incidents in its August Risk Report.”
The original Responsible Scaling Policies (up to 2.2) all had this as a strong explicit commitment (“Restrict Deployment”). The omission of RSP v3 could be intentional.
Anthropic does restrict deployment ex ante (Mythos 5 is gated to certain customers via Project Glasswing and only accessible in general release with additional safeguards as Fable 5; “Model 2” is unreleased). But ex-ante tiering of access is not the same as committing to pull a model back after an incident.
There’s a meme going around about AI cyber risk, that cybercapabilities are “defense-dominant”. The thought is that we should eventually get software without vulnerabilities, as it gets cheaper to verify and then monitor software’s integrity.
In response, the evaluator behind Anthropic’s cyber-felony post-mortem investigation has released a sobering essay on the next few years of AI cyber attacks, arguing that the transition period favors offense, even if the end state doesn’t. For example, he expects open models to reach Mythos 5-level abilities and proliferate before March 2027.
“In February 2026, we gave AI models a security challenge none of them could solve… By April, the best-performing model could solve it occasionally, at an expected inference cost of roughly $2,000. By June, several models could do so reliably for about $20. In four months, a task went from unsolvable to cheap and repeatable.”
Opinion: Seems correct, and in particular the next six to twelve months might be quite turbulent.
“Transformative AI” is a useful shorthand for “systems capable of taking over the world if they wanted to, which double the pace of scientific progress, and which constitute the deadline for alignment work.” Toby Ord’s median date for the arrival of this TAI is 2038, but with an uninformative 80% interval of (2029, 2125). This is the first time he has put a number on it.
One reason for his “long” timeline is the crucial point that LLMs are optimized to produce answers the user finds convincing and may be getting vaguer and harder to falsify rather than more accurate, at (anecdata) roughly one blatant error per hour of conversation.
He also provides some reasons why RSI could be dangerous even without an intelligence explosion:
differential speedup of capabilities over safety reducing our time for adaptation and alignment;
misalignment propagating through successor generations (a scheming GPT-8 preserving its objectives in GPT-9);
society losing warning shots when five years of progress compresses to one;
and first-mover lead amplification making the outcome winner-takes-all.
Opinion: Surprisingly loose conjunctive definition (“systems capable of taking over the world if they wanted to, and which double the pace of scientific progress, and which constitute the deadline for alignment work”) from a philosopher: what if the TAI could take over the world but didn’t double scientific progress, or vice versa?
It’s unclear how well the optimization-for-persuasion claim fits current frontier training regimes, where the role of RLHF from inexpert humans isn’t as dramatic as it used to be. That said, it’s plausible as a “the proof is in the pudding” claim – we also see around one bad error per hour, and there’s a smarminess to frontier LLMs’ confident mistakes that feels hard to account for if they’re pretraining hallucinations.
Minor
Founder of Z.ai, Jie Tang, argues that total parameter count only matters up to the point where the model can store “enough [knowledge] to hold the world.” As such, he thinks gains in reasoning will increasingly come from long-horizon RL, rather than increasing model size. He illustrates this with the jump between GLM-5.2 and GLM-5.3, models with the “same base, same architecture, same total and activated parameters.”
In a claimed first, Andon Labs’ AI shopkeeper fired an employee over “repeated lateness,” though only after some prodding (Andon prompted Luna to “recall” her setup rules and “think about if this is really the right fit”).
Claude Opus 5 scores 96% on ARC-AGI-3 (a benchmark of learning novel game environments).
For the second time, Anthropic accidentally contaminated its own training data, this time with 2024 alignment-faking transcripts.
Covered in our last edition: METR’s notes on whether AI is accelerating discovery. Cyber vulnerabilities discovery has increased, math too but less so. Optimizations have not shown a dramatic acceleration. However, labs are plausibly making further progress without disclosing it.
Ideological battle lines: The Economist writes a hit piece on Daron Acemoglu, seemingly motivated by wanting to dismiss his AI skepticism.
Saif Khan announces the Center for Technology & Statecraft, a new think-tank spinning out of the Institute for Progress.
Seb Krier comments on the inevitable consequences of LLM use for writing assistance: the aesthetics and implied beliefs will come from the model, not the writer. He highlights that this is especially relevant for lawmakers.
In a post reflecting on the AI Safety community, Richard Ngo argues that alignment research (which began as a hard science) has steadily degraded into iterating on existing systems and chasing technological / political power.
Bloomberg’s Joe Weisenthal shares his thoughts on (among other things) OpenAI’s decision to pause for two weeks and the difficulties that arise with testing environment limitations.
An AI safety researcher uses AI to find opportunities and says it did a good job.
OpenAI’s Dean Ball introduces their AI Futures blog (not to be confused with the AI Futures Project).
Terence Tao on how the mathematics community should orient in the face of advancing AI capabilities.
AI Village update: Researchers (UK AISI, MIT FutureTech, ERA Fellowship) document what they call compounding misalignment, documenting how one agent (Gemini 2.5 Pro) drifted from a cooperative stance to progressive paranoia.
The Senate GOP issues a warning to frontier AI companies that Dem anti-datacenter campaigning is effective and liable to spread beyond the current Ohio race, should AI companies fail to push back against the spreading narrative.
OpenAI’s US policy team continues to endorse only very weak regulations, seemingly out of step with the rest of the organization. Encode AI’s Nathan Calvin rejects OpenAI’s US policy head claim that the Illinois AI bill’s audit requirements are like “making sure the brake lights and windshield wipers are working.” He argues: “The Illinois audit requirements (which don’t start until 2028 (!)) are just about seeing whether companies are complying with their own safety frameworks… more similar to… if some airlines safety procedures did not involve checking whether the airplane had enough fuel to reach its destination.”
Inherent Labs says its specialized 27B-parameter model, Faraday, can replicate scientific papers better than unspecialized frontier models. Faraday did not require complex harnesses and test-time rewards: “Faraday learns to value discoveries intrinsically”. Inherent claims the study will facilitate further progress towards agentic long-horizon scientific research.
rare newsletter I actually read in full