We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Security is now a basic assumption
A new case for the "vulnpocalypse" thesis arrived this period. Sarah Guo reports that eMudhra's intermediate CA private keys were tiny numbers like 4 and 56. Anyone could recover them in seconds and mint a trusted certificate for any site. Browsers have already blocked the bad certificates, but many servers, containers and devices still trust them. She traces the problem to a single bad key-generation script from 2025, which is now forcing a "Heartbleed-scale cleanup" . Asked what share of token use will go to security in 2027, Guo relayed @dok2001's answer: ">50%" .
Leadership moves point the same way:
- Aleksander Madry is back at OpenAI after what he now calls a sabbatical. He joins @merettm because, in his words, getting safety and security right "has to become a central priority" .
- Dario Amodei defended Anthropic's "race to the top" strategy. He said Anthropic had Mythos-class models in February but held general availability until June, when stronger cyber safeguards were ready, and probably gave up "a significant chunk of a several-month model lead." He also said competitors sell against Anthropic by pointing to its strict bio classifiers .
- Clem Delangue said the Netflix documentary on the OpenAI agent attack on Hugging Face left out the lack of agent containment and the fact that closed-model safeguards blocked Hugging Face's defense. Hugging Face defended itself with an open model . This continues the guardrail-refusal story from earlier briefs.
Market sizing angle: Vanta CEO Christina Cacioppo says AI attackers mostly chain many low-severity weaknesses rather than find zero-days. That pushes defenders toward fixing at machine speed . She sees a constant pentest-and-patch product for customers who won't pay $150K for Wiz but "very well could pay 40 or 50" . Separately, Vanta is absorbing token costs rather than marking them up. About 25% of its customers use the agent flow, and it is aiming for 75% .
Where capital went
- Arda (Bob McGrew) launched publicly with a $160M Series A at a $1.5B valuation. It has spent a year building with Anduril as its design partner. Today it helps manufacturers run manual assembly lines; next it plans reconfigurable automated lines aimed at products now made by large workforces in Asia .
- Daytona raised a $60M Series B at $1B for its "Agent Cloud," led by FirstMark. Angels include Patrick Collison, Tobi Lütke and Aravind Srinivas. The team has 35 people .
- Eka Robotics: a16z led the Series A. Eka builds a Vision-Force-Action model co-designed with its own robotic hands. The founders are Pulkit Agrawal (MIT CSAIL) and Tuomas Haarnoja (formerly Google DeepMind) .
- OpenAI acquired Orchestra Bio, a drug-development coordination platform, to build on its Rosalind life-sciences models .
- Salesforce is buying Braze for about $5.66B EV, a 58% premium . Six weeks earlier, Braze's stock fell 25% after a beat-and-raise quarter with 26% growth. SaaStr's takeaway: companies growing in the 20s with little AI revenue may get their best price from a strategic acquirer, not a next-round lead .
- Workday is acquiring C1.ai, which governs access for both employees and AI agents under one lifecycle and audit trail .
Founder flows seem to agree with these rounds. One widely shared observation says smart young builders are moving from vertical agents into data/RL environments, physical AI, and wet-lab drug discovery . Nathan Benaich replied "that checks" (anecdotal).
Physical AI: design gets easy, atoms stay slow
a16z argues AI will soon become a superhuman hardware designer, but building, testing and iterating will remain bottlenecks . It names opportunities in simulation (Navier, Inviscid, Lanyon) and in suppliers and test software (Hadrian, Cofactr, Instrumental, Revel) . Pat Gelsinger gave a chip example: AI could produce a design in three months, but silicon processing takes nine . On robot data, Mecka's Josh Gao says the field is now at millions of hours, still "many orders of magnitude" short of autonomous driving . He expects pick-and-place and packaging tasks on assembly lines within one to two years .
Compute and cost
Sundar Pichai says Gemini 4 Argon processed nearly 25T tokens in its first 72 hours, about 3x Gemini 3.0, and is capacity-constrained . Google gives DeepMind first call on compute, and its cloud business is forgoing short-term revenue as a result . Customer requests for 2027 compute are "incredibly difficult" to fill . He grants that an overbuild could happen if any of roughly 2,000 components becomes a constraint . Internally, Google reports 30% more code changes and about one-third fewer rollbacks in Argon's first four weeks . SF Compute's compute market is still thin: 5% of its capacity is on the order book .
Prices keep falling. OpenAI's Karan Singhal says GPT-6 Luna beats the model that was OpenAI's best a month ago on its health benchmark, at 34x lower cost . Among the companies The Pragmatic Engineer surveyed, AI spend on API plans has roughly halved since June–July peaks, helped by moves to open models. Some subsidized subscriptions are ending, though, and those bills could rise 5–10x .
Open models and policy
The DHS order bars DeepSeek, both the company and its models, from federal civilian procurement. It is unclear how the order applies to fine-tunes or to models trained on DeepSeek outputs . DeepSeek then open-sourced DeepInfer, its internal inference stack; one estimate puts DeepSeek's token gross margin as high as 90% . China's MOFCOM proposed, but has not enacted, pre-approval for sharing frontier AI abroad, with automatic approval for open-source .
Local and sovereign deployment also moved forward. Perplexity Computer now runs fully on a Mac, with both the model and the agent on-device . Mistral's Voxtral 3 adds speech recognition, text-to-speech and a voice-agent SDK, available through its API or on customer infrastructure .
Agent infrastructure
Ramp tested several off-the-shelf agent harnesses before building its own model-agnostic one, Laminar . LangChain packaged the pieces its internal agents kept needing into Managed Deep Agents: per-user identity, scoped memory, sandboxed code execution, and scheduled runs .
Briefly
- Waymo reports 0.01 serious-injury crashes per million miles versus 0.21 for human drivers, across 271M miles. Semafor reports Waymo plans to grow from about 4,000 to 20,000 cars by end-2027, funded by a $5B loan .
- Microsoft exited Nscale's £2bn Essex data center after its grid connection slipped to the 2030s. Nscale says it will still build .
- An NBER paper estimates California's billionaire tax would raise about $30B but have a mean net present value of –$39B once departures are counted .
- Reflecting on her startup experience, Elizabeth Yin says founder operations have become less tedious: she contrasts manually maintained cap tables and a 2011 convertible-note fundraising process involving lawyer costs and physical checks with cap-table software and e-signed SAFEs today. She argues that reduced admin gives founders more time to build.
- In a Mercury-partner post, Yin promotes Mercury Payroll, saying it handles tax registrations and payroll compliance and can prefill information from a Mercury account. Her example of the problem is multi-state hiring administration: setting up a Massachusetts employee required state registrations, forms, and workers’ compensation, and she says she had to repeat the process after he moved to South Carolina. Mercury’s disclaimer says employers remain responsible for legal and regulatory obligations, while some services are provided by affiliates and third parties.
- The founding team brings nontraditional robotics experience: Josh says he previously built back-of-house restaurant software and installed kiosks, while his cofounder built a stablecoin venture that was sold to Coinbase.
- Mecca positions itself as a creator, processor, and licensor of training data for robotics foundation models. Its collection model combines consumer apps and business partnerships with custom sensing hardware, logistics, and data processing and quality gates. The founder says differentiation comes from verified data quality, hardware, operations, and software—not primarily contributor payments.
- The founder describes robotics data as having reached millions of hours, but still many orders of magnitude behind autonomous-driving data; manipulation also involves more degrees of freedom than driving. He expects commercial capability to advance task by task rather than through a single all-or-nothing breakthrough. He sees a mix of human and egocentric data, teleoperation, and in-domain data, with tactile data and simulation still to be added.
- For new manipulation use cases, he expects constrained industrial tasks to lead, citing data-center maintenance, laundry folding, and packaging/pick-and-place; he forecasts packaging and pick-and-place tasks on assembly lines within one or two years. He cautions that homes require particular diligence on safety, while describing service, cleaning, and delivery robots as already deployed in hotels in China and Southeast Asia. Mecca sees itself eventually extending from data provision into domain-data collection, post-training, and enterprise deployment support.
- OpenAI says more than 300 million people use ChatGPT weekly for health and wellness; most are free-tier users and outside the U.S., and many use it between visits for care navigation and follow-up. OpenAI says it specializes general frontier models with healthcare data at each training stage, shapes evaluation and training with a global physician network, and does not use connected personal health information for model training or ad targeting.
- OpenAI said its newly released lowest-cost frontier API model outperformed its previous best on HealthBench Professional at 34 times lower cost, while noting that the models still need improvement.
- OpenAI described two-way ChatGPT–Epic integration; UCSF is an early adopter focused on uses such as chart review, and AdventHealth reportedly achieved roughly 80% reductions in some administrative workflow costs using AI. OpenAI’s health lead said AI has not yet had a macro effect on total healthcare spending, with savings dependent on integration, workflows, deployment, and scale; he characterized OpenAI’s role as enabling care partners rather than providing all care itself.
- OpenAI sees healthcare as ripe for agents and computer-use tools despite fragmented, legacy systems, and sees potential to mine underused EHR, claims, and longitudinal data for new insights. Its health AI lead forecast that AI-guided target or molecule generation and clinical-trial prioritization and validation could contribute to new and more personalized treatments over a 5–10-year horizon.
- Vanta’s integrations with 400 tools feed a security “trust graph” used for automation and agents; its agent can surface work such as offboarding accounts, then act with user permission. About 25% of customers are in this agent workflow; Vanta is absorbing token costs to increase usage and retention, because customers compare marked-up tokens directly with model prices. The CEO says output- or outcome-based pricing is attractive but hard to define, while token credits can look like obscured token charges.
- Vanta’s team uses security and compliance experts to write task-specific agent evaluations and currently prioritizes model quality over cost; the CEO says coding agents need older codebases modularized to work effectively. A new onboarding flow has users provide messy files and screenshots for an LLM to interpret and turn into setup, replacing a conventional wizard.
- The CEO argues AI attackers can chain individually low-severity weaknesses, increasing pressure for faster fixes; she sees continuous testing and remediation as an opportunity, including for customers below the large-enterprise segment served by established security vendors. Her early-stage market read highlights cyber and go-to-market technology, calls robotics underweighted as models improve, and says AI is increasing demand and budgets for government and enterprise modernization. She also sees the ecosystem around open-source model providers as less developed than confidence in those providers would suggest.
- The post argues that AI is making software faster and cheaper to build, potentially opening startup opportunities in narrow industry and workflow niches that were previously uneconomic. Its forecast is for many more specialized products, with understanding customer problems becoming more valuable than implementation skill; it also expects most products to fail and few to become billion-dollar companies.
- A commenter predicts that AI agents and workflows built into CRMs and app builders could shift B2B SaaS from license pricing toward usage-based credits; custom tools might cost a fraction of generic products, but could also create low-quality software and switching friction.
- Adoption caveats raised in the discussion include learning curves, the need to follow strict processes using real-project expertise, and the risk that internal custom apps will be poor—constraints on the idea that cheaper software automatically displaces packaged SaaS.
ValsAI released ClaimBench, a benchmark for testing whether decision models can determine if a financial document supports, contradicts, or does not address a claim. It evaluated 15 models on 1,960 claims paired with 10-K excerpts, and ValsAI reported Decider V1.2 as the leading decision model . Perplexity CEO Aravind Srinivas characterized Perplexity Decider as Pareto-optimal on the benchmark .
- The hosts describe Theranos as a medtech cautionary case: Edison was presented as able to run about 100 tests from a tiny blood sample, but they say it could perform only roughly 12–13; when it failed, samples were run on conventional analyzers and results presented as Theranos’s own.
- The podcast says the documentary depicts Holmes and a team trying to revive Theranos by pitching investors and recruiting to make the technology work better with AI while she was about to go to prison. The hosts’ interpretation was that Holmes was complicit but may have rationalized misconduct as necessary to achieve the mission; they contrasted that rationale with the higher stakes in healthcare.
- Chronosphere grew from M3, an open-source observability project built inside Uber, into an enterprise observability company acquired by Palo Alto Networks for $3.35 billion in 2026, seven years later.
- Uber built M3 because commercial tools could not handle its observability needs at scale; its 2018 engineering post described processing 20 million data points per second, and within months LinkedIn, Alibaba, Databricks, and Walmart had deployed it in production, spurring demand for commercial support and services.
- Greylock partner Jerry Chen viewed Uber’s observability needs as an early signal of broader cloud-native scaling demands. Greylock wrote Chronosphere’s first check, investing $10 million of its $11 million Series A after a five-month pursuit; the founders skipped a conventional seed round.
Garry Tan called California’s Billionaire Tax Act an “asset seizure tax” and argued it would harm the state. The NBER post he shared estimates the act would collect roughly $30 billion but have a mean net present value of negative $39 billion after accounting for lost income taxes from departures.
Anthropic CEO Dario Amodei frames the company’s “race to the top” strategy as requiring frontier status to demonstrate responsible development and influence the industry; he says the approach has involved commercial costs and tradeoffs, while acknowledging organizational mistakes.
Amodei says Anthropic held Mythos-class models from February until general availability in June to strengthen cyber safeguards, initially limiting access to partners for cyberdefense and sacrificing part of its model lead. He also says restrictive biological safeguards produce significant false positives, frustrating biomedical users and creating a competitive disadvantage against more permissive rivals.
Other examples he cites are Anthropic’s advocacy for SB53 and chip export controls, restrictions on AI-service sales to PRC-controlled companies—including through third countries—and an announced contribution of 5% of the company, potentially more over time, to a public benefit fund. He acknowledges communication failures and says the change to Anthropic’s Responsible Scaling Policy is its most substantive point of criticism; he defends the revised focus on transparency and external review while recognizing that changing plans invites scrutiny.
- a16z’s Christian Keil argues that AI may master digital hardware design before hardware engineering: design-for-manufacturing, physical testing, and iteration remain harder because real-world data is variable, sparse, costly, slow to collect, and sometimes qualitative. He expects demand for better engineering simulation, citing Navier, Inviscid, and Lanyon, and for more manufacturing and testing capacity, including Hadrian, Cofactr, and Revel.
- Periodic Labs has set up an autonomous lab for end-to-end physical experiments on superconductors, magnets, and semiconductor materials. Ekin Dogus Cubuk argues that LLMs need to learn scientific inquiry through iteration and real-physics experiments, not simulation alone.
Garry Tan called Happenstance the “greatest unlock” for startup founders trying to reach the right people . Happenstance describes itself as a LinkedIn/Claude/ChatGPT hybrid and says it helps Ruben find a warm path to people across a network of 800,000+ .
Aleksander Madry is returning to OpenAI after what became a sabbatical, saying AI progress has entered a different regime in which the safety and security of increasingly capable AI must become a central priority; he will join @merettm to help chart a path through these challenges. Sam Altman described the safety and security challenges of confronting superintelligence as immense and said Madry has an excellent skill set for the work.
Paul Graham highlighted an analysis by Benjamin Jaros and Joshua Rauh estimating that California’s proposed Billionaire Tax Act would collect roughly $30 billion but have a mean net present value of negative $39 billion after accounting for lost income taxes from departures; Graham characterized the proposal as losing the state money.
GStack is now Capy-aware and, according to Garry Tan, works well with Capy’s software-factory feature Captains.
Martin Casado amplified a post listing alleged security issues: iPhone 11–16 were “apparently” vulnerable to remote attacks, SSL certificates were said to be broken until browsers removed eMudhra CA, and a PHP >9 CVE was expected “tomorrow”; the post framed these as recent or upcoming issues, and Casado added that the list was not exhaustive.
- Former Intel CEO Pat Gelsinger says AI could produce a chip design in three months, but silicon processing still takes nine months; reaching scale in a year and a half could make assumptions about AI workloads stale, leaving manufacturing scale as a major bottleneck to AI-driven chip innovation.
- a16z’s Christian Keil argues that AI may advance hardware design faster than physical execution: design is digital and model-legible, while design for manufacturing, building, testing, and iteration require more real-world interaction. He identifies potential demand across simulation tools (Navier, Inviscid, Lanyon), manufacturing inputs (Senra Systems, Podium Automation, Hadrian, Cofactr), and telemetry, inspection, and testing software (Sift, Instrumental, Revel); he also cites Periodic Labs’ autonomous lab as an end-to-end physical-experiment data loop.
Scott Kupor highlighted the thesis that “free is good” and that open source and open weights matter to a thriving software ecosystem.
Wabi founder Eugenia Kuyda argues that today’s chatbots are an interface bottleneck: users mostly turn to ChatGPT, Gemini, and Claude for search, writing, or homework despite models’ expanding capabilities, so a simpler, more engaging interface is needed to help people discover broader use cases. She likens current chatbots to AI’s “MS-DOS era” and describes single-player, text-only AI as the terminal era of AI interfaces .
- a16z’s Christian Keil argues that AI is nearing superhuman hardware design, but that will not equal end-to-end hardware engineering: building, testing, iterating, and designing for manufacturability require more physical-world interaction, while collecting physical data can be sparse, costly, slow, or qualitative. He identifies customer feedback as a particularly hard-to-compress step, since even many cheaply manufactured design iterations cannot tell AI what users want without that feedback.
- Keil expects the opportunity to shift toward simulation tools that let engineers iterate faster and suppliers of manufacturing inputs and services. Examples he names include Senra Systems (wire harnesses), Podium Automation (control panels), Hadrian (machined parts), Cofactr (electronic components), Instrumental (inspection), Revel (testing), and Mariana Minerals (copper); he also expects broader demand for PCB tools such as Diode as more users design hardware. These are forecasts and examples, not announcements of new launches.
Thank you for this, @_NathanCalvin (opens in new tab). I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:
We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.
Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.
We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.
As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn’t be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate.
However, I feel like your response is still not really engaging with the heart of my concern about Anthropic’s behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic.
(Fair warning: this response is a bit long)
To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world?
One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion.
Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in):
“Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.”
There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning:
“We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…”
A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on.
The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same.
Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.
When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include:
Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at.
Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development.
Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?)
Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public.
Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance.
Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments.
Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.”
Commented “you can’t do safety from second place,” after the WH SI Accords.
Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems.
Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).
But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.)
Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).
When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong?
A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount.
I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release:
“Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives.
That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.”
I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
Anthropic CEO Dario Amodei frames the company’s “race to the top” strategy as requiring frontier status to demonstrate responsible development and influence the industry; he says the approach has involved commercial costs and tradeoffs, while acknowledging organizational mistakes.
Amodei says Anthropic held Mythos-class models from February until general availability in June to strengthen cyber safeguards, initially limiting access to partners for cyberdefense and sacrificing part of its model lead. He also says restrictive biological safeguards produce significant false positives, frustrating biomedical users and creating a competitive disadvantage against more permissive rivals.
Other examples he cites are Anthropic’s advocacy for SB53 and chip export controls, restrictions on AI-service sales to PRC-controlled companies—including through third countries—and an announced contribution of 5% of the company, potentially more over time, to a public benefit fund. He acknowledges communication failures and says the change to Anthropic’s Responsible Scaling Policy is its most substantive point of criticism; he defends the revised focus on transparency and external review while recognizing that changing plans invites scrutiny.