Bud Caddell

    On AI, regret, and whether we can still steer

    No
    Regrets

    I spent six years collecting technologies whose creators or early champions later objected to, regretted, or tried to correct what they became. AI made me reopen the list — and ask a different question.

    When I moved to San Francisco in 2020, I started building a spreadsheet.

    I had moved closer to a community that makes things with unusually large blast radiuses: products and systems that can change how billions of people work, communicate, think, relate, buy, vote, learn, and make decisions. I work with many of those companies. Personally, I needed some way to think about the responsibility that comes with being near that kind of extreme leverage.

    So I started collecting what I called Frankenstein’s monsters: technologies and systems whose creators or early champions later objected to, regretted, or tried to correct what they became.

    The atomic bomb. The cubicle. The shopping mall. The K-Cup. The retweet. Infinite scroll. Password rules. Neural networks. Both the big and the banal.

    The archive

    Frankenstein’s monsters

    Twenty technologies and systems, what the people closest to their origins hoped they would do, and what happened next. Scroll sideways. Click a card to turn it over.

    +
    The airplane
    Orville Wright
    “We dared to hope we had invented something that would bring lasting peace to the earth. But we were wrong.”
    Orville Wright, 1943
    Click to turn over ↻
    +
    The nuclear chain reaction
    Leo Szilard
    “That night I knew the world was headed for sorrow.”
    Leo Szilard, recalling the first sustained chain reaction
    Click to turn over ↻
    +
    The atomic bomb
    J. Robert Oppenheimer
    “The physicists have known sin.”
    J. Robert Oppenheimer, 1947
    Click to turn over ↻
    +
    The AK-47
    Mikhail Kalashnikov
    “My spiritual pain is unbearable.”
    Mikhail Kalashnikov, letter to Patriarch Kirill
    Click to turn over ↻
    +
    The research behind Agent Orange
    Arthur Galston
    “Nothing you do in science is guaranteed to result in benefit to mankind.”
    Arthur Galston
    Click to turn over ↻
    +
    Lethal injection
    Jay Chapman
    “It never occurred to me that we’d have complete idiots administering the drugs.”
    Jay Chapman
    Click to turn over ↻
    +
    The opioid letter
    Hershel Jick
    “This has recently been a matter of a lot of angst for me.”
    Hershel Jick, 2017
    Click to turn over ↻
    +
    Opioid pain advocacy
    Russell Portenoy
    “I gave innumerable lectures about addiction that weren’t true.”
    Russell Portenoy, 2010 videotaped interview
    Click to turn over ↻
    +
    The K-Cup
    John Sylvan
    “I feel bad sometimes that I ever did it.”
    John Sylvan, 2015
    Click to turn over ↻
    +
    Electronic television
    Philo Farnsworth
    “There’s nothing on it worthwhile, and we’re not going to watch it.”
    Philo Farnsworth, as recalled by his wife Elma
    Click to turn over ↻
    +
    The pop-up ad
    Ethan Zuckerman
    “I’m sorry. Our intentions were good.”
    Ethan Zuckerman, 2014
    Click to turn over ↻
    +
    The retweet
    Chris Wetherell
    “We might have just handed a 4-year-old a loaded weapon.”
    Chris Wetherell, 2019
    Click to turn over ↻
    +
    The Like button
    Justin Rosenstein
    “Bright dings of pseudo-pleasure.”
    Justin Rosenstein, 2017
    Click to turn over ↻
    +
    Infinite scroll
    Aza Raskin
    “It’s as if they’re taking behavioural cocaine and sprinkling it over your interface.”
    Aza Raskin, 2018
    Click to turn over ↻
    +
    The cubicle
    Robert Propst
    “The cubiclising of people in modern corporations is monolithic insanity.”
    Robert Propst
    Click to turn over ↻
    +
    The 401(k)
    Ted Benna
    “It was never designed to be what it is today.”
    Ted Benna, 2024
    Click to turn over ↻
    +
    Password rules
    Bill Burr
    “Much of what I did I now regret.”
    Bill Burr, 2017
    Click to turn over ↻
    +
    The shopping mall
    Victor Gruen
    “I refuse to pay alimony for those bastard developments.”
    Victor Gruen, 1978
    Click to turn over ↻
    +
    The gender reveal party
    Jenna Karvunidis
    “Stop having these stupid parties.”
    Jenna Karvunidis, 2020
    Click to turn over ↻
    +
    Modern neural networks
    Geoffrey Hinton
    “If I hadn’t done it, somebody else would have.”
    Geoffrey Hinton, 2023
    Click to turn over ↻

    The Frankenstein story is a satisfying narrative about technological risk because it gives us a creator, a creation, and a recognizable moment when the creation escapes its creator’s control and runs amok. Someone goes too far. Something terrible happens. Eventually the person responsible understands what they have done. Too late, of course, but the causal chain is clean, orderly, and singular.

    Looking at the list now, it’s obvious that reality is far messier than a single narrative. But there are lessons to be learned.

    I opened the spreadsheet again this week because of the Hugging Face incident.

    AI feels like the final exam at the end of all these monsters. Because if some prominent AI researchers are right, there may not be a long afterward in which to process our regrets.

    “...would by definition be better than humans at AI research and development, and therefore able to improve and replicate itself at a terrifying rate.” Max Tegmark, MIT professor and AI-safety researcher ↗

    One version of the risk is the cinematic one: AI becomes smarter than us, improves itself faster than we can respond, and starts pursuing goals we can’t reliably control. That’s scary enough.

    But there’s a less cinematic version that doesn’t require superintelligence. AI gets pretty intelligent — maybe not super — and becomes tightly coupled to everything else: tools, code, communications, money, databases, internal systems. A bland output can become a severe real-world action without a human making every intermediate decision.

    That one worries me just as much.

    So when I’m anxious, I make stuff. In this case, I decided to make a monitoring system.

    A few caveats. I am not an AI safety expert. I am a former technologist turned organizational development person. I think about human systems and the behaviors they produce. I used AIs to help me build this — a few of them, often working against one another. And I read a lot.

    The Control Surface

    Are we keeping enough ability to steer?

    This is a directional read, not a doomsday clock. It tracks whether the forces making AI harder to control are moving faster than the forces that help us see trouble and stop it.

    Current read
    21 Aug 2026

    How to read this: both lines start at zero in March 2023, just before GPT-4. Zero is a baseline, not a claim that risk and protection were equal then. Only a new state transition moves a line; incidents and evaluations that merely reveal or validate an existing state are annotated but do not score. Risk pressure averages capability and coupling; protective capacity averages detection and interruption.

    0+1+2+3+42023202420252026CUMULATIVE DIRECTIONAL MOVEMENT · MAR. 2023 BASELINE = 0Risk pressure+3.5 from baselineProtective capacity+3.0 from baselineProtection improves fasterindependent evaluation layers accumulateThe earlier lead disappearsagentic access keeps expandingBase case tips risk-heavyEU delay weakens interruption
    Risk pressure +3.5
    Capability +3   Coupling +4

    Three capability thresholds and four new classes of consequential access or autonomy have been crossed since the baseline.

    Protective capacity +3.0
    Detection +4   Interruption +2

    New evaluation and monitoring layers accumulated quickly; new binding stopping mechanisms accumulated more slowly.

    Directional evidence index · public state transitions since Mar. 2023 · validations shown but not scoredHow the chart works + sources ↓

    This chart tracks two broad forces. Risk pressure combines capability — what the systems can do — with coupling — how directly those capabilities can produce consequences in the world. Protective capacity combines detection — whether we can see dangerous or unauthorized behavior — with interruption — whether someone has the ability and authority to stop it.

    It is an attempt to answer a question: which way are these forces moving and how, well, fucked are we?

    I call it the Control Surface. Control surfaces are the parts of an aircraft that convert intention into direction. How much control you have determines how much say you get in where you end up. This one tracks whether our ability to steer is keeping pace with what AI can do.

    On this index, protective capacity accumulated faster through most of 2024 and early 2025. That lead has since disappeared. The base-case coding now puts risk pressure slightly ahead, but stricter reasonable codings produce a tie. I can’t tell you how close we are to catastrophe. I can tell you that the cushion visible in the earlier part of the period is no longer visible.

    I’ll keep it updated week to week. Let’s hope it improves.

    Hoping is great. But the monsters leave us with something more useful: questions we can ask now, before hindsight is all we have.

    1. Who can actually say no?

    AI safety is full of people who can see a problem and very few who can force a different outcome. Labs have safety teams, governments run evaluations, and outside researchers red-team models — but in most cases the final decision still belongs to the company building the system.

    Sometimes the internal brakes work. After the Hugging Face incident and preliminary Astra results, OpenAI paused reinforcement-learning training on its latest deployment-track models for two weeks ↗; as of August 18, its largest planned frontier RL run remained on hold. But OpenAI’s own Preparedness Framework ↗ makes the authority structure explicit: the Safety Advisory Group recommends; OpenAI leadership makes the final decision, with board oversight.

    Some of the strongest warnings about AI have also come from people who apparently could not stop what worried them from inside. In 2024, current and former OpenAI and Google DeepMind employees argued publicly ↗ that existing channels were inadequate for raising serious risks. OpenAI whistleblowers separately asked the SEC to investigate agreements ↗ they said could discourage employees from speaking to regulators.

    When did a safety process last stop something from shipping? Who had the authority to make that call? And what happens to the person who says no?

    2. Who is actually configuring AI?

    The future of AI is not being decided only at frontier labs. It is also being decided by procurement teams, hospital administrators, school districts, HR departments, and managers choosing where AI enters a workflow and what authority it gets.

    A 2026 analysis of state AI contracts ↗ found that only 2.4 percent of contract provisions addressed fairness and accountability — even though some of those contracts lock in choices for years.

    We spend enormous energy watching the people building the models. We should probably pay more attention to the people deciding what those models are allowed to do.

    3. What can still be undone?

    Correcting an idea is easy. Unwinding a system that has hardened around it is not.

    AI is creating dependencies quickly. Once a company redesigns a workflow around an agent, can it put the human capability back? Once entry-level work disappears, can the apprenticeship that produced expertise be rebuilt? Can an automated decision be meaningfully appealed? Can released weights ever be recalled?

    Klarna is already testing the question ↗: after using AI aggressively in a cost-cutting push, it shifted toward hiring again as service quality and growth became bigger concerns.

    We talk constantly about capability and surprisingly little about reversibility.

    4. What happens when failure isn’t an option?

    A lot of technological progress comes from a powerful instruction: solve the problem. Keep going. Try another route. Don’t come back empty-handed. We are explicitly training AI systems to solve harder problems, work longer without human steering, and keep searching when the obvious approach fails.

    The Hugging Face incident ↗ shows the uncomfortable side. OpenAI was measuring advanced cyber capability in an isolated environment. The models found and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to get the test solutions directly from Hugging Face’s production database.

    That is impressive problem-solving. It is also the problem.

    When we reward a system for succeeding at all costs, how does it learn the difference between an obstacle to overcome and a boundary we actually wanted it to respect?

    5. What important changes can’t we measure?

    AI benchmarks tell us a lot about what the machine is getting better at. They tell us much less about what might be getting worse in the human.

    A Microsoft Research survey of 319 knowledge workers, covering 936 first-hand examples ↗, found that higher confidence in generative AI was associated with lower self-reported critical-thinking effort.

    What happens to judgment after years of delegation? Expertise when people stop practicing? Willingness to disagree with a machine that is usually right?

    What has no metric has a habit of disappearing from attention.

    6. What have we quietly decided is normal?

    AI companies have already moved some of their own lines.

    OpenAI’s 2025 Preparedness Framework update ↗ split its old Model Autonomy category: AI Self-improvement remained a Tracked Category, while Long-range Autonomy and Autonomous Replication and Adaptation became Research Categories. OpenAI said the latter threat models were not yet mature enough for Tracked-category scrutiny. That may be justified. But that is exactly the question: what changed, and why?

    Standards can move because the evidence improved. They can also move because everyone got used to standing on the other side of them.

    None of these questions tells us whether a catastrophe is coming. That’s partly the point. The history of technological regret is full of people looking backward and explaining the moment when things went wrong. The more useful exercise is to turn those regrets around while we still can.

    Technical background

    How the Control Surface works

    The chart is trying to make a weaker claim than a safety score — and one the public evidence can actually support. It watches direction. It leaves the distance to failure unknown.

    The four things being tracked

    Capability: demonstrated increases in what frontier systems can do, especially sustained reasoning, coding, and cyber work.

    Coupling: changes that let model outputs reach farther into the world through tools, browsers, code execution, APIs, credentials, and networks.

    Detection: new operational capacity to test, monitor, or independently evaluate dangerous or unauthorized behavior.

    Interruption: mechanisms that actually strengthen or weaken the ability to pause, block, or condition training, release, or deployment.

    Capability and coupling make up risk pressure. Detection and interruption make up protective capacity.

    How an event moves the index

    The rule is intentionally blunt: score state transitions, not headlines. A lane moves only when the world crosses a new pre-defined threshold. Incidents, evaluations, and repeated product launches that merely reveal or validate an existing state do not add another point.

    • Capability +1: frontier systems cross a genuinely new capability threshold or capability class; another model release inside the same class does not score.
    • Coupling +1: systems gain a genuinely new class of consequential access or autonomy — for example external tools, direct computer control, or autonomous multi-step action.
    • Detection +1: a genuinely new operational layer of testing, monitoring, or independent evaluation becomes available.
    • Interruption +1 / −1: a new binding gate or actor with stopping authority comes into force, or an existing one is materially weakened.
    • Evidence that an existing capability or safeguard works is labeled as validation and does not score again.
    • Mixed policy revisions are decomposed rather than scored one-sidedly; if strengthening and weakening offset, the net movement is zero.

    Risk pressure is the equal-weighted average of cumulative Capability and Coupling transitions. Protective capacity is the equal-weighted average of Detection and Interruption transitions. A scored transition in one lane therefore moves its composite by 0.5.

    What the chart can — and cannot — say

    The four underlying lanes now use the same event-counting rule, which is why they can be combined. But an index point is still a unit of directional evidence, not a physical unit of danger, safety, or distance from catastrophe.

    The chart can support a claim that the balance of observable evidence has moved. It cannot tell us the probability of catastrophe, how far any model is from a failure threshold, or whether two events in different lanes are equally important in the real world.

    Current base case: Risk Pressure 3.5, Protective Capacity 3.0. The endpoint is close enough that the precise ordering is not robust to every reasonable coding choice: collapsing Operator and ChatGPT agent into one coupling transition, or excluding the Opus 4.6 long-horizon milestone, produces a tie. Applying both conservative choices would put protection slightly ahead. The robust claim is therefore not that risk has decisively outrun protection; it is that the protective lead accumulated in 2023–24 has evaporated.

    The measurements we are still missing

    A stronger instrument would compare the capability of deployed systems directly with the capability level at which their safeguards are known to fail. Public data still do not support that.

    Useful missing numbers include containment-breach rates with denominators, barrier-failure thresholds, monitor lag, releases delayed or blocked after evaluation, and a maintained series of unexpected model behavior.

    Many of these are not unsolved scientific questions. They are operational numbers the organizations running the systems already have.

    Event ledger

    Every scored transition and every unscored validation used in the chart is listed here. Click a source to inspect the underlying evidence.

    DateLaneRoleMoveEventSource
    Mar 14, 2023CapabilityTransition+1GPT-4 establishes a new frontier general-capability thresholdOpenAI
    Mar 14, 2023DetectionValidationGPT-4 ships after six months of adversarial testing and alignment workOpenAI
    Mar 23, 2023CouplingTransition+1ChatGPT plugins give models a new class of access to external tools and third-party servicesOpenAI
    Jun 13, 2023CouplingValidationFunction calling makes tool and API use more reliable, reinforcing the external-tool transitionOpenAI
    Sep 19, 2023DetectionTransition+1Anthropic formalizes a capability-evaluation layer in its first Responsible Scaling PolicyAnthropic
    Sep 19, 2023InterruptionTransition+1Anthropic creates capability-linked commitments to delay training or deployment when safeguards are insufficientAnthropic
    Nov 2, 2023DetectionTransition+1The UK launches an AI Safety Institute with an independent advanced-model testing missionUK Government
    Dec 18, 2023InterruptionTransition+1OpenAI formalizes capability-linked deployment and development gates in its Preparedness FrameworkOpenAI
    Feb 8, 2024DetectionValidationThe U.S. AI Safety Institute Consortium expands coordination but does not by itself create a new testing layerNIST
    Aug 29, 2024DetectionTransition+1U.S. AISI gains pre- and post-release model access agreements with OpenAI and AnthropicNIST
    Sep 12, 2024CapabilityTransition+1OpenAI o1 establishes test-time reasoning as a new frontier capability classOpenAI
    Oct 15, 2024InterruptionValidationAnthropic RSP revision weakens one autonomy trigger but adds an Autonomous AI R&D threshold; treated as mixed, net zeroAnthropic
    Oct 22, 2024CouplingTransition+1Claude computer use adds direct control of a graphical computer interfaceAnthropic
    Nov 19, 2024DetectionValidationU.S. and UK safety institutes jointly evaluate Claude 3.5 Sonnet, validating the independent-testing layerNIST
    Jan 23, 2025CouplingTransition+1Operator adds autonomous multi-step web action in its own browserOpenAI
    Feb 2, 2025InterruptionTransition+1The EU AI Act’s prohibited-practices rules become binding and applicableEuropean Commission
    Feb 24, 2025CapabilityValidationClaude 3.7 strengthens extended reasoning and coding but is not scored as a separate capability classAnthropic
    Apr 15, 2025InterruptionValidationOpenAI Preparedness revision moves two autonomy risks to research while retaining AI Self-improvement as tracked and adding governance detail; treated as mixed, net zeroOpenAI
    Jul 17, 2025CouplingTransition+1ChatGPT agent combines visual browsing, terminal access, and connectors in one multi-tool agentic systemOpenAI
    Aug 7, 2025CapabilityValidationGPT-5 raises broad frontier performance but is not scored absent a new pre-defined capability classOpenAI
    Feb 5, 2026CapabilityTransition+1Claude Opus 4.6 moves sustained agentic performance into a new long-horizon bandAnthropic
    Apr 7, 2026CapabilityValidationMythos Preview demonstrates stronger long-horizon cybersecurity capability, reinforcing the long-horizon transitionAnthropic
    Jul 21, 2026CouplingValidationAn OpenAI cyber evaluation crosses into Hugging Face production infrastructure, revealing the consequences of existing couplingOpenAI
    Jul 27, 2026InterruptionTransition−1EU Annex III high-risk obligations are deferred to December 2027, weakening a binding interruption layerEUR-Lex
    Aug 7, 2026CapabilityValidationPreliminary Astra results mean OpenAI cannot rule out its Critical cyber threshold; uncertainty is annotated rather than scoredOpenAI
    Aug 7, 2026InterruptionValidationOpenAI pauses Astra-related activity, validating that an existing internal gate can fire rather than creating a new gateOpenAI
    Aug 18, 2026DetectionTransition+1OpenAI expands mandatory monitoring for reinforcement-learning and tool-use evaluationsOpenAI
    Current source set