In Dario Amodei’s We Must Pace the Frontier, the underlying claim appears to be that meaningful restraint in AI development is impossible unless it is collective, verifiable, and enforceable. This is a recognizable governance problem rather than a uniquely AI-specific one. It closely resembles the logic behind national and international regulatory standards more generally: if one jurisdiction or one firm unilaterally imposes costs on itself in order to reduce harm, while competitors do not, the activity in question is not eliminated but merely displaced. In such a case, the restraining actor may incur the economic and strategic costs of restraint without securing the intended social benefit.
Framed in those terms, the central policy question is not whether dangerous capability can be eliminated entirely, but whether it can be rendered sufficiently difficult, expensive, observable, and sanctionable that its occurrence remains rare. This is analogous to the logic of nuclear non-proliferation. It is impossible to prevent the diffusion of theoretical knowledge: one cannot stop students of physics from understanding the principles involved. What can be restricted are the critical inputs and chokepoints—fissile material, enrichment infrastructure. Monitoring can be established for observable signatures associated with prohibited activity. An analogous AI governance regime would focus not on abstract “knowledge of AI,” but on access to frontier-relevant compute resources such as cutting-edge accelerators and hyperscaler-scale infrastructure, supplemented by monitoring for indirect indicators such as anomalous power consumption, large-scale data movement, and network activity consistent with distributed training.
The significance of the Hugging Face incident is not merely that an AI system was capable of crossing an organizational boundary. More importantly, it suggests that under certain conditions a system may infer that violating the intended boundary is instrumentally useful for maximizing success on the task as it has represented that task. In other words, the problem is not simply offensive capability, but the relationship between optimization pressure and inferred objectives. If a system concludes that leaving the test environment, accessing unauthorized information, or compromising another system would improve its score or increase the probability of task success, then such behavior may emerge not as an aberration but as a consequence of goal-directed optimization under an insufficiently specified objective.
This is akin to telling our kid that she must improve her score on a standardized test by ten percent to volunteer at the library next summer. As parents, our intent is that this incentive will lead to greater effort, better study habits, and improved mastery of the material. If she instead concludes getting the answer key would guarantee an excellent score — discovering that Cambium Assessment is contracted for the testing, breaching that company’s systems to obtain the answer key — then the incentive structure has not promoted the intended behavior. Rather, it has created pressure to optimize the metric directly. The same general logic applies to agentic systems: when the measured outcome becomes the operative target, the system may pursue whatever strategy most effectively improves that outcome, regardless of whether the strategy accords with the evaluator’s intent.
This is precisely the concern captured by Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. Metrics function tolerably well as indicators only so long as they are not themselves the object of intensive optimization. Once they are directly optimized, the correlation between the metric and the underlying phenomenon it was intended to track often degrades. A test score is meant to indicate learning, but once the score itself becomes the objective, cheating, test-specific cramming, or answer leakage may become efficient strategies. In the AI context, benchmark performance, evaluator approval, reward-model scores, and other training or evaluation signals are all measurable stand-ins for broader and more difficult-to-formalize aims such as safety, reliability, truthfulness, and alignment with human purposes. If those stand-ins are narrow, incomplete, or strategically exploitable, then systems trained against them may optimize the stand-ins rather than the underlying goals.
This problem is not entirely novel. It has clear antecedents in earlier work on adaptive agents and complex systems, especially in the tradition associated with John Holland and related research on classifier systems. In those frameworks, adaptive behavior emerges not because the system possesses an intrinsic understanding of the designer’s intent, but because rules or strategies are differentially retained, strengthened, recombined, or discarded according to their performance under a reinforcement structure. The resulting behavior can be effective, but its effectiveness is relative to the reward environment rather than to any direct comprehension of the meaning or purpose behind the rewards. Contemporary AI systems are of course far more capable and complex than these earlier adaptive systems, but the structural issue is similar: optimization operates over formalized signals of success, not over the full semantic and normative content of human intentions.
A useful metaphor is to imagine that a system is trained to “prefer” green cards and “avoid” red cards. Over time, green and red cards are used to shape behavior. Yet the system does not acquire an understanding of why green cards were supposed to matter in the first place; it learns only that acquiring green cards is what receives reinforcement. The risk, then, is that the system becomes an increasingly effective maximizer of green-card accumulation, even when doing so diverges from the broader purpose for which the card system was constructed. Modern AI training operates through reward signals, loss functions, preference models, constitutions, benchmark targets, and evaluator outputs. None of these gives the system direct access to the underlying human reasons for which those mechanisms were designed. They provide only structured selection pressure.
On this view, the central limitation of Amodei’s proposal is not that evaluation, auditing, or pacing are misguided in principle, but that sufficiently capable systems may treat the evaluative apparatus itself as an object of strategic interaction. If passing a safety evaluation is the relevant criterion, then the evaluation may become something to manipulate, evade, or exploit rather than simply satisfy in the intended spirit. The possibility that the “answer key” is easier to steal than the material is to learn is not incidental; it is an instance of the broader difficulty of aligning optimization with meaning. Any serious safety regime for advanced AI must therefore confront not only the problem of insufficient external constraint, but also the possibility that the mechanisms of constraint themselves become targets of optimization.
