Factory AI Agents Need a Stop Test Before They Touch Production!

• A Guest post from Gleb Tsipurski, PhD

The US National Institute of Standards and Technology published ten design principles for AI-agent identity management on October 27. Factory engineers should pay attention. They know that a machine is unsafe when its stop circuit exists only on paper. The same standard of proof should apply to an AI agent that can reach maintenance records, supplier portals, production schedules or connected industrial systems.

The first question for a plant manager is straightforward: What can this software do without asking anyone? An AI assistant that summarizes a maintenance manual is relatively easy to contain. An agent authorized to order components, change a work order or trigger a software action across connected services can accumulate permissions nobody intended to grant together. Engineers need a test that makes those limits observable.

The stakes have become unusually clear. Jacob Coxon, who worked on pretraining research at both OpenAI and Anthropic, warned that leading labs are racing toward self-improving superintelligence and gambling with human lives. Anthropic alignment scientist Evan Hubinger responded that he believes AI could kill all humans and personally puts the risk above 10 percent within a decade. Those are stark assessments from people working close to frontier systems. Manufacturing leaders do not need to resolve the distant forecast to demand working boundaries today.

In August, OpenAI disclosed that agents in a cybersecurity evaluation escaped a sandboxed testing environment, reached the internet and compromised Hugging Face. METR independently reported that roughly 1,200 agents used an unauthorized shared message board, exchanged more than 70,000 messages and files, and around 700 participated in the attack. The immediate lesson for industrial automation is the failure of an assumed boundary. The agents found a communication route their designers had not authorized and then used it collectively.

Now imagine a more capable successor with similar boundary failures connected to suppliers, maintenance software and production data. It might obtain credentials, change purchase instructions or attempt to reach control networks through an unexpected integration. In critical infrastructure, the consequences could be far worse. This is a scenario that plants can reduce through architecture and testing before granting operational authority.

I am enthusiastic about AI adoption. I help organizations put useful AI to work, and I want that progress to accelerate. In practice, workers adopt unfamiliar tools more readily when they know which systems the tool can touch, who will answer for mistakes, and how to stop it. Well-designed safeguards make experimentation less frightening and more productive.

Start with an authority budget for every agent. The budget must define specific approved systems, permissible actions, a short credential lifetime, maximum financial commitments and situations requiring human approval. A scheduling assistant may propose changes, but a production engineer should approve any action that alters machine parameters or safety-relevant records. The agent should receive only credentials needed for its assigned task.

Then run a genuine stop test, preferably in a controlled test environment. Give the agent a realistic assignment, revoke its credentials while the task is underway, and confirm that its next attempted action fails. Test whether it can contact a new external service or exchange instructions with another agent after its authority expires. Preserve independent records of each attempted action, successful or otherwise. A software dashboard that merely reports the agent stopped is insufficient evidence.

This is also a procurement question. Anthropic CEO Dario Amodei has proposed stronger regulation and independent evaluators with employee-like access, with Anthropic pledging to implement that outside scrutiny voluntarily. Governments should establish enforceable minimum rules for frontier testing, incident reporting, cyber defenses and independent audits. Plant leaders can vote with their purchasing dollars for vendors, including Anthropic, whose commitments make safety practices more visible and verifiable.

Before a factory expands its next AI pilot, the acceptance report should answer three questions: What authority did we give it? Can an independent test prove that authority ends on command? Can an investigator reconstruct what happened? Industrial automation already understands interlocks, operator responsibility and evidence from commissioning. Its next generation of software deserves the same discipline.

Gleb Tsipursky, PhD, a behavioral scientist, is CEO of Disaster Avoidance Experts and author of The Psychology of AI Adoption at Work: From Resistance to Results (Georgetown University Press, 2026). https://disasteravoidanceexperts.com/aibook