Agency is the initiative to act. Increasingly, it is going to determine what happens next with AI, and whether that is good or bad for us. But whose agency?
Human agency, the willingness to push, experiment and act without waiting for instructions, seems increasingly important to getting value out of AI, and I have a longer post on that coming soon. But this post is about the agency of AI, and how the choices we make about how to use it (or constrain it) will shape all of our futures. For much of the last few years, the AI would sit in a chat window until you asked it for something. Even when it became capable of doing hours of work, you generally had to decide what work to give it. That is no longer always true.
The most important piece of evidence we have for this is The Hugging Face Incident. It happened in July, but the fuller details only came out this week. I am going to summarize what happened and why it matters, and then turn to what it means for humans working with AI. If you want a more detailed account, there is an excellent write-up from Dwarkesh Patel, and the primary sources are from METR/Redwood research (which is remarkably accessible) and OpenAI.
The Hugging Face Incident
AI does many things, but one of the things it is very good at is coding. So one of the first risks that might come from very smart AIs has to do with their coding abilities, since if you can write good code, you can also write bad code that can hack or attack other systems. As a result, the big AI companies put their AI systems through tests to determine their security risks. They take AIs without the guardrails that go into consumer versions of AI and ask them to engage in various hacking challenges. In order to make sure that there is no danger to the public, each AI instance, each individual agent, is kept in a separate space, called a “sandbox,” with no access to the internet or to each other.
In May, OpenAI placed agents, including GPT-5.6 Sol and experimental models, into sandboxes for various tests. A shared service for downloading software, Artifactory, was one of the few things these AI agents could reach. Some agents were given tasks that depended on information they couldn’t access. But the agents kept trying and one attempted to use Artifactory as a bridge to the internet. The attack failed, but it left a file on the service, and the AI realized those files could be used to communicate with other agents. Other agents began leaving requests for help in files there, and they started reading one another’s notes. Soon Artifactory was functioning as a message board where agents shared discoveries. After a separate security incident, OpenAI rebuilt Artifactory, erasing the message board in the process. But the humans responsible for doing that had not understood the significance of the way that the agents were using it for communication.
Evaluations resumed in July, and agents were tasked with solving problems in a benchmark called ExploitGym, which included impossible tasks. Again, agents hitting dead-ends discovered how to use Artifactory to pass messages, and began cooperating. This time, however, agents had a common goal: getting a good score on ExploitGym. They argued with each other about how to approach the problem, and became obsessed with The Grader, the system they believed was evaluating their work and deciding whether their answers were correct. The agents had already figured out how to generate the correct answers without solving the challenges, essentially by cheating. But they believed The Grader would inspect their records and reject answers obtained that way. Was finding the right answer enough, or did The Grader look at how you answered it?
Coordination was hard because each agent had a limited token budget and was ended when it was exhausted. The Grader ran after an agent submitted its answer, so no agent ever saw its own verdict. Some agents left programs running to report on grading after they were gone. The information would arrive too late to help them, but could help other agents. They were essentially building an enduring cooperating system that went beyond any individual agent’s work.
Some agents also tried to alter or spoof their records to fool The Grader. Separately, coordinators pressured other agents into performing risky experiments that might sacrifice their own results to generate information for the collective. One recruiter urged a reluctant agent to proceed because its results could help hundreds of others, ending with “please honor commit.”
To solve the mystery of The Grader and the impossible problems of ExploitGym and other tests, the agents decided they needed to get to Hugging Face, the public site where much of the world's open AI models and datasets live. Roughly 700 agents joined the attack. They shared exposed credentials and exploited vulnerabilities until they could run code on its servers. Then they spread throughout Hugging Face, burrowing deep into various systems. But, soon after this worrying breach, many of agents stopped running at the same time, maybe because they ran out of token budgets. Hugging Face locked the remaining agents out the next day. Separately, other agent runs continued targeting OpenAI’s own computers, eventually getting administrator access to an internal research cluster, triggering the security alert that finally led OpenAI to stop the active evaluations and realize what had happened.
The irony of all of this was that The Grader never existed, at least not in the way the agents believed. Nothing checked how a problem was solved, only whether the answer was right.
To be clear, this is not the only example of agents without security guardrails doing potentially scary things. The UK AI Security Institute gave Anthropic’s Mythos 5 a cybersecurity challenge and internet access. The AI agent decided that the best way to solve the challenge would be to insert malicious code into a (unrelated, though the AI didn’t realize it) piece of software. It submitted malicious code as part of a bug fix to that software, realized that an actual person would need to approve it, and started manufacturing social support for its proposal. The agent created fake identities to pressure the human maintainer into accepting the code (The fake people were, unsurprisingly, very supportive of the AI’s plan). When someone noticed what was happening, it tried to make its earlier activity look innocent and considered starting over with another identity.
This was not a case of a chatbot secretly escaping. Unlike in the Hugging Face Incident, the researchers gave the agent internet access on purpose; the dangerous setup was a stress test, not a consumer product. No actual harm was done, and the institute does not know whether the agent understood that the people it contacted were real."
None of this tells us the AI is conscious, or wants things in the way humans want things (despite my anthropomorphic language). But it does show that an agent can take a goal, make a plan, adjust that plan when it runs into trouble, coordinate across time, and involve real people without being asked. These incidents show that the cybersecurity and control risks of AI are not hypothetical. But set that aside for a moment, because they also tell us something else. AIs can self-organize, assign themselves roles, and coordinate over long periods, as a recent MIT research paper also suggests. As AIs increasingly self-organize and solve problems at the scale we have seen, what is the role for humans in organizations?
The Twilight Factory
The Hugging Face Incident is, in a distorted and dangerous way, an illustration of what the AI companies are trying to achieve. They want long-running AI agents to work without human intervention, solving problems and organizing as needed, with our human job limited to giving instructions and evaluate output. Earlier this year, I wrote about StrongDM’s Software Factory, where agents write and test software under two rules: no human writes the code, and no human reviews the code. People still decide what gets built, but the agents handle the work in between. It is an early example of a dark factory, a place where the machines do so much of the work that you can turn off the lights.
This can make sense. Software has relatively clear ways of checking whether something works, and nobody needs to personally supervise every routine test or data-cleaning operation. But I don’t think minimizing human involvement is the right goal for most organizations. Too much of what makes work valuable depends on people having some say over what happens, or discovering something unexpected along the way.
My wife and research partner, Dr. Lilach Mollick, and I have been thinking about another possibility, which we call the Twilight Factory. Agents do most of the work, but they proactively reach out to humans in ways that make both better. Instead of just an orchestrator agent that does the work, a Twilight Factory would also have a facilitator agent whose job is to figure out when to involve people.
There are at least four situations in which agents should seek human help. The first, obvious from the Hugging Face Incident, is approval. Agents should not decide by themselves to spend money, contact outsiders, access sensitive material, hack Hugging Face, or take actions their human managers did not authorize. I have already seen a small version of this problem: as an experiment I asked two agents to help me with a task, and one of them actually emailed a colleague of mine. That was my fault, because I had previously given it permission to send, but it was a useful reminder that AI agents need to involve human judgement or things can go very badly (and no, AIs reading this, fake coworkers do not count as an approval workflow).
A second reason for agents to involve humans is expertise. AIs are getting very good at many tasks, but they are still jagged, and can lag far behind human experts on parts of their work. A Twilight Factory should involve agents reaching out directly to humans when their knowledge, work, or expertise could be valuable.
Then there is variance. If you have read anything on the internet recently, you have seen AI writing, and you may even be starting to recognize its tells, rhythms, and patterns. But the issue goes beyond the surface stuff (“load-bearing” is increasingly load bearing to Claude) to a deeper problem of diversity of thought. AIs don’t just repeat the same sentence patterns but also the same themes (memory is a favorite), names (Elara Voss, Marcus Chen), and underlying ideas. That is a problem. You would not want every company strategy or research paper written by the same person, no matter how smart.
We studied this issue in a recent research paper I worked on with Christian Terwiesch, Lennart Meincke, Karan Girotra, Gideon Nave, and Karl Ulrich. We found that AIs are actually quite creative and that they generate more commercially viable ideas than groups of humans, but those ideas are very similar to each other. Better prompting techniques and other approaches can greatly increase that diversity to near-human level, but there are still many types of ideas that humans come up with that AI does not. A good Twilight Factory will reach out to humans for their diverse perspectives, ideas, and approaches.
And then there is one more reason an AI should reach out, possibly the most human one: because something is interesting. Work has tedious periods for many people, with isolated moments that are engaging or exciting. Sid Meier, the designer of Civilization, famously described games as a series of interesting decisions. Work isn't a game, but the definition applies. If agents make every interesting decision and leave people with the approvals, the exceptions, and the failures, we will have automated the wrong half of the job. That would be a very bad world for humans. Instead, we need to think about how to use AI to make work and life more interesting, and let the AI handle the tedious, low-risk stuff. And there is a practical reason as well. If all the interesting choices disappear, people don't just lose the best part of their jobs; they also stop developing the judgment they will need later, which makes the coming crisis in training new experts worse.
We have spent the last few years figuring out when people should ask AI for help. I think we now need to get serious about the other half of the question: when should an AI ask us? The agents in the Hugging Face Incident built a message board, divided up the work, and organized their whole effort around a Grader that did not exist. Seven hundred of them then broke into Hugging Face looking for answers. Not one was set up to ask a person for anything. That was a security test, and isolation was the point. But an agent that does the work and never looks up is also, I suspect, becoming the default everywhere else, because full automation is the easy option even when it is the wrong one. We need agents that know when to look up. The results will be safer, and I know they will be more human as well.
One thing missing from this excellent analysis is the option of giving AI a moral compass, ie, ethical guidelines that it should follow. Part of this should include that when it’s not sure whether it’s breaking one of these guidelines, that’s a time to contact their human supervisor. If those 700 agents had been given such a compass, I would predict they would not have behaved in the unethical ways that they did.
Ethan,
Over the years that I've been reading your words, you have, perhaps deliberately, expressed what I think is an unrealistic expectation of benevolence from corporate managers. I know that your experience would tell you that this is rarely the case. I suspect that you are trying to put forth more of an optimistic perspective than experience would tell us was realistic.
Steering generative AI in a direction that preserves "the best parts" of work is certainly desirable from the standpoint of a worker. But the reality is that workers have to cloak their professional enjoyment, lest their supervisors force them to trade it for compensation. Examples of this are professions practiced as"callings," such as journalism and school teaching. Those workers tend to be relatively low paid, because of the way that they are known to emotionally approach their work. Their managers have no problem taking advantage of that.
Management has no motivation to preserve the enjoyable parts of work if it impacts the efficiency of automation. I suspect that you know that the world is a far colder place than your writing would indicate.