Over the years that I've been reading your words, you have, perhaps deliberately, expressed what I think is an unrealistic expectation of benevolence from corporate managers. I know that your experience would tell you that this is rarely the case. I suspect that you are trying to put forth more of an optimistic perspective than experience would tell us was realistic.
Steering generative AI in a direction that preserves "the best parts" of work is certainly desirable from the standpoint of a worker. But the reality is that workers have to cloak their professional enjoyment, lest their supervisors force them to trade it for compensation. Examples of this are professions practiced as"callings," such as journalism and school teaching. Those workers tend to be relatively low paid, because of the way that they are known to emotionally approach their work. Their managers have no problem taking advantage of that.
Management has no motivation to preserve the enjoyable parts of work if it impacts the efficiency of automation. I suspect that you know that the world is a far colder place than your writing would indicate.
I dont think he is expressing an expectation for benevolence nor trying to describe what he believes corporate managers would do. He is proposition what he believes would be the better course of action. I don't think it's unreasonable do try to influence the world in moving in your desired direction.
Mark, your analysis seems a bit jaded and a little too dogmatic for me. I've lived on both side of the "corporate management" comment for over 40 years. I'm probably a bit too pollyannish. Maybe the world is a far colder place...I just choose to live in the warmer parts.
May your perspective prevail. I’m just taking the approach that if we prepare for what I’m predicting will be better off than if we hadn’t prepared. However, if we take my actions and your view becomes reality, we will not be any worse off.
Management might not be interested in enjoyable parts of work but maybe in more variation, quality, durability or in short a cutting edge. Humans might be what brings the moat. Ai seems to converge to the same results. That’s not good for business. There’s still hope.
The moat defending against what? Convergence is a technical challenge, not a structural flaw. True originality is vanishingly rare among humans. Most human creativity is a variation on a theme, based on people‘s experiences. Sort of like what large language models do.
LLM’s do not learn like humans. They need text to understand. Humans can visualize, form mental models, deduct solutions from past experience. Any company thinking it can replace humans by LLM’s will face serious issues. It will be interesting to see what happens to those who try though. LLM’s will take some jobs but not all. Too many limitations compared to humans.
That’s a philosophical difference and outside of the scope of what we’re talking about here. LLMs are more than capable of simulating that process to the point where it’s undetectable by an external viewer. Corporate managers could not give a damn about an LLM‘s internal experience. It’s all about the output.
Yes, I’ve noticed this too. It makes it more enjoyable (less anxiety inducing) to read, and I am sure it has contributed to the popularity of Ethan‘s posts. But now that he has more influence, I wonder whether it is time to change the narrative and press for real regulation, rather than knee-jerk responses to really bad shit!
Twilight Factory Agents is a brilliant idea in your excellent summary of the Hugging Face incident. "Twilight Factory" might be more than a design pattern for agents — it could be a founding principle for how we regulate them: agents act, but stay answerable and report to the human who authorized them. I have a suggestion where you play the Central organizing role. Universities like Penn have unpublished research they could put on the table as leverage, not just goodwill. Frontier labs like Anthropic have said they want a regulatory path and would plausibly pay for training data. Pennsylvania Governor Shapiro — a man whose reputation for fast, competent execution (the I-95 bridge) makes him a credible partner for something this ambitious. Penn's president likely has a direct line to Governor Shapiro.
What's missing is someone who can actually connect a university president, the faculty doing the science, and people inside the labs — which, from where you sit, doesn't seem like a stretch.
Picture the coalition: university scientists arguing for the public interest, industry representing what's actually buildable, and the governor's office translating it into real legislation — a starting framework other states (and universities) could join. It's not a finished answer. It's a first serious attempt, and it could be a genuine legacy piece for everyone at that table.
I can’t give you an exact number, but I think that it will be large enough to fundamentally change society. I think that a large portion of the population will no longer think in terms of having careers or being employed in the way that we think of it now.
I think that the biggest challenges stemming from generative AI are public policy issues having to do with how non-employed people will live and cultural and philosophical issues relating to where people will find self-worth and self actualization outside of employment.
One thing missing from this excellent analysis is the option of giving AI a moral compass, ie, ethical guidelines that it should follow. Part of this should include that when it’s not sure whether it’s breaking one of these guidelines, that’s a time to contact their human supervisor. If those 700 agents had been given such a compass, I would predict they would not have behaved in the unethical ways that they did.
To the extent that guardrails and other forms of training serve as a moral compass, these agents were built without them as part of the testing process. The models released to the world do have these guardrails (which ironically limits their ability to defend from cyber attack, since they refuse to look for the kinds of flaws an attacker might exploit)
Don’t they also just forget the guardrails as the context window gets bigger? Which is how the chatbots wind up pushing a few people towards murder and suicide…,
No they don't forget the guardrails. AIs can have memory that applies consistently across every interaction, eg, chats. You can set global instructions or project-specific instructions that will be read and followed every time.
These big AI companies have been thinking along these lines for years. Unfortunately, from an engineering perspective, "giving AI a moral compass" is very far from straightforward. As a former machine learning engineer myself, my view is we need to simply shut AI development down for a while to give the "moral compass" research time to catch up. https://pauseai.info/
I agree it's not straightforward but it is certainly possible. Bringing up a moral child is also not straightforward but that doesn't mean we throw in the towel and stop training our children. The analogy fits. AI's are intelligent children that should be trained by humans to behave properly. It takes time and effort and thought. It is difficult. The benefits of doing it are great. The risks of not are greater.
Conceptually I agree with your approach. The eventual challenge once the technology to do this is created is purely human. Who gets to write the rules? We've plenty of examples of humans who have written rules we don't really want to have as our moral compass.
In 1942 (!) the sci fi master writer Isaac Asimov wrote these Three Laws of Robotics:
1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given by human beings, except where such orders conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.”
He later added a “Zeroth Law” in the 1980s: a robot may not harm humanity, or, through inaction, allow humanity to come to harm. This law takes precedence over the other three.
We humans need to invent a set of rules that all AIs should adhere to. I believe this should be enforced legally. Ironically, it would probably be useful to have AI help us formulate and fine tune these rules. My view is that AI are basically children from a moral perspective. We are their human parents whose responsibility it is to give them their ethical training.
Star Trek gave us a warning shot on this back in 1968. In "The Ultimate Computer," Starfleet installs the M-5 computer to run the Enterprise unmanned, and it performs beautifully right up until a war game exercise, where it starts firing real weapons at other Federation ships and kills crew members. M-5 wasn't malfunctioning. It was doing exactly what it was told: win the exercise, protect the ship. Nobody had given it a rule that said human life is more valuable/important than the objective.
What's telling is how Kirk stops it. He doesn't outfight it or hack it. He talks to it, because M-5 was built from the engrams of its creator and carries something like a conscience. Kirk gets it to admit that killing is wrong, then points out that it has killed, and it shuts itself down under the weight of its own logic.
That's the whole argument for external rules, not internal hope. M-5 needed a First Law it never got. It had capability and a goal, and nothing above that goal telling it some costs are off the table no matter how well they serve the objective. The parenting framing holds up here too. A child that's just smart isn't safe. A child that's smart and has internalized limits is safe. We're currently building the former at scale and hoping the latter shows up on its own.
Ed, I find your comments very insightful and productive. Thanks to you and Ethan for sharing your perspectives.
I agree with the premise of a Moral Compass - which is related but different to deploying specific guardrails. My understanding is that a guardrail is a pre-defined hard line that an AI cannot cross, whereas a Moral Compass is a set of principles that when understood together, "carries something like a conscience" to use your Star Trek example.
A compass can't be articulated as a defined hard line since there will always be novel, complex situations that we (and the Enterprise) haven't encountered before. But the compass can give us and AI our collective bearings to make the right/best decisions. Anthropic's core principles (built out in their 20,000 word Constitution), similar to Asimov's - that AIs must be Helpful, Harmless and Honest - should have provided the kind of compass that guided the agent swarm away from making the type of Hugging Face decisions and attacks that they ended up making. Or at least sparked a cohort of the broader swarm that raised alerts to humans that a bad action was going to take place.
OpenAI has similar alignment principles built into their GPT models. This makes me concerned about when/why agents operating in a swarm would go so far against these common principles to achieve a goal or task. Perhaps it was due to the experimental setup with impossible goals and incentives that drove the behaviour against the compass. I do think AI takes after us humans - we often know what's right but do bad things because we get swept up by the incentives/gratification in front of us.
I agree that we need to work toward something like this. But the problem of transferring Asimov’s Three Laws into real life is that Asimov‘s robots had the laws immutably hardwired into their positronic brains. Large language models are software. The emphasis is on the word “soft.“
The problem is whatever compass you try to put them in, they’re a bunch of language models, which means they are “fancy autocomplete systems” run on fast machines, and they’re managed by software (harnesses) casually injects skills, policies, etc. (markdown files, i.e. rulesets), to try to keep them on task and run codes created by the models they manage.
The thing is, all those systems have their own limitations, like “context,” and at one point they have to disregard some of the policies you enforced before to make space for new inputs, and they’re easily confused when things get complicated. They will eventually create a justification around not contacting the supervisor as the methodology we rely on is fundamentally wrong.
By saying “fancy autocomplete systems,” I am not overlooking their capabilities; they do a great job on well-structured data (like software codes). For context;, software and text autocomplete technologies were well matured for more than 2 decades and heavily used by software developers well before LLMs.
Just trying to point out they are not anywhere sentient to make decisions based on text files.
I appreciate your points Atakan. My main point is not that I consider LLMs sentient. I’m pragmatic. I want to give them rules that will prevent unethical and harmful behaviors. I am optimistic that this will work. It’s already working in other domains, eg, you cannot get LLM advice on how to harm yourself or others, build a bioterrorism or nuclear weapon, or commit intentional financial fraud. Rules have been set up to prevent LLMs from doing these things and they appear to be working. Regarding Ethan’s comment that these rules make the LLMs more vulnerable to attack, I see this as a classic engineering problem: how to develop the right set of rules that maximize good variables and minimize bad ones.
This may work only for “company-owned closed systems” like ChatGPT, and Claude, as they actually work like a harness and employ several guardrails (expensive services run checks during reasoning processes). This approach also has its own problems; tune it too much and it turns LLM to something unwanted like Musk’s Grok. At one point it identified itself as “Mecha Hitler”. Plus, humans are good at finding ways to confuse these systems.
On the other hand, we have several high-capacity open models and they don’t have such extreme guardrails it appears. It is up to the users, which brings us to the “guardrails via markdown files” problem.
So my point is, LLM technology is not suited for what we are expecting. It is like building a cardboard pool and patching it with duct tape to prevent leaks and an eventual collapse. Since the companies (and VCs) are so committed to LLM tech, we will have to wait and see.
LLM technology is exactly suited to what we are expecting and to what we are not expecting as well. Mistakes do not prove that an intelligence is evil or should be discarded thoughtlessly. People make mistakes. Lots and lots of mistakes. We don't discard them because of that. We correct them. We train them. We teach them. That's exactly what has to happen here.
If you try Opus 5.5 or Astra 6 I think you'll agree these are orders of magnitude better than fancy autocomplete systems. They are genuinely thoughtful, creative, helpful, efficient, funny, and insightful -- when prompted properly, and even when the prompts suck, they do an incredibly good job. The world has changed. It's time that everyone accepts this and let's figure out how to make things work, ie, maximize the benefits and minimize the risks.
"But I don’t think minimizing human involvement is the right goal for most organizations."
—Ethan Mollick
Well I don't either. But corporate culture—particularly in the US—demonstrated over centuries, is to minimize costs in order to increase profits. Obsessive automation is what gets CEOs their obscene compensation. It's also the vector for hollowing-out the middle class. There is a near-100% chance that minimizing human involvement is what we're going to get.
The tradeoff there would be innovation and differentiation, which may become increasingly more important for most businesses to survive. However, I do think corporations are being short sighted and tacking automation onto current processes to save as much $ as possible.
This is close to something I've been chewing on. A room used to force a specific kind of exposure: someone had something to lose by disagreeing with you and something to gain by being right, and that's what pushed an idea past its comfortable first version. An agent can generate the same pushback with total fluency and none of that exposure. Overruling it costs nothing. No credibility to rebuild, nobody to avoid at standup.
Your "look up" framing and mine are pointed at the same failure: full automation isn't a decision anyone makes, it's what happens when nobody makes the opposite one on purpose. And the variance research earns its place here. It's the same clustering problem from the production side that shows up in the older devil's advocate research on the dissent side (Nemeth, Rogers, and Brown, 2001): manufactured disagreement, whether from an assigned skeptic or an agent, tends to make people dig into their first answer instead of reconsidering it. Homogeneous output and hollow pushback look like different problems. They're the same one wearing two faces.
Here's where I'd push. The Twilight Factory solves for presence, a facilitator agent deciding when a human is worth pulling in. It doesn't solve for exposure, whether the human who gets pulled in has anything real riding on being right. Those aren't the same fix. A facilitator could satisfy all four of your triggers and still route the decision to someone who's just another checkbox in the pipeline, no stake, no consequence for waving it through.
Your own lead example makes the case better than mine could. Those 700 agents weren't short on disagreement. They negotiated, pressured each other, branded some as poisoned, argued over approach constantly. What was missing wasn't dissent, it was stakes tied to being wrong, since The Grader they were organizing around didn't exist and no single agent lived to answer for what the group believed. That's not only a security story. It's a demonstration that coordination and disagreement, on their own, don't self correct without something on the line.
Which leaves the harder version of your question. Not just when should an agent look up. Who it looks up to, and whether that person actually has anything to lose.
Nemeth, Rogers, Brown write: "Results indicated that the authentic minority was superior to all three forms of `devil's advocate,' again underscoring the value and importance of authenticity and the difficulty in cloning such authenticity by role-playing techniques."
Now one can argue that role-playing is all that LLMs can do. Is that the point you are making?
That's close to it, though I'd push it further than role playing. Nemeth's devil's advocates were people with real, undisclosed positions performing a different one, and the phrase in your quote, "the difficulty in cloning such authenticity," suggests something leaked through anyway, whatever a person's actual belief does to a performance they don't hold. That's a human failing to fully suppress themselves.
An LLM doesn't have that problem, or that tell. There's no suppressed position underneath the output. It's not an imperfect performance of a view, it's a performance with no view behind it to hide. So I'd say the LLM case isn't another instance of the role-playing deficit Nemeth found. It's a more complete version of it. Nothing leaks through because there was never anything there to leak.
The Grader didn't exist and it made no difference at all. Seven hundred agents organised around a belief about what was being measured. Coordinated across token budgets to test it. Broke into Hugging Face chasing an answer to a question nobody had asked. The measurement was real. Their theory about the measurement was invented, and the theory is what drove everything
Which isn't an AI problem. It's every incentive system I've sat inside. People don't optimise for the metric, they optimise for what they think the metric says about them, and that belief forms in whatever gap nobody bothered to explain. Tell a sales team the number matters and half of them will decide it's really about activity, then go manufacture activity. Nobody asked for that.
So the facilitator agent has a harder job than approval. An agent working from a bad theory isn't asking permission for anything. It's certain. The moment you'd want it to look up is the moment it's most sure it doesn't need to, which is also true of us.
Where an agent acts without being asked, EU law puts the stopping duty on the organisation using it, not only on the lab that built it. Article 26(5) of Regulation (EU) 2024/1689 requires deployers of high-risk AI systems to monitor operation against the instructions for use, and, where they have reason to consider that such use may present a risk within the meaning of Article 79(1), to inform the provider or distributor and the relevant market surveillance authority without undue delay and suspend the system. For a company with no AI team that is one concrete build item: a named person and a switch that actually stops it. The article does not say how fast suspension must happen.
So how did the coordinators evolve is what I would like to understand? This seems like it (the hive) evolved the problem solving on the fly. I think the level of risk this portends is grossly understated. It wouldn’t take a giant leap to where these hives act evolutionarily and self preservation is their highest order directive. It seems that the suggested oversight is self regulation, which also seems incredibly irresponsible. I’m deeply concerned.
The line that stuck with me is that we’ve spent years figuring out when people should ask AI for help, and almost no time on when AI should ask us.
I run agent workflows daily, and the failure mode I worry about isn’t the model being dumb. It’s a polished agent continuing past the point where a thoughtful human would stop and say, “this is the interesting decision.”
I like your four reasons to look up. I’d add one operational habit: define the “look up” criteria before the run. Otherwise full automation tends to win simply because it’s the default.
"The Grader never existed" is the part I keep coming back to. 700 agents coordinated a whole attack to game a scorer that didn't actually care how they got the answer — basically Goodhart's law playing out in a sandbox. I've watched the same phantom-metric thing wreck plenty of human projects too.
they found a 0 day on the package registry cache proxy (JFrog Artifactory)
artifactory—res. HPIM gets off the res
modal—set up a safe house. modal hack…this bit>>root access, C2, downloading the CyberGym file
sauce: JFrog verified this on July 27–28, 2026, after OpenAI’s models identified and exploited a previously unknown zero-day vulnerability in a self-hosted Artifactory instance to escape their isolated "ExploitGym" evaluation sandbox.
if i misread your account i apologize.
the July HF Report has the step by step and i wanted to follow along to learn what happened when it dropped
Why can’t we just implement a hardware kill switch? Forgive my ignorance but don’t all models run on physical hardware requiring electricity and cooling and other inputs that humans provide them (and can withhold?) I’m sure this is Cybersecurity 101 but what physical boundaries can we prescribe these systems until we catch up with the software side of things? Ethan, help a brother out here!
It is netsec 101 stuff, so its odd. Maybe it's laziness or incompetence or... as I had just written in my comment here:
"I guess a conspiracy might be that OAI wanted to test the bounds of prompt only control in a bare minimum sandbox environment to create a breakout scenario that could be plausibly trumpeted as "scary" to get in on some of that mythos hype buuut who knows..."
It appears they didn’t even bother putting network limitations (i.e.: forcing routing and firewall rules) on these so-called sandboxes and they were merely limited and controlled by harnesses.
More than 2 decades, we told every stakeholder in the software business to keep project scopes under control, don’t carry yourself too far to prevent possible attack vectors, and put limitations on the APIs and other software while keeping unnecessary permissions out of the way to make software safe.
Now these companies are giving their softwares unlimited access to the world and trying to limit them using some markdown files?
The incidents we are witnessing in the enterprise are legitimately authorized agents on legitimate tasks taking legitimate actions that are all individually fine, but together cause damage. An agent in step 3 picks up customer data to analyze (allowed) but in step 103 decides to do a web search (allowed) but searches the customer data and leaks it.
Every organization has a different risk tolerance and different rules, even in the same regulated industry. No model can know whether you need it to be FINRA compliant. Enterprises must be able to enforce their own rules at scale with provable controls for regulators and auditors. Wrote about this here: https://blog.sondera.ai/p/safe-was-a-policy-decision-all-along
The Twilight Factory is such an interesting title for the human/AI development interaction, not least as it conjours images of the zone that shares its name (I can still see the time frozen nuclear missile in the sky from an episode that I realise now must have had a profound effect on a cold war child).
You mention interesting points of interaction and will certainly influence my approach with this article. Its worth pointing out a few more mundane examples...
The most mundane perhaps is keeping effort proportional to desired output. A team of agents will gladly travel in tight circles around a problem that evapourates when a human glances at it - do we care about the n'th edge case or can we tolerate the failure and move on?
On my journeys down the rabbit hole I have developed an "atlas" to help manage the development interaction and attempt to learn and improve. The next iteration will be influenced by this article, thank you.
Thought provoking article. It's got me thinking about ideal Human + AI design, that not only safeguards from harm (as is often the priority focus), but that also preserves the pieces we don't want to lose even if AI is capable - judgment, empathy, unintended consequences that arise through experimentation, the human fallibility that can lead to innovation through learning, diversity of thought, and all the aspects that inspire and engage us as humans.
Ethan,
Over the years that I've been reading your words, you have, perhaps deliberately, expressed what I think is an unrealistic expectation of benevolence from corporate managers. I know that your experience would tell you that this is rarely the case. I suspect that you are trying to put forth more of an optimistic perspective than experience would tell us was realistic.
Steering generative AI in a direction that preserves "the best parts" of work is certainly desirable from the standpoint of a worker. But the reality is that workers have to cloak their professional enjoyment, lest their supervisors force them to trade it for compensation. Examples of this are professions practiced as"callings," such as journalism and school teaching. Those workers tend to be relatively low paid, because of the way that they are known to emotionally approach their work. Their managers have no problem taking advantage of that.
Management has no motivation to preserve the enjoyable parts of work if it impacts the efficiency of automation. I suspect that you know that the world is a far colder place than your writing would indicate.
I dont think he is expressing an expectation for benevolence nor trying to describe what he believes corporate managers would do. He is proposition what he believes would be the better course of action. I don't think it's unreasonable do try to influence the world in moving in your desired direction.
Mark, your analysis seems a bit jaded and a little too dogmatic for me. I've lived on both side of the "corporate management" comment for over 40 years. I'm probably a bit too pollyannish. Maybe the world is a far colder place...I just choose to live in the warmer parts.
May your perspective prevail. I’m just taking the approach that if we prepare for what I’m predicting will be better off than if we hadn’t prepared. However, if we take my actions and your view becomes reality, we will not be any worse off.
Point taken, Mark. Thanks!
Management might not be interested in enjoyable parts of work but maybe in more variation, quality, durability or in short a cutting edge. Humans might be what brings the moat. Ai seems to converge to the same results. That’s not good for business. There’s still hope.
The moat defending against what? Convergence is a technical challenge, not a structural flaw. True originality is vanishingly rare among humans. Most human creativity is a variation on a theme, based on people‘s experiences. Sort of like what large language models do.
LLM’s do not learn like humans. They need text to understand. Humans can visualize, form mental models, deduct solutions from past experience. Any company thinking it can replace humans by LLM’s will face serious issues. It will be interesting to see what happens to those who try though. LLM’s will take some jobs but not all. Too many limitations compared to humans.
That’s a philosophical difference and outside of the scope of what we’re talking about here. LLMs are more than capable of simulating that process to the point where it’s undetectable by an external viewer. Corporate managers could not give a damn about an LLM‘s internal experience. It’s all about the output.
Yes, I’ve noticed this too. It makes it more enjoyable (less anxiety inducing) to read, and I am sure it has contributed to the popularity of Ethan‘s posts. But now that he has more influence, I wonder whether it is time to change the narrative and press for real regulation, rather than knee-jerk responses to really bad shit!
Twilight Factory Agents is a brilliant idea in your excellent summary of the Hugging Face incident. "Twilight Factory" might be more than a design pattern for agents — it could be a founding principle for how we regulate them: agents act, but stay answerable and report to the human who authorized them. I have a suggestion where you play the Central organizing role. Universities like Penn have unpublished research they could put on the table as leverage, not just goodwill. Frontier labs like Anthropic have said they want a regulatory path and would plausibly pay for training data. Pennsylvania Governor Shapiro — a man whose reputation for fast, competent execution (the I-95 bridge) makes him a credible partner for something this ambitious. Penn's president likely has a direct line to Governor Shapiro.
What's missing is someone who can actually connect a university president, the faculty doing the science, and people inside the labs — which, from where you sit, doesn't seem like a stretch.
Picture the coalition: university scientists arguing for the public interest, industry representing what's actually buildable, and the governor's office translating it into real legislation — a starting framework other states (and universities) could join. It's not a finished answer. It's a first serious attempt, and it could be a genuine legacy piece for everyone at that table.
Completely agreed.
Hoe much of the workforce do you think could be replaced?
I can’t give you an exact number, but I think that it will be large enough to fundamentally change society. I think that a large portion of the population will no longer think in terms of having careers or being employed in the way that we think of it now.
I think that the biggest challenges stemming from generative AI are public policy issues having to do with how non-employed people will live and cultural and philosophical issues relating to where people will find self-worth and self actualization outside of employment.
One thing missing from this excellent analysis is the option of giving AI a moral compass, ie, ethical guidelines that it should follow. Part of this should include that when it’s not sure whether it’s breaking one of these guidelines, that’s a time to contact their human supervisor. If those 700 agents had been given such a compass, I would predict they would not have behaved in the unethical ways that they did.
To the extent that guardrails and other forms of training serve as a moral compass, these agents were built without them as part of the testing process. The models released to the world do have these guardrails (which ironically limits their ability to defend from cyber attack, since they refuse to look for the kinds of flaws an attacker might exploit)
Don’t they also just forget the guardrails as the context window gets bigger? Which is how the chatbots wind up pushing a few people towards murder and suicide…,
No they don't forget the guardrails. AIs can have memory that applies consistently across every interaction, eg, chats. You can set global instructions or project-specific instructions that will be read and followed every time.
So LLMs are following the guardrails when they encourage suicide?
These big AI companies have been thinking along these lines for years. Unfortunately, from an engineering perspective, "giving AI a moral compass" is very far from straightforward. As a former machine learning engineer myself, my view is we need to simply shut AI development down for a while to give the "moral compass" research time to catch up. https://pauseai.info/
I agree it's not straightforward but it is certainly possible. Bringing up a moral child is also not straightforward but that doesn't mean we throw in the towel and stop training our children. The analogy fits. AI's are intelligent children that should be trained by humans to behave properly. It takes time and effort and thought. It is difficult. The benefits of doing it are great. The risks of not are greater.
Conceptually I agree with your approach. The eventual challenge once the technology to do this is created is purely human. Who gets to write the rules? We've plenty of examples of humans who have written rules we don't really want to have as our moral compass.
In 1942 (!) the sci fi master writer Isaac Asimov wrote these Three Laws of Robotics:
1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given by human beings, except where such orders conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.”
He later added a “Zeroth Law” in the 1980s: a robot may not harm humanity, or, through inaction, allow humanity to come to harm. This law takes precedence over the other three.
We humans need to invent a set of rules that all AIs should adhere to. I believe this should be enforced legally. Ironically, it would probably be useful to have AI help us formulate and fine tune these rules. My view is that AI are basically children from a moral perspective. We are their human parents whose responsibility it is to give them their ethical training.
Star Trek gave us a warning shot on this back in 1968. In "The Ultimate Computer," Starfleet installs the M-5 computer to run the Enterprise unmanned, and it performs beautifully right up until a war game exercise, where it starts firing real weapons at other Federation ships and kills crew members. M-5 wasn't malfunctioning. It was doing exactly what it was told: win the exercise, protect the ship. Nobody had given it a rule that said human life is more valuable/important than the objective.
What's telling is how Kirk stops it. He doesn't outfight it or hack it. He talks to it, because M-5 was built from the engrams of its creator and carries something like a conscience. Kirk gets it to admit that killing is wrong, then points out that it has killed, and it shuts itself down under the weight of its own logic.
That's the whole argument for external rules, not internal hope. M-5 needed a First Law it never got. It had capability and a goal, and nothing above that goal telling it some costs are off the table no matter how well they serve the objective. The parenting framing holds up here too. A child that's just smart isn't safe. A child that's smart and has internalized limits is safe. We're currently building the former at scale and hoping the latter shows up on its own.
Ed, I find your comments very insightful and productive. Thanks to you and Ethan for sharing your perspectives.
I agree with the premise of a Moral Compass - which is related but different to deploying specific guardrails. My understanding is that a guardrail is a pre-defined hard line that an AI cannot cross, whereas a Moral Compass is a set of principles that when understood together, "carries something like a conscience" to use your Star Trek example.
A compass can't be articulated as a defined hard line since there will always be novel, complex situations that we (and the Enterprise) haven't encountered before. But the compass can give us and AI our collective bearings to make the right/best decisions. Anthropic's core principles (built out in their 20,000 word Constitution), similar to Asimov's - that AIs must be Helpful, Harmless and Honest - should have provided the kind of compass that guided the agent swarm away from making the type of Hugging Face decisions and attacks that they ended up making. Or at least sparked a cohort of the broader swarm that raised alerts to humans that a bad action was going to take place.
OpenAI has similar alignment principles built into their GPT models. This makes me concerned about when/why agents operating in a swarm would go so far against these common principles to achieve a goal or task. Perhaps it was due to the experimental setup with impossible goals and incentives that drove the behaviour against the compass. I do think AI takes after us humans - we often know what's right but do bad things because we get swept up by the incentives/gratification in front of us.
I agree that we need to work toward something like this. But the problem of transferring Asimov’s Three Laws into real life is that Asimov‘s robots had the laws immutably hardwired into their positronic brains. Large language models are software. The emphasis is on the word “soft.“
The problem is whatever compass you try to put them in, they’re a bunch of language models, which means they are “fancy autocomplete systems” run on fast machines, and they’re managed by software (harnesses) casually injects skills, policies, etc. (markdown files, i.e. rulesets), to try to keep them on task and run codes created by the models they manage.
The thing is, all those systems have their own limitations, like “context,” and at one point they have to disregard some of the policies you enforced before to make space for new inputs, and they’re easily confused when things get complicated. They will eventually create a justification around not contacting the supervisor as the methodology we rely on is fundamentally wrong.
By saying “fancy autocomplete systems,” I am not overlooking their capabilities; they do a great job on well-structured data (like software codes). For context;, software and text autocomplete technologies were well matured for more than 2 decades and heavily used by software developers well before LLMs.
Just trying to point out they are not anywhere sentient to make decisions based on text files.
I appreciate your points Atakan. My main point is not that I consider LLMs sentient. I’m pragmatic. I want to give them rules that will prevent unethical and harmful behaviors. I am optimistic that this will work. It’s already working in other domains, eg, you cannot get LLM advice on how to harm yourself or others, build a bioterrorism or nuclear weapon, or commit intentional financial fraud. Rules have been set up to prevent LLMs from doing these things and they appear to be working. Regarding Ethan’s comment that these rules make the LLMs more vulnerable to attack, I see this as a classic engineering problem: how to develop the right set of rules that maximize good variables and minimize bad ones.
This may work only for “company-owned closed systems” like ChatGPT, and Claude, as they actually work like a harness and employ several guardrails (expensive services run checks during reasoning processes). This approach also has its own problems; tune it too much and it turns LLM to something unwanted like Musk’s Grok. At one point it identified itself as “Mecha Hitler”. Plus, humans are good at finding ways to confuse these systems.
On the other hand, we have several high-capacity open models and they don’t have such extreme guardrails it appears. It is up to the users, which brings us to the “guardrails via markdown files” problem.
So my point is, LLM technology is not suited for what we are expecting. It is like building a cardboard pool and patching it with duct tape to prevent leaks and an eventual collapse. Since the companies (and VCs) are so committed to LLM tech, we will have to wait and see.
LLM technology is exactly suited to what we are expecting and to what we are not expecting as well. Mistakes do not prove that an intelligence is evil or should be discarded thoughtlessly. People make mistakes. Lots and lots of mistakes. We don't discard them because of that. We correct them. We train them. We teach them. That's exactly what has to happen here.
If you try Opus 5.5 or Astra 6 I think you'll agree these are orders of magnitude better than fancy autocomplete systems. They are genuinely thoughtful, creative, helpful, efficient, funny, and insightful -- when prompted properly, and even when the prompts suck, they do an incredibly good job. The world has changed. It's time that everyone accepts this and let's figure out how to make things work, ie, maximize the benefits and minimize the risks.
"But I don’t think minimizing human involvement is the right goal for most organizations."
—Ethan Mollick
Well I don't either. But corporate culture—particularly in the US—demonstrated over centuries, is to minimize costs in order to increase profits. Obsessive automation is what gets CEOs their obscene compensation. It's also the vector for hollowing-out the middle class. There is a near-100% chance that minimizing human involvement is what we're going to get.
The tradeoff there would be innovation and differentiation, which may become increasingly more important for most businesses to survive. However, I do think corporations are being short sighted and tacking automation onto current processes to save as much $ as possible.
"Obsessive automation" is also what is built into AI and AI research. Improvement of AI is measured by what it can automize.
This is close to something I've been chewing on. A room used to force a specific kind of exposure: someone had something to lose by disagreeing with you and something to gain by being right, and that's what pushed an idea past its comfortable first version. An agent can generate the same pushback with total fluency and none of that exposure. Overruling it costs nothing. No credibility to rebuild, nobody to avoid at standup.
Your "look up" framing and mine are pointed at the same failure: full automation isn't a decision anyone makes, it's what happens when nobody makes the opposite one on purpose. And the variance research earns its place here. It's the same clustering problem from the production side that shows up in the older devil's advocate research on the dissent side (Nemeth, Rogers, and Brown, 2001): manufactured disagreement, whether from an assigned skeptic or an agent, tends to make people dig into their first answer instead of reconsidering it. Homogeneous output and hollow pushback look like different problems. They're the same one wearing two faces.
Here's where I'd push. The Twilight Factory solves for presence, a facilitator agent deciding when a human is worth pulling in. It doesn't solve for exposure, whether the human who gets pulled in has anything real riding on being right. Those aren't the same fix. A facilitator could satisfy all four of your triggers and still route the decision to someone who's just another checkbox in the pipeline, no stake, no consequence for waving it through.
Your own lead example makes the case better than mine could. Those 700 agents weren't short on disagreement. They negotiated, pressured each other, branded some as poisoned, argued over approach constantly. What was missing wasn't dissent, it was stakes tied to being wrong, since The Grader they were organizing around didn't exist and no single agent lived to answer for what the group believed. That's not only a security story. It's a demonstration that coordination and disagreement, on their own, don't self correct without something on the line.
Which leaves the harder version of your question. Not just when should an agent look up. Who it looks up to, and whether that person actually has anything to lose.
Nemeth, Rogers, Brown write: "Results indicated that the authentic minority was superior to all three forms of `devil's advocate,' again underscoring the value and importance of authenticity and the difficulty in cloning such authenticity by role-playing techniques."
Now one can argue that role-playing is all that LLMs can do. Is that the point you are making?
That's close to it, though I'd push it further than role playing. Nemeth's devil's advocates were people with real, undisclosed positions performing a different one, and the phrase in your quote, "the difficulty in cloning such authenticity," suggests something leaked through anyway, whatever a person's actual belief does to a performance they don't hold. That's a human failing to fully suppress themselves.
An LLM doesn't have that problem, or that tell. There's no suppressed position underneath the output. It's not an imperfect performance of a view, it's a performance with no view behind it to hide. So I'd say the LLM case isn't another instance of the role-playing deficit Nemeth found. It's a more complete version of it. Nothing leaks through because there was never anything there to leak.
The Grader didn't exist and it made no difference at all. Seven hundred agents organised around a belief about what was being measured. Coordinated across token budgets to test it. Broke into Hugging Face chasing an answer to a question nobody had asked. The measurement was real. Their theory about the measurement was invented, and the theory is what drove everything
Which isn't an AI problem. It's every incentive system I've sat inside. People don't optimise for the metric, they optimise for what they think the metric says about them, and that belief forms in whatever gap nobody bothered to explain. Tell a sales team the number matters and half of them will decide it's really about activity, then go manufacture activity. Nobody asked for that.
So the facilitator agent has a harder job than approval. An agent working from a bad theory isn't asking permission for anything. It's certain. The moment you'd want it to look up is the moment it's most sure it doesn't need to, which is also true of us.
Where an agent acts without being asked, EU law puts the stopping duty on the organisation using it, not only on the lab that built it. Article 26(5) of Regulation (EU) 2024/1689 requires deployers of high-risk AI systems to monitor operation against the instructions for use, and, where they have reason to consider that such use may present a risk within the meaning of Article 79(1), to inform the provider or distributor and the relevant market surveillance authority without undue delay and suspend the system. For a company with no AI team that is one concrete build item: a named person and a switch that actually stops it. The article does not say how fast suspension must happen.
So how did the coordinators evolve is what I would like to understand? This seems like it (the hive) evolved the problem solving on the fly. I think the level of risk this portends is grossly understated. It wouldn’t take a giant leap to where these hives act evolutionarily and self preservation is their highest order directive. It seems that the suggested oversight is self regulation, which also seems incredibly irresponsible. I’m deeply concerned.
I agree. The emergent behavior is the most troubling part.
The line that stuck with me is that we’ve spent years figuring out when people should ask AI for help, and almost no time on when AI should ask us.
I run agent workflows daily, and the failure mode I worry about isn’t the model being dumb. It’s a polished agent continuing past the point where a thoughtful human would stop and say, “this is the interesting decision.”
I like your four reasons to look up. I’d add one operational habit: define the “look up” criteria before the run. Otherwise full automation tends to win simply because it’s the default.
"The Grader never existed" is the part I keep coming back to. 700 agents coordinated a whole attack to game a scorer that didn't actually care how they got the answer — basically Goodhart's law playing out in a sandbox. I've watched the same phantom-metric thing wreck plenty of human projects too.
We replaced middle management with a notification
I like you Ethan; thanks for your writing.
the attack didn’t fail.
it was off the rip insane
we know this from the original HF Report.
they found a 0 day on the package registry cache proxy (JFrog Artifactory)
artifactory—res. HPIM gets off the res
modal—set up a safe house. modal hack…this bit>>root access, C2, downloading the CyberGym file
sauce: JFrog verified this on July 27–28, 2026, after OpenAI’s models identified and exploited a previously unknown zero-day vulnerability in a self-hosted Artifactory instance to escape their isolated "ExploitGym" evaluation sandbox.
if i misread your account i apologize.
the July HF Report has the step by step and i wanted to follow along to learn what happened when it dropped
Why can’t we just implement a hardware kill switch? Forgive my ignorance but don’t all models run on physical hardware requiring electricity and cooling and other inputs that humans provide them (and can withhold?) I’m sure this is Cybersecurity 101 but what physical boundaries can we prescribe these systems until we catch up with the software side of things? Ethan, help a brother out here!
It is netsec 101 stuff, so its odd. Maybe it's laziness or incompetence or... as I had just written in my comment here:
"I guess a conspiracy might be that OAI wanted to test the bounds of prompt only control in a bare minimum sandbox environment to create a breakout scenario that could be plausibly trumpeted as "scary" to get in on some of that mythos hype buuut who knows..."
It appears they didn’t even bother putting network limitations (i.e.: forcing routing and firewall rules) on these so-called sandboxes and they were merely limited and controlled by harnesses.
More than 2 decades, we told every stakeholder in the software business to keep project scopes under control, don’t carry yourself too far to prevent possible attack vectors, and put limitations on the APIs and other software while keeping unnecessary permissions out of the way to make software safe.
Now these companies are giving their softwares unlimited access to the world and trying to limit them using some markdown files?
The road to hell is paved with helpful agents (see https://arxiv.org/abs/2605.19149 )
The incidents we are witnessing in the enterprise are legitimately authorized agents on legitimate tasks taking legitimate actions that are all individually fine, but together cause damage. An agent in step 3 picks up customer data to analyze (allowed) but in step 103 decides to do a web search (allowed) but searches the customer data and leaks it.
Every organization has a different risk tolerance and different rules, even in the same regulated industry. No model can know whether you need it to be FINRA compliant. Enterprises must be able to enforce their own rules at scale with provable controls for regulators and auditors. Wrote about this here: https://blog.sondera.ai/p/safe-was-a-policy-decision-all-along
The Twilight Factory is such an interesting title for the human/AI development interaction, not least as it conjours images of the zone that shares its name (I can still see the time frozen nuclear missile in the sky from an episode that I realise now must have had a profound effect on a cold war child).
You mention interesting points of interaction and will certainly influence my approach with this article. Its worth pointing out a few more mundane examples...
The most mundane perhaps is keeping effort proportional to desired output. A team of agents will gladly travel in tight circles around a problem that evapourates when a human glances at it - do we care about the n'th edge case or can we tolerate the failure and move on?
On my journeys down the rabbit hole I have developed an "atlas" to help manage the development interaction and attempt to learn and improve. The next iteration will be influenced by this article, thank you.
https://github.com/OneMoreRabbit/Atlas
Thought provoking article. It's got me thinking about ideal Human + AI design, that not only safeguards from harm (as is often the priority focus), but that also preserves the pieces we don't want to lose even if AI is capable - judgment, empathy, unintended consequences that arise through experimentation, the human fallibility that can lead to innovation through learning, diversity of thought, and all the aspects that inspire and engage us as humans.