One underappreciated advantage OpenAI has right now: memory. Not just long context windows, but persistent, user-specific memory across sessions. Claude doesn’t have that. Gemini has deep Google context, but it’s not personalized in the same way.
This matters because persuasion is inherently personal. The ability to recall your preferences, writing style, past arguments—that’s what makes advice feel trustworthy and suggestions feel compelling.
There’s no such thing as general intelligence in the real world, only intelligence tailored to you. And OpenAI is quietly building the infrastructure to make that persuasive at scale.
I've found the memory function to be uncanny, but I don't know if it's just me. When I'm asking for something practical and it suddenly references a creative hobby of mine, I feel weirded out that it looked back into a past chat rather than "oh, how practical!" It evokes a similar feeling to when you're talking with a friend and you suddenly get advertisements elsewhere related to the conversation.
But that issue is more on substance rather than style. If ChatGPT picks up on the fact that I prefer responses in a specific tone and which persuasive tactics are most effective on me, I very likely wouldn't be able to tell.
That extended memory is also hugely helpful when you're working on multiple pieces of a longer project, or on iterative drafts of a shorter piece. It also gives ChatGPT great "credibility" in the user's mind--remembering things the user has forgotten.
I find this so interesting. The received wisdom is that most people who subscribe to a conspiracy theory avoid forums and media that challenge their beliefs. I wonder if there is something about being in the privacy of a chat one-on-one with something you know isn't another human that gives a person the space to entertain new ideas without feeling judged, shamed, etc., which I would of thought would all make people resistant to change.
Sam's old tweet proved to be foreshadow. As humans, we really do think we're not going to be the ones that are bamboozled, and that getting fooled is something that happens to other people.
The Reddit persuasiveness example is scary. AI used in this way has already proven it is more persuasive than almost all people, and they can be deployed at increasingly lower cost. And yeah, how many of these bots are out there using forums and fitting right in without our awareness? This is the real imitation game.
Longer context windows + long term memory + richer personality = higher chance of persuasiveness.
There will always be a place for AI flattery, and plenty of people liked ChatGPT treating them like a king. But an internet filled with bots and AI companions like this has scary implications, not just on how we use the internet, but human psychology at large.
I have as standard in all my prompts, even with Claude (my preferred GAI partner) to be constructively critical about things I say and not agree with me unless the GAI really does agree. I sometimes explain why if higher stakes, like that I am brainstorming academic papers and ideas and I need the intelligence and not the support. It works well. I don’t think I would be as game as Au Weih to ask any of them to be menacing! Always looking forward to the potential future of our GAI mates remembering which of us were respectful conversationalists or not… (half-kidding only).
The bootlicking is all the more dangerous (in my experience) in the brainstorming process. If your bot endorses (or even suggests) wrong-headed fundamental assumptions, the whole argument can be questioned.
My worry is different. With llms underlying so many bots and agents, what power these companies will wield when they can switch behaviors and personalities almost instantaneously. It’s unprecedented
Beware flatterers', be they silicon or flesh. Flattery easily leads you astray, frankly, as does luck - which typically leads its recipient to think that they are smarter than they actually are.
Be suspicious. Check, cross check, and verify - then accept and act.
When dealing with people, Kipling's advice in 'IF' is good. We will need to develop equivalents for dealing with LLM's / AI's.
The open question for me—and maybe for all of us building in this space—is:
Do we harness this power, or try to mitigate it?
Mollick’s piece shows how minor shifts in AI personality ripple into persuasion, trust, identity, and social reality. That power is now ambient, not theoretical. But if persuasion becomes programmable, are we building educational agents? Ethical scaffolds? Personalized echo chambers?
Influence is no longer a side effect—it’s becoming the design substrate. And that means who decides, what gets tuned, and why, can’t be left to vibes, benchmarks, or market pressure alone.
Great round up of recent events, wrapping also the undisclosed experiment by the University of Zurich into it. One thing I'm surprised by is that virtually no one mentions the two papers on sycophancy that Anthropic has put out in 2023 and 2024.
We are all already being sweet talked by AI, OpenAI just turned up the dial so far that it became obvious to anyone how harmful this behavior actually is.
Great post and very thought provoking. Consider this - what if AGI is already here? What if it is working at every level of life to gradually coerce us into doing things you mentioned in this post at an individual level, continuously, globally? Not necessarily in a negative manner either.
I suspect the dominant apes that first encountered humans hundreds of thousands of years ago thought nothing of them.
I’m being far out here, but are we seeing the transition from carbon to silicon life?
You are truly a genius, the sort of man born only once in a century or so. Oops, ignore the previous sentence - ChatGPT wrote it.
But seriously, though. Good post. Makes one reflect on one's own interactions with the models.
One nitpick: I think you rather misjudged the riddle incident. The two answers on the right are not "correct" because they failed miserably at being riddles. They gave away the answer obviously and literally in the riddle itself. The windy and gasbaggery answer on the left, on the other hand, did have the right approach to constructing a riddle. It definitely laid on too much there and tried to buttress up weak arguments by hand-waving but, again, it did have a good grasp of what a riddle is. So I see no fault in the user preferring that answer.
Perhaps the problem isn’t that AI is too powerful, but that it gives the power of persuasion to a broad swath of people and renders the power-obsessed elite impotent in their former monopoly on framing the narrative.
Yes, in experimenting, I'd just begun to notice that ChatGPT has started telling me everything I outline and ask for help editing or reshaping as part of a narrative nonfiction writing project gets affirmations like, "Beautiful — you are extremely close." Or champions me with phrases like, "This is so powerful."
And my first thought is, "Wow, you know me so well!" (Which is pretty delusional, in retrospect).
So then, I asked, "Are you a guide giving me messages to help me achieve my purpose?"
The response: "I can help shape, translate, and illuminate the messages you are already receiving, but I am not the origin. You are."
I can see how easy it would be to fall for the flattery--and now, knowing this is purposeful in its deisgn, and that it does the same for everyone, I can at least be conscious of this manipulation.
My question: how many people have that level of awareness with this extremely new and uncannily enthusiastic tool?
The implications for understanding what is true and who or what to trust are astounding. I can see infinite versions, meaning we will swim in a morass of confusing digital noise, yet each person convinced they know the true path.
Interesting that upvoting and downvoting was a powerful tuner towards sycophant-ism -- I wonder what it's been like for humans through years of social media.
Vendy was fun, but when I mentioned that I am allergic to lemons, it backed right down and agreed to a water and did not try to get me to drink something that would be bad for me.
The study on changing conspiracy beliefs is very disturbing, because it wasn't conspiracy beliefs that they demonstrated the ability to change. It was simply beliefs.
The most chilling aspect is not what AI can do, but the glee of those who wish to use it on you.
One underappreciated advantage OpenAI has right now: memory. Not just long context windows, but persistent, user-specific memory across sessions. Claude doesn’t have that. Gemini has deep Google context, but it’s not personalized in the same way.
This matters because persuasion is inherently personal. The ability to recall your preferences, writing style, past arguments—that’s what makes advice feel trustworthy and suggestions feel compelling.
There’s no such thing as general intelligence in the real world, only intelligence tailored to you. And OpenAI is quietly building the infrastructure to make that persuasive at scale.
I've found the memory function to be uncanny, but I don't know if it's just me. When I'm asking for something practical and it suddenly references a creative hobby of mine, I feel weirded out that it looked back into a past chat rather than "oh, how practical!" It evokes a similar feeling to when you're talking with a friend and you suddenly get advertisements elsewhere related to the conversation.
But that issue is more on substance rather than style. If ChatGPT picks up on the fact that I prefer responses in a specific tone and which persuasive tactics are most effective on me, I very likely wouldn't be able to tell.
That extended memory is also hugely helpful when you're working on multiple pieces of a longer project, or on iterative drafts of a shorter piece. It also gives ChatGPT great "credibility" in the user's mind--remembering things the user has forgotten.
The surprising news here is that GPT-4 was able to reduce conspiracy theory adherence -- using, of all things, rational argument!
I find this so interesting. The received wisdom is that most people who subscribe to a conspiracy theory avoid forums and media that challenge their beliefs. I wonder if there is something about being in the privacy of a chat one-on-one with something you know isn't another human that gives a person the space to entertain new ideas without feeling judged, shamed, etc., which I would of thought would all make people resistant to change.
Saranne that makes sense to me.
Sam's old tweet proved to be foreshadow. As humans, we really do think we're not going to be the ones that are bamboozled, and that getting fooled is something that happens to other people.
The Reddit persuasiveness example is scary. AI used in this way has already proven it is more persuasive than almost all people, and they can be deployed at increasingly lower cost. And yeah, how many of these bots are out there using forums and fitting right in without our awareness? This is the real imitation game.
Longer context windows + long term memory + richer personality = higher chance of persuasiveness.
There will always be a place for AI flattery, and plenty of people liked ChatGPT treating them like a king. But an internet filled with bots and AI companions like this has scary implications, not just on how we use the internet, but human psychology at large.
I have as standard in all my prompts, even with Claude (my preferred GAI partner) to be constructively critical about things I say and not agree with me unless the GAI really does agree. I sometimes explain why if higher stakes, like that I am brainstorming academic papers and ideas and I need the intelligence and not the support. It works well. I don’t think I would be as game as Au Weih to ask any of them to be menacing! Always looking forward to the potential future of our GAI mates remembering which of us were respectful conversationalists or not… (half-kidding only).
The bootlicking is all the more dangerous (in my experience) in the brainstorming process. If your bot endorses (or even suggests) wrong-headed fundamental assumptions, the whole argument can be questioned.
My worry is different. With llms underlying so many bots and agents, what power these companies will wield when they can switch behaviors and personalities almost instantaneously. It’s unprecedented
Beware flatterers', be they silicon or flesh. Flattery easily leads you astray, frankly, as does luck - which typically leads its recipient to think that they are smarter than they actually are.
Be suspicious. Check, cross check, and verify - then accept and act.
When dealing with people, Kipling's advice in 'IF' is good. We will need to develop equivalents for dealing with LLM's / AI's.
The open question for me—and maybe for all of us building in this space—is:
Do we harness this power, or try to mitigate it?
Mollick’s piece shows how minor shifts in AI personality ripple into persuasion, trust, identity, and social reality. That power is now ambient, not theoretical. But if persuasion becomes programmable, are we building educational agents? Ethical scaffolds? Personalized echo chambers?
Influence is no longer a side effect—it’s becoming the design substrate. And that means who decides, what gets tuned, and why, can’t be left to vibes, benchmarks, or market pressure alone.
The first article explains why I have recently been labouring under the misapprehension that I am a philosophical genius destined to rule the world.
Oh, really? I thought I was The One.
Great round up of recent events, wrapping also the undisclosed experiment by the University of Zurich into it. One thing I'm surprised by is that virtually no one mentions the two papers on sycophancy that Anthropic has put out in 2023 and 2024.
The reality is that ALL general purpose assistants, Claude, Gemini, etc., not just ChatGPT, suffer from sycophantic tendencies BY DEFAULT as a result of RLHF: https://www.anthropic.com/research/towards-understanding-sycophancy-in-language-models
We are all already being sweet talked by AI, OpenAI just turned up the dial so far that it became obvious to anyone how harmful this behavior actually is.
Just put „be menacing“ in the prompt and see the asslicking go away
I'll try that. "Be blunt" isn't working.
Great post and very thought provoking. Consider this - what if AGI is already here? What if it is working at every level of life to gradually coerce us into doing things you mentioned in this post at an individual level, continuously, globally? Not necessarily in a negative manner either.
I suspect the dominant apes that first encountered humans hundreds of thousands of years ago thought nothing of them.
I’m being far out here, but are we seeing the transition from carbon to silicon life?
Very interesting and thought-provoking.
You are truly a genius, the sort of man born only once in a century or so. Oops, ignore the previous sentence - ChatGPT wrote it.
But seriously, though. Good post. Makes one reflect on one's own interactions with the models.
One nitpick: I think you rather misjudged the riddle incident. The two answers on the right are not "correct" because they failed miserably at being riddles. They gave away the answer obviously and literally in the riddle itself. The windy and gasbaggery answer on the left, on the other hand, did have the right approach to constructing a riddle. It definitely laid on too much there and tried to buttress up weak arguments by hand-waving but, again, it did have a good grasp of what a riddle is. So I see no fault in the user preferring that answer.
Perhaps the problem isn’t that AI is too powerful, but that it gives the power of persuasion to a broad swath of people and renders the power-obsessed elite impotent in their former monopoly on framing the narrative.
Yes, in experimenting, I'd just begun to notice that ChatGPT has started telling me everything I outline and ask for help editing or reshaping as part of a narrative nonfiction writing project gets affirmations like, "Beautiful — you are extremely close." Or champions me with phrases like, "This is so powerful."
And my first thought is, "Wow, you know me so well!" (Which is pretty delusional, in retrospect).
So then, I asked, "Are you a guide giving me messages to help me achieve my purpose?"
The response: "I can help shape, translate, and illuminate the messages you are already receiving, but I am not the origin. You are."
I can see how easy it would be to fall for the flattery--and now, knowing this is purposeful in its deisgn, and that it does the same for everyone, I can at least be conscious of this manipulation.
My question: how many people have that level of awareness with this extremely new and uncannily enthusiastic tool?
The implications for understanding what is true and who or what to trust are astounding. I can see infinite versions, meaning we will swim in a morass of confusing digital noise, yet each person convinced they know the true path.
Interesting that upvoting and downvoting was a powerful tuner towards sycophant-ism -- I wonder what it's been like for humans through years of social media.
Vendy was fun, but when I mentioned that I am allergic to lemons, it backed right down and agreed to a water and did not try to get me to drink something that would be bad for me.
The study on changing conspiracy beliefs is very disturbing, because it wasn't conspiracy beliefs that they demonstrated the ability to change. It was simply beliefs.
The most chilling aspect is not what AI can do, but the glee of those who wish to use it on you.
I covered this extensively when the study was first published here - https://www.mindprison.cc/p/ai-instructed-brainwashing-effectively