Very interesting summary of some presently available AI capabilities people may be unaware of-but your overall thesis that we shouldnt worry about “future” capabilities ignores the fact that according to the leading experts in AI that “future” and the possibility of catastrophic harm is no longer hypothetical or a worry for down the road - its arrived, its here - so its essential to immediately try to deal with that reality
Except that is very clearly not my thesis, I state several times that it is reasonable to worry about the risks of continued AI development, including quite directly: "it seems reasonable for governments and AI labs to be arguing about how to manage the speed of development to mitigate these risks."
My thesis is that just worrying about existential risk alone is not enough, we also need to decide how we are using the AI we have today in ways that help humans thrive while trying to avoid dangerous or disempowering uses. I think we can do that, but it is not automatic, and it is an important focus that is independent of existential risk concerns.
The near term risk is from people thinking it is more capable than it is - for example, the US government nearly started a war with China over a false, AI generated report. It can do stuff but you still can’t trust it if it matters.
Public predictions about human extinction caused by AI since the 50s were audited in a recent paper. 79% are unfalsifiable (undefined nor measurable), 8% are wrong and just 13% remain open. "Predictions from commercial labs and company leaders score worst on average in our sample".
Ethan, your perspective was very refreshing after listening to doomsday panic all week. Focusing on the "capability overhang" keeps us in the equation and challenged me to keep learning.
Here's the lesson I took away: you can't outsource taste. Cultivating instinct requires actual work; you don't get there by letting a model do the thinking for you.
Deep and wide knowledge take a long time to develop, but having the judgment to direct AI tools effectively makes those hours worth every second. Moreover, I think staying curious about the process is the only true way to build something novel.
I think the entire direction of the internet - how news is disseminated, how entertainment is disseminated - underlies the importance of curation. We don’t need gatekeepers necessarily but we need indicators of what is good, what has authority, to distinguish it from low quality work.
How can taste and judgment be developed when the opportunities to develop those are going away? The "output production " part of work, now being taken over by AI, provided the reps for newbies to develop their skills.
I also believe the AI we have now has deep unmined depths and uses , and my point of view is that we should take time tomlearn how to use and more importantly manage what AI can do now before rushing ahead and making it bigger, faster, stronger.
Especially when it comes at such a huge ecological cost at a time where clean water availability is shrinking in shocking leaps and bounds.
I do wonder if these datacenter planners have asked ai how to make it environmentally friendly and cost effective, or if they just decided that reusable cooling gel is too expensive and haven't even bothered looking at alternatives 🤷
"GPT-6 Astra and Fable 5.1 are already enough to change how large parts of the economy work"
I think if AI frontier work stopped right not (I'm sure it won't, and I'm not sure I think it should), we would be able profitably to mine present AI's value for decades. It's so easy to end a day with AI with more items on your to-do list than you started with, and it can soak up every minute you give it.
Azeem Azhar writes today that successful alignment (with human values) by AI instances does not imply that they will remain aligned when groups of them work together. That raises a second-order threat which even a pause of frontier work would not assure against.
I found this post very helpful in thinking about the recent calls for oversight in AI development and what that might look like. I began thinking about my experience in oversight of an evolving technology which had both potential great risks and advantages when I served on the NIH’s Recombinant DNA Advisory several decades ago. This oversight activity used members who had varying amounts of deep and wide knowledge, different tastes(viewpoints and beliefs) and varying degree of agency. However we decide to approach the oversight of AI these will be critical issues in choosing those that execute that oversight.
The agency piece is critical- in academic fields next to nobody is using these tools at all, and if they are, only at a very basic level. I have yet to meet anyone in my field who didn’t give me a blank stare when I asked if they were using a harness.
I try to build things that are unrelated to my actual work just to learn from my failures and get better at prompting- because with each new release, all of a sudden an area of my field that AI didn’t really get is suddenly extremely proficient
Ethan, I want to reframe this from the direction the technology is actually heading, which is personal AI agents rather than AI as a tool you consult. The four advantages read as a defense of the human in the loop, and it is a good one, but agents change the structure of the argument rather than merely challenging it.
Deep knowledge and wide knowledge are both forms of accumulated context, and context is exactly what persistent agents are getting good at holding. An agent that has worked alongside someone for a year will have both: the depth of that person's domain and the width of everything it has touched across tasks. Taste is the interesting one, because taste looks like it survives, except that selection among abundant generated options is itself a learnable preference function. Every time you pick option three and reject the rest, you are training a taste model, whether you intend to or not.
That leaves agency, and I think agency is where your framework points to something genuinely durable. Agency is the only one of the four that cannot be delegated without ceasing to be yours. The moment an agent exercises taste on your behalf, the taste is no longer quite yours. The four advantages are really one advantage with three supporting skills: the willingness to be the person who decides.
So the practical question extends past which skills to cultivate, though I agree with your list. It becomes which decisions you refuse to hand over even when the agent could make them well. That is a smaller and harder list than most people expect.
The four advantages hold up, and your own examples argue for a fifth that did not make the list.
The Blender trailer took 45 minutes and involved thousands of small decisions: framing, timing, which take to keep. The library reconstruction inferred roughly 5,000 books, some of them in areas never actually seen on the videos. In both cases the output is excellent and the decision trail is gone.
For personal projects that costs nothing. Inside an organisation it is the whole adoption problem. Taste tells you which output to keep. It does not tell anyone afterwards which choices were yours, and that second question is the one that shows up in a procurement review, an audit, or a dispute.
So I would add being able to account for the work to the list, and I would argue it is the advantage most likely to be scarce, because nothing in the current tooling produces it as a byproduct.
Do you think the need for deep knowledge + wide knowledge to get an impact from AI will be reduced over the coming years, or likely it will get easier but that it will always be an advantage to have this.
I'm looking forward to the day when an AI can generate an entire movie from a novel. There are lots of my favourite books that will never be made into a decent movie or the movies that exist are bad. At the present rate of improvement, hopefully it won't be too long to wait.
Bad is a very ambiguous word. Maybe it would be nice to sit down with that feeling and ask what exactly makes it bad from your perspective. And maybe a tool fed with all the information out there could figure out how to satisfy your particular sense of what makes a good adaptation.
Your jagged edge framing might be useful for discussing taste as well.
A couple of years ago I was helping a startup get into the AI consulting space for asset and wealth managers. My pitch to prospective clients was that we had the technology, talent and taste to be their preferred partner. And of the three, taste should be considered our real differentiator. All of senior leadership had decades of experience in their respective niches, and well-honed senses of what good looks like.
The emphasis on taste had the virtue of being both true and comforting. Prospects appreciated the idea there would still be competent humans-in-the-loop, and we liked feeling that our experience—nuance and secret sauce not captured by the models—was still worth something.
A lot of people feel this way. I read my fair share, and a common theme is that “taste” is our collective safe harbor. I used to nod along, but had an uneasy sense. If AI can learn what we know, why not what we like? The words needed to finish the thought escaped me until recently.
Taste is pattern matching.
The patterns are what we like or don’t like. What’s true for me might not be true for you, but that’s beside the point. As long as one can articulate their preferences, and leave enough breadcrumbs along the way, AI can learn to be a good enough arbiter most of the time. If AI is good at anything it is pattern matching.
Hi Ethan. Thanks for sharing this out. I have been attempting to build some similar models over the summer to share out with teachers but was never getting things quite right. Having access to your two models was SUPER helpful. I tried it first with ChatGPT (Astra and then Sol when it timed out) but hated the results. THen I used the Educational Version of Claude and it was much superior. Here is a link the interactive tinkering studio that I built out. This is version one with no input from me other than the prompt (below) as well as some pictures of the room. It got the space about 80% correct and picked up most of the key tools that I wanted to have highlighted. That was pretty amazing. I ened up putting it in a Google Site for easy sharing: https://sites.google.com/catlin.edu/interactivetinkingstudio/home
Here is the prompt that I used:
I want to create an intereactive model of my tinkering studio. I want it to be based on the work that Ethan Molick describes in a recent posting that is linked here. https://www.oneusefulthing.org/p/the-overhang?utm_source=substack&utm_medium=email#media-28d6f8bb-81e2-42cd-83a9-90be6f9ab894. He also has provided the open source code for his Zork 3D model here emollick/zork-underground-empire and his library project here emollick/eco-library. I have included a video of my tinkering studio and want to create a interactive model of the space and then ways to learn about how to use each of the tools. So just like Zork you can move around the space and interact with it and just like the library you can move around the space and learn about all the books. So you need to provide videos or links to each of the kinds of tools that I have so people can learn about them and learn about how to use them well. This could be for parents or for students or for teachers to access. You can go to Blender and create what you need and use any of the materials that Ethan has created to help you more efficientely build an itneractive space.
I think what we're witnessing now is an exponential improvement in AI's ability to engage with 'deep' and 'wide' human knowledge at a comparable level of sophistication and complexity.
For instance- as a law student, a lot of my academic and co-curricular work involves extensive case-law research, and drafting. While AI remains a step behind most good writers, I can confidently state that it is now a few steps ahead of most decent researchers.
Earlier, and I mean just a year or two ago, research was one of the most tentative and untrustworthy points along the frontier (think- hallucinations and lawyers citing made up cases before a court).
However, recent frontier models exhibit an impressive ability to problem solve accurately, and to conduct research using sophisticated analogical frameworks. These models, in my experience, manage to substantially solve complex legal research propositions in just a single "deep search", with almost no hallucinations. They manage to match many days, and even weeks, worth of human research effort in 30-60 minutes.
Even when its research isn't comprehensive, or when it misunderstands the true meaning of a case-law, what I find is that it it generates enough research leads to shave off at least a few days worth of blind goose-chasing, which has traditionally been accepted as the natural process of legal research.
Its effectiveness at research is only going to improve as Agent Harnesses are improving. For instance, while its earlier research tasks were executed largely by relying on training data and RAG, Claude can now access private databases such as Westlaw and OUP through the chrome connector harness. This, in particular, has been a rather exciting development in the research potential of frontier models.
I find legal research to be an especially potent heuristic for keeping track of improvements on the frontier, since such tasks require incredibly complicated reasoning. It requires, firstly, a cogent understanding of the research proposition itself. For instance, what level of abstraction the proposition sits at, or in other words, its degree of specificity to a set of real-world facts. Once a researcher has understood the proposition itself, the process of research is essentially a process of looking for apposite authority; i.e., reasoning analogically across varying degrees of abstraction.
For instance, my research proposition may be incredibly abstract ("can a person profit from their own wrong?"), and in my research I am unlikely to find any cases that explicitly articulate such a principle at a comparable degree of abstraction. Instead, I am likely to find cases where courts have refused to allow persons to inherit property from their father or mother, when such heir has murdered their parent to expedite this process of inheritance.
To the untrained researcher such cases stand on an entirely distinct footing from my proposition, but the truth of legal research is that one rarely turns up authorities that are perched at a comparable level of abstraction. Therefore, the task of a researcher is most often to identify a common determining principle that reconciles the research output with the research proposition, although they stand at different levels of abstraction.
In our example for instance, I would argue that the courts refused to allow a heir guilty of patricide to inherit his fathers property, since one cannot profit from one's own wrong. Of course, such an assertion requires independent reasoning and justification, but this is only to illustrate the essentially analogical nature of legal research.
To tie this back to the original point, earlier frontier models (just a year ago) were completely incapable of dealing with research propositions in this manner, no matter how good your prompt was. They were simply unable to reason analogically; they rejected cases as irrelevant, when in fact they were simply perched at a different, though comparable, level of abstraction. They were also incapable of synthesising or reducing seemingly irrelevant research findings to the requested level of abstraction.
This is no longer the case. Claude Opus and Fable models in particular are now exceptionally competent at executing such tasks. Having witnessed such an exponential change within just a year is both encouraging and daunting. I am now faced with the challenge of thinking through other means of 'deepening' my ability to research, since raw effort and 'blind goose-chasing' are now simply unproductive uses of time.
This leaves me at a crossroads- whatever little understanding and intuition of research I have now was developed precisely through a few moments of inadvertent discovery nested within hundreds of hours offruitless effort. Now that outsourcing time-consuming initial stages of research to AI is simply a more productive use of my time, I must create new challenges that require sustained cognitive effort. That, to me, is the shifting frontier of human intelligence in the age of AI.
I had a genuine WTF moment when I played Zork 3D.
Very interesting summary of some presently available AI capabilities people may be unaware of-but your overall thesis that we shouldnt worry about “future” capabilities ignores the fact that according to the leading experts in AI that “future” and the possibility of catastrophic harm is no longer hypothetical or a worry for down the road - its arrived, its here - so its essential to immediately try to deal with that reality
Except that is very clearly not my thesis, I state several times that it is reasonable to worry about the risks of continued AI development, including quite directly: "it seems reasonable for governments and AI labs to be arguing about how to manage the speed of development to mitigate these risks."
My thesis is that just worrying about existential risk alone is not enough, we also need to decide how we are using the AI we have today in ways that help humans thrive while trying to avoid dangerous or disempowering uses. I think we can do that, but it is not automatic, and it is an important focus that is independent of existential risk concerns.
The near term risk is from people thinking it is more capable than it is - for example, the US government nearly started a war with China over a false, AI generated report. It can do stuff but you still can’t trust it if it matters.
Public predictions about human extinction caused by AI since the 50s were audited in a recent paper. 79% are unfalsifiable (undefined nor measurable), 8% are wrong and just 13% remain open. "Predictions from commercial labs and company leaders score worst on average in our sample".
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7456700
Do we really need to believe in catastrophic predictions that are there mostly to attract our attention with fear?
Ethan, your perspective was very refreshing after listening to doomsday panic all week. Focusing on the "capability overhang" keeps us in the equation and challenged me to keep learning.
Here's the lesson I took away: you can't outsource taste. Cultivating instinct requires actual work; you don't get there by letting a model do the thinking for you.
Deep and wide knowledge take a long time to develop, but having the judgment to direct AI tools effectively makes those hours worth every second. Moreover, I think staying curious about the process is the only true way to build something novel.
I think the entire direction of the internet - how news is disseminated, how entertainment is disseminated - underlies the importance of curation. We don’t need gatekeepers necessarily but we need indicators of what is good, what has authority, to distinguish it from low quality work.
How can taste and judgment be developed when the opportunities to develop those are going away? The "output production " part of work, now being taken over by AI, provided the reps for newbies to develop their skills.
I also believe the AI we have now has deep unmined depths and uses , and my point of view is that we should take time tomlearn how to use and more importantly manage what AI can do now before rushing ahead and making it bigger, faster, stronger.
Especially when it comes at such a huge ecological cost at a time where clean water availability is shrinking in shocking leaps and bounds.
I do wonder if these datacenter planners have asked ai how to make it environmentally friendly and cost effective, or if they just decided that reusable cooling gel is too expensive and haven't even bothered looking at alternatives 🤷
"GPT-6 Astra and Fable 5.1 are already enough to change how large parts of the economy work"
I think if AI frontier work stopped right not (I'm sure it won't, and I'm not sure I think it should), we would be able profitably to mine present AI's value for decades. It's so easy to end a day with AI with more items on your to-do list than you started with, and it can soak up every minute you give it.
Azeem Azhar writes today that successful alignment (with human values) by AI instances does not imply that they will remain aligned when groups of them work together. That raises a second-order threat which even a pause of frontier work would not assure against.
Unlimited output has made knowing what you want a surprisingly technical skill.
I found this post very helpful in thinking about the recent calls for oversight in AI development and what that might look like. I began thinking about my experience in oversight of an evolving technology which had both potential great risks and advantages when I served on the NIH’s Recombinant DNA Advisory several decades ago. This oversight activity used members who had varying amounts of deep and wide knowledge, different tastes(viewpoints and beliefs) and varying degree of agency. However we decide to approach the oversight of AI these will be critical issues in choosing those that execute that oversight.
The agency piece is critical- in academic fields next to nobody is using these tools at all, and if they are, only at a very basic level. I have yet to meet anyone in my field who didn’t give me a blank stare when I asked if they were using a harness.
I try to build things that are unrelated to my actual work just to learn from my failures and get better at prompting- because with each new release, all of a sudden an area of my field that AI didn’t really get is suddenly extremely proficient
I have often considered using ai in my research but right now, I don't know enough about using AI for that to be a wise idea for me.
I suppose it is the difference between someone learning how to use AI to improve what they do and AI replacing what that someone can currently do.
Ethan, I want to reframe this from the direction the technology is actually heading, which is personal AI agents rather than AI as a tool you consult. The four advantages read as a defense of the human in the loop, and it is a good one, but agents change the structure of the argument rather than merely challenging it.
Deep knowledge and wide knowledge are both forms of accumulated context, and context is exactly what persistent agents are getting good at holding. An agent that has worked alongside someone for a year will have both: the depth of that person's domain and the width of everything it has touched across tasks. Taste is the interesting one, because taste looks like it survives, except that selection among abundant generated options is itself a learnable preference function. Every time you pick option three and reject the rest, you are training a taste model, whether you intend to or not.
That leaves agency, and I think agency is where your framework points to something genuinely durable. Agency is the only one of the four that cannot be delegated without ceasing to be yours. The moment an agent exercises taste on your behalf, the taste is no longer quite yours. The four advantages are really one advantage with three supporting skills: the willingness to be the person who decides.
So the practical question extends past which skills to cultivate, though I agree with your list. It becomes which decisions you refuse to hand over even when the agent could make them well. That is a smaller and harder list than most people expect.
The four advantages hold up, and your own examples argue for a fifth that did not make the list.
The Blender trailer took 45 minutes and involved thousands of small decisions: framing, timing, which take to keep. The library reconstruction inferred roughly 5,000 books, some of them in areas never actually seen on the videos. In both cases the output is excellent and the decision trail is gone.
For personal projects that costs nothing. Inside an organisation it is the whole adoption problem. Taste tells you which output to keep. It does not tell anyone afterwards which choices were yours, and that second question is the one that shows up in a procurement review, an audit, or a dispute.
So I would add being able to account for the work to the list, and I would argue it is the advantage most likely to be scarce, because nothing in the current tooling produces it as a byproduct.
brilliant post Ethan.
Do you think the need for deep knowledge + wide knowledge to get an impact from AI will be reduced over the coming years, or likely it will get easier but that it will always be an advantage to have this.
I'm looking forward to the day when an AI can generate an entire movie from a novel. There are lots of my favourite books that will never be made into a decent movie or the movies that exist are bad. At the present rate of improvement, hopefully it won't be too long to wait.
Bad is a very ambiguous word. Maybe it would be nice to sit down with that feeling and ask what exactly makes it bad from your perspective. And maybe a tool fed with all the information out there could figure out how to satisfy your particular sense of what makes a good adaptation.
Your jagged edge framing might be useful for discussing taste as well.
A couple of years ago I was helping a startup get into the AI consulting space for asset and wealth managers. My pitch to prospective clients was that we had the technology, talent and taste to be their preferred partner. And of the three, taste should be considered our real differentiator. All of senior leadership had decades of experience in their respective niches, and well-honed senses of what good looks like.
The emphasis on taste had the virtue of being both true and comforting. Prospects appreciated the idea there would still be competent humans-in-the-loop, and we liked feeling that our experience—nuance and secret sauce not captured by the models—was still worth something.
A lot of people feel this way. I read my fair share, and a common theme is that “taste” is our collective safe harbor. I used to nod along, but had an uneasy sense. If AI can learn what we know, why not what we like? The words needed to finish the thought escaped me until recently.
Taste is pattern matching.
The patterns are what we like or don’t like. What’s true for me might not be true for you, but that’s beside the point. As long as one can articulate their preferences, and leave enough breadcrumbs along the way, AI can learn to be a good enough arbiter most of the time. If AI is good at anything it is pattern matching.
Hi Ethan. Thanks for sharing this out. I have been attempting to build some similar models over the summer to share out with teachers but was never getting things quite right. Having access to your two models was SUPER helpful. I tried it first with ChatGPT (Astra and then Sol when it timed out) but hated the results. THen I used the Educational Version of Claude and it was much superior. Here is a link the interactive tinkering studio that I built out. This is version one with no input from me other than the prompt (below) as well as some pictures of the room. It got the space about 80% correct and picked up most of the key tools that I wanted to have highlighted. That was pretty amazing. I ened up putting it in a Google Site for easy sharing: https://sites.google.com/catlin.edu/interactivetinkingstudio/home
Here is the prompt that I used:
I want to create an intereactive model of my tinkering studio. I want it to be based on the work that Ethan Molick describes in a recent posting that is linked here. https://www.oneusefulthing.org/p/the-overhang?utm_source=substack&utm_medium=email#media-28d6f8bb-81e2-42cd-83a9-90be6f9ab894. He also has provided the open source code for his Zork 3D model here emollick/zork-underground-empire and his library project here emollick/eco-library. I have included a video of my tinkering studio and want to create a interactive model of the space and then ways to learn about how to use each of the tools. So just like Zork you can move around the space and interact with it and just like the library you can move around the space and learn about all the books. So you need to provide videos or links to each of the kinds of tools that I have so people can learn about them and learn about how to use them well. This could be for parents or for students or for teachers to access. You can go to Blender and create what you need and use any of the materials that Ethan has created to help you more efficientely build an itneractive space.
I think what we're witnessing now is an exponential improvement in AI's ability to engage with 'deep' and 'wide' human knowledge at a comparable level of sophistication and complexity.
For instance- as a law student, a lot of my academic and co-curricular work involves extensive case-law research, and drafting. While AI remains a step behind most good writers, I can confidently state that it is now a few steps ahead of most decent researchers.
Earlier, and I mean just a year or two ago, research was one of the most tentative and untrustworthy points along the frontier (think- hallucinations and lawyers citing made up cases before a court).
However, recent frontier models exhibit an impressive ability to problem solve accurately, and to conduct research using sophisticated analogical frameworks. These models, in my experience, manage to substantially solve complex legal research propositions in just a single "deep search", with almost no hallucinations. They manage to match many days, and even weeks, worth of human research effort in 30-60 minutes.
Even when its research isn't comprehensive, or when it misunderstands the true meaning of a case-law, what I find is that it it generates enough research leads to shave off at least a few days worth of blind goose-chasing, which has traditionally been accepted as the natural process of legal research.
Its effectiveness at research is only going to improve as Agent Harnesses are improving. For instance, while its earlier research tasks were executed largely by relying on training data and RAG, Claude can now access private databases such as Westlaw and OUP through the chrome connector harness. This, in particular, has been a rather exciting development in the research potential of frontier models.
I find legal research to be an especially potent heuristic for keeping track of improvements on the frontier, since such tasks require incredibly complicated reasoning. It requires, firstly, a cogent understanding of the research proposition itself. For instance, what level of abstraction the proposition sits at, or in other words, its degree of specificity to a set of real-world facts. Once a researcher has understood the proposition itself, the process of research is essentially a process of looking for apposite authority; i.e., reasoning analogically across varying degrees of abstraction.
For instance, my research proposition may be incredibly abstract ("can a person profit from their own wrong?"), and in my research I am unlikely to find any cases that explicitly articulate such a principle at a comparable degree of abstraction. Instead, I am likely to find cases where courts have refused to allow persons to inherit property from their father or mother, when such heir has murdered their parent to expedite this process of inheritance.
To the untrained researcher such cases stand on an entirely distinct footing from my proposition, but the truth of legal research is that one rarely turns up authorities that are perched at a comparable level of abstraction. Therefore, the task of a researcher is most often to identify a common determining principle that reconciles the research output with the research proposition, although they stand at different levels of abstraction.
In our example for instance, I would argue that the courts refused to allow a heir guilty of patricide to inherit his fathers property, since one cannot profit from one's own wrong. Of course, such an assertion requires independent reasoning and justification, but this is only to illustrate the essentially analogical nature of legal research.
To tie this back to the original point, earlier frontier models (just a year ago) were completely incapable of dealing with research propositions in this manner, no matter how good your prompt was. They were simply unable to reason analogically; they rejected cases as irrelevant, when in fact they were simply perched at a different, though comparable, level of abstraction. They were also incapable of synthesising or reducing seemingly irrelevant research findings to the requested level of abstraction.
This is no longer the case. Claude Opus and Fable models in particular are now exceptionally competent at executing such tasks. Having witnessed such an exponential change within just a year is both encouraging and daunting. I am now faced with the challenge of thinking through other means of 'deepening' my ability to research, since raw effort and 'blind goose-chasing' are now simply unproductive uses of time.
This leaves me at a crossroads- whatever little understanding and intuition of research I have now was developed precisely through a few moments of inadvertent discovery nested within hundreds of hours offruitless effort. Now that outsourcing time-consuming initial stages of research to AI is simply a more productive use of my time, I must create new challenges that require sustained cognitive effort. That, to me, is the shifting frontier of human intelligence in the age of AI.