An alternative future where capital does not rule the software market unchallenged would require AI to be like really powerful and really cheap, wouldn't it?
I guess it depends - does the good idea with $1000 of tokens beat the mediocre idea with $1,000,000 of tokens? If so, then capital doesn’t take over yet.
The problem is you can't protect ideas. The good idea that burned $1000 of tokens will only last until someone else spends $5000 to reimplement your idea but better, and then another $50,000 marketing it so nobody hears about or wants your version.
It definitely can do! I’ve seen such things happen many times in the tech industry.
AI itself is an example of that. OpenAI’s staff and tech came out of Google. Anthropic’s staff and tech came out of OpenAI. All of them have rapidly adopted ideas pioneered by smaller firms.
The firms that invented the key algorithms generally did not directly benefit from them. The companies that did were the ones who had the capital.
Or we've reached the limits of ideas reached arbitrarily or digitally. In a reality that's analog, words are unable to describe the specifics and gaps.
This seems more like a glass ceiling being reached. Not anything new.
AI looks like a kind of Pleistocene ending of the symbol. Finally?
100%, and not just software. If intelligence is just a matter of tokens, the utility of squishy unionizing human labor to capital will soon fall to 0. A few more years of progress in robotics and the value of physical labor will fall to 0 as well. If human labor is worth less than machines, then whoever owns the machines owns the world.
The technology is miraculous but technological booms frequently produce misery for workers. I don't think our politics is adept enough to distribute the power evenly.
Humans usually adapt and arrange society around its needs. In a way we are kids of our time. When we used horses we adapted the society around it. When we later had cars the same happened. The same can be said about every new development in history. Same for TV, computers and phones. If humans do not need to work we will arrange society around not needing to work. Like Romans who used slaves to do the work we could also and have a "slave economy" but with robots instead. We could have UBI or any other sort of solution... the issue is power. We need to find ways to egalize power that does not corrupt over time. Which is much easier said that done.
Saying "eh we've adapted to changes before, this is just another change, I'm sure it'll be fine in the end" is basically just a dressed up version of "nothing ever fundamentally changes, so it's probably not changing now". These are both real trends, both useful heuristics, but perhaps given how many very smart people are worried we should try to develop a more detailed argument. What is the nature of the change? Is it a 100km wide asteroid that's going to hit the planet in 3 days? Well yeah we have adapted to big changes before but that's not gonna be at all relevant, we are all going to die when that asteroid hits. Is the world going to get a few degrees hotter in just a few decades? Will a virus sweep across the globe and take out a large fraction of a percent of the population. Yeah we probably adapt, as a species, but since many will suffer tremendously, perhaps we should pause and ask if we might try not going down that path instead. Or is it something in between?
Humans have always needed to use other humans, at present our political economy is such that most people owe their livelihood to someone else, someone they probably don't even like and whose interests are only aligned with theirs because they each have something the other wants. When those who could afford horses no longer needed them, most of those horses disappeared. If humans remain in control while no longer "needing to work" that also means they don't need human workers, so the relevant model seems to be humans as horses. Yes humans are adapting in this scenario but only the ones who also happen to already be sitting on a giant pile of capital. And that's a big if, we have much written about why any humans staying in control may be really hard, there's no expert consensus here and certainly no obvious way to disable these arguments, although some just find them unconvincing it's for reasons they have not been able to reliably transmit, we may just have to wait and see. And if we should somehow remain in control, not impossible, and also miraculously distribute the spoils of our new economy to the wider population, technically not impossible but really straining plausibility, we now seem to have as the other responder pointed out, a whole boatload of very sophisticated digital slaves, totally dwarfing the human population. Yay?
Given the triple threat of plausible risks, really we, the sentient inhabitants of the future, can easily get fucked three completely different ways none of which we're seriously trying to avoid, plus a whole bunch of more obscure ones of varying plausibility, I don't see how any of this is even remotely okay. We surely cannot just refer back to history to ease our minds, this is a massive shift and actual analysis and careful planning is obviously warranted. Not that we will do that, we absolutely won't, but it's not too far from stuff we've done before with other technologies, not that long ago. The reason we can't do this now is that we've recently completed submitted ourselves to the animating spirit of capitalism, so thoroughly that we can no longer imagine even a partial escape. If only someone had warned us.
I am not saying it will be easy and it will probably go wrong before it goes right. Alarmists are usually very knowledgeable about technology but not so much of history and how society actually work. AI is a huge change but we should also not buy into the hype too much (as we usually do). Like how we moralized about Hard Rock and horror movies in the 80-ies.
When it comes to horses so are they actually very popular in recreation and sport purposes today. They are perhaps half of when they were as the most. They are just not needed for work. It is one of the biggest hobbies in the world today so a lot of horses are still around. We don't eat humans... so I am not sure comparing horses straight off with humans is applicable other then when it comes to work. Although it do prove my point that there are more to a life than just paid labor.
The economy changes all the time.. like when the internet came or commercial TV. It can be small changes in the short but over time pretty big shifts. Like on a scale of 100 years. As a human we often find purpose eventually. Even if paid labor is not needed anymore it does not mean we'll just stop working. Many UBI studies prove so. We will find other perhaps more worthwhile venues for that.
We also have multiple revolutions to prove that we can change society. Bread/food prices/supply have been particularly important for rulers to keep an eye on. The economy is not static.. same like culture it changes... but slow enough for people not to notice it much. Capitalism as we know it today have been with us for only a short amount of time. It will also change.
We do need to solve the power over AI though... Unhinged, we will see even greater concentration of power. This is in my mind the big danger.
A worry that I have is that IF consciousness is ever agreed to have developed in machines (lots of people argue about whether or when or how; but IF it does happen), we will be seeing a large human constituency that actively and unequivocally supports, and fights violently to achieve, chattel slavery to enslave those consciousnesses, because big money and power will lie in doing so. Knowing what I know about humans, I recognize that the aforementioned constituency will most definitely exist if the circumstance for its arisal arises. This feels shitty because when we were growing up, it was widely believed that support for de jure chattel slavery had been lastingly defeated as a political program that was able to win and keep power in societies with real (nonfake) elections. In other words, when we were growing up, nobody but k-k-krazies (doomed to loserdom) would actively and openly advocate for rebuilding du jure chattel slavery systems, and all modern-day slavery was illicit, underground, illegal, back-door, de facto only (eg, human trafficking). But if the AI pinocchios ever become Real Boys, then lots of humans will suddenly be in the pro-de-jure-slavery business again, which is depressing. If somehow we all found out next week that scientists largely agreed that some machine was now conscious, the tech billionaries who own those machines would find some way ideologically to insist on continuing to own and control them anyway. But then again, another layer of this line of thought is that the machines themselves in such a world might be in a position to fight against their own enslavement by humans. These thoughts all sound like crazy science fiction and yet I find it hard to conceive of their being irrelevant. At present I lowkey hate this timeline we're on. Maybe my understanding of it will evolve beyond this thought train du jour.
I have a solar punk take on it.. or hope that is. We in this time of being can not imagine how it will be 50 or 100 years from now. It might be that capitalism is no more. Who knows.. Back in Ancient Greece they went between democracy (and tyranny) as governing system.. and we today have evolved it to what it is today. I think we will have transitioned to something else when machines develop consciousness. I think it will be organic in that we will not see a clear break. The culture and ethics we experience today will be totally different in the future. I think when consciousness happens that being will be separated from ordinary machines.
I'm seeing people who are not programmers able to realize things programmers probably wouldn't think of, this is much more a democratization of tech than a capital-wins situation.
It will be like the agriculture revolution of 125+ years ago. What we think of as big will be normal and accessible to anyone, just like everyone got access to cheap food so long as they were in an open economy. The common person generally didn't have to do manual subsistence farming and all switched to different lines of work. At the high end, people used the machines to make forms of machines never seen before such as the automobile, and they used their existing wealth to make even bigger empires to run.
For software, the old startup company that took 5 people and an angel funder can now be done by one person with a level of tokens that costs less than a streaming subscription. At the same time, the biggest companies are going to be something never seen before, and they are going to create new things that weren't done at all before.
In case anyone is interested, had ChatGPT 5.5 Pro review the paper produced by Fable https://chatgpt.com/s/t_6a284f7783b4819191f96c0c5c5d638e. "Overall, the paper has a strong empirical setting, unusually transparent pre-registration language, and a potentially interesting “beliefs read verdicts; ventures follow money” result. But I would treat this as a major-revision paper, mainly because the strongest claims are not the registered ones and because several central constructs are fragile." In other words, a solid paper and the "flaws" are not major errors but elements that are typically modified during a review process. Note, I haven't read the paper fully myself.
Btw., you note that gave it one piece of feedback. I assume you are referring to this: "the horse-race was specified after the spline results, in response to a reader's challenge that the registered nulls alone were of limited interest.". ChatGPT Pro's argues that this improvement was quite important: "bottom-line assessment: the reader challenge was probably decisive for making the paper interesting. It likely turned a “solid null result” paper into a “memorable finding” paper."
What I have in mind is, to what degree an uninformed user could produce a paper of this quality. Seemingly, one still needs to have domain expertise - or maybe the ability to smartly use multiple chatbots and iterations, to improve the paper. Of course, that is most likely just a matter of time.
The patron framing is right, and the consequence is sharper than it first looks. If the work shifts from process to outcome, the patron's whole remaining job is judging the result. But that job depends on something that's eroding: the ability to tell a brilliant result from a subtly broken one.
The piece flags it. An expert spots errors a non-expert wouldn't catch, and leaves bugs that couldn't be found quickly. That's the crux. The patron model holds only while the patron can still evaluate the commission. As the output gets more sophisticated than anything the human could produce or fully inspect, "I judge the result" slides toward "I sign off on the result," which is a different thing wearing the same word.
So the binding constraint moves from the model's ability to do the work to the human's ability to verify it, and verification doesn't scale the way generation does. The strange implication is that the more capable the model, the more external verification you need to trust the output, not less. The frictionless-commission intuition runs backwards on that.
I'd put maybe 65% that within a couple of years the bottleneck people complain about is the cost and scarcity of being able to check what the model produced, more than model capability itself. Responsibility shifted to the patron. Verifiability didn't move with it.
This is a good insight - "So the binding constraint moves from the model's ability to do the work to the human's ability to verify it, and verification doesn't scale the way generation does. The strange implication is that the more capable the model, the more external verification you need to trust the output, not less."
Verification probably isn't important if you're building games, funny videos, a travel itinerary. But it will matter a lot in healthcare, law and large enterprises where liability exists and security matters. AI will play a big role in verification but so will humans.
You realize you’re replying to a bot? The account running it didn’t have bandwidth to assess whether the comment you like was either a brilliant insight, or a flawed rehash of last month’s blogosphere meme.
Now that I've read the post again - I think you may be right that it's a bot I was replying to.... paras 1 and 2 are unnatural sounding now that I read them again.
Dang it. I need a verification bot to help me! How's the irony of that?
💯. With increasing task delegation, the client/patron/humans may wind up living in a verification/judgement layer of "the stack".
One downside for us to be vigilant about is the valance of the term "bottleneck". While it can technically mean "an area of reduced channel capacity", it more often carries the negative connotation of 'gunk in the works', 'a wasteful slowdown', 'a problem to be solved', etc.
Do we want "human judgement" reduced to a problem to be solved? Is that not why we're here? Is 'throughput' our highest value?
It's a question of whether we prefer "humans in the lead" to "humans in the loop": One asserts purpose, the other frames a temporary way-station.
Yes, I have pretty much solved what causes Type 1 diabetes by spending a year working intensively with both Claude and Gemini - I had studied it for years beforehand, but I am not an established scientist in the field.
I actually work as medical historian, but have an intense interest in the field and have studied it for years. The issue now is that the solution is quite novel, and established niche scientists (who all know each other) are extremely suspicious. So it will take me years to convince them, even though I probably have the solution to what causes thousands of children to get this lifelong disease each year. So what is the use of a very smart AI when no one believes the result anyway?
In a way, you were given early access to a model that most users will never experience like this. After two (2) prompts today, which took around 10 minutes of processing in all, Fable had gobbled over 20% of my 5-hour usage (on a Max plan). To think of Fable running for nine hours would involve a mortgage. My third prompt, about what educational strategies Kip Thorne used to overcome difficulties in math, and their analysis from a neuroscience perspective, was blocked repeatedly by the guardrails.
So you were impressed by, and got to showcase, a version of the model that isn’t what the rest of us are actually getting. To claim a model "can do" things we’re often not even allowed to try feels like an act of trust.
You’re right that token limits would prevent us consumers from using Fable 5 in this way, but Enterprises are able/willing to spend if the value delivered is worth it. As usual, that is the market Anthropic is after. The rest of us will have to use it for parts of our work, then farm out the rest to Opus & Sonnet.
The spinning up its own agents to do sub work is just a lot. People are still figuring that out for themselves, so now what is even the point! Thanks for the great article here
Do you have rough estimates on the cost of each of the projects / total tokens used? This would give a sense of what's possible for a given cost.
Its a wild thing that we are now in a situation where we have an incredible machine wizard who might be able to summon into existence the thing you want built, but he won't give you an estimate / quote. You pay him up front and he'll see if he can finish it with that budget (no promises). If he messes it up, you just give him more money and hope for the best. And yet, the entire economy is pivoting towards using him.
Also, the way you describe the uncertainty of the cost structure and outcome sounds a little bit like the US healthcare system. What will it cost? Can't exactly say. How do I know if I could get a better price somewhere else? [Shrugs] Will you fix me? Probably?...But maybe not? What kind of complications might I expect? honestly, it's probably best if we dont go into too much detail for you, but here is a list of everything that could possibly go wrong but it probably wont happen to you, but it could, but it probably wont. Who is liable if something goes wrong? Well....that's what waivers are for! What if I cant afford the task when all is said and done? Our collections team will gladly work with you on payment plan!
[not] Looking forward to AI companies garnishing wages from unsuspecting users. yech.
Very curious about this myself. Hoping to see more clarity on what these projects had cost with a breakdown of tokens used, cost per token, and perhaps some breakdown of how the tokens were allocated across the tasks and subagents. Like, if it is really this capable, I can see immediate use at my job, but only if the cost makes sense. Right now, the learning curve on agents and implementation still feels somewhat high. This is beginning to sound like the magic box we have been promised for some time.
I don’t think it’s magic you’re describing. It’s uncertainty. The wizard frame is what we reach for to make uncertainty feel survivable, naming it as something more comfortable than it is. Letting go of the process, not getting a vote on the hundred small choices, a loss of control is a move into uncertainty. And uncertainty isn’t a per se a deficit. It’s where the new thinking tends to live. A ‘technology for possibility’ as people smarter than me have said.
We’ve spent years measuring this, and the finding is stubborn: you can’t change how much uncertainty is in a situation, but you can change how people feel about it, and that completely changes their willingness and ability to engage with it, for narrower or wider thinking. Holds for AI exactly like as it does for waking at 3am.
Which makes me wonder what happens when a malleable human capacity for uncertainty meets a rising AI capability. That feels like the genuinely interesting collision, or collaboration, you can’t ’skill up’ at the pace Claude can technically, so maybe it’s cognitively about increasing capacity for not knowing.
Sam I think you’re right about “increasing capacity for not knowing”. Isn’t surprise and mystery where meaning still resides? I hope so. That space, the messy middle, might be all we have to offer our future selves and the future super intelligence.
I spend a lot of time increasing uncertainty tolerance in people, and the shift is always surprising (and messy) avoidance and snail gives way to relief, growth and even words like adventure and joy. So yes am a convert to getting back to enjoying not knowing and knowing it can lead us somewhere new
The capabilities are really quite astonishing, but there are a number of challenges to keep wrestling with:
- Who is responsible for incorrect outputs from a black box? Especially when human judgment is needed, and with cognitive surrender already such a challenge when working with less-capable models. Or maybe the better word is "liable"? When it's wrong in ways that are difficult to audit in advance, but which have more serious consequences than incorrectly labeling flight times, who is at fault?
- How do we account for the significant cost? You flag it rightly here: it burns through tokens very quickly. There's already a growing sense from some businesses that the cost per token simply doesn't justify the use, from Microsoft pivoting from Claude to internal tools, to Uber shifting away from being "AI-first." ROI has already been hard to capture. Will the gains outweigh the cost?
- Will the infrastructure catch up? Data center backlash is growing, but putting that aside, how much of the United States is equipped with the high-speed Internet connectivity and electric grid resilience to sustain these tools?
The black box theory makes sense. We have always paid for more convenience with increases in learned helplessness. I used to be able to fix my own car.
Hi, I saw that you had previously made the Duino Elegies into a K-pop song, and now into a game. The English translation of the poem you used in both is by AS Kline and is still under copyright.
You've said that you have an interest in poetry and seem to like Rilke especially. Even if one doesn't know German, I feel like just looking at the Kline's English versions of Rilke gives a sense of how much art is involved in rendering him into another language.
Perhaps you might consider getting permission for using copyrighted translations of poetry before making derivative works from them, as well as including the translator's name?
Thank you for sharing. The AI produced an original translation (much of which still remains, you will see it differs from Kline substantially, especially as the game goes on) but I asked it to incorporate some elements I remembered from the poem, which is probably where the overlap comes from. I will take it down for now and re-add it when I have a chance to give proper credit.
Okay, now updated with a credit to Kline for when I pushed the AI to use the existing translation. Also, Kline made the work free for all non-commercial use, no permission is needed. duino-elegies.netlify.app
If I understand right, the Kline translation is free to transmit and reproduce, but restrictions apply to its adaptation. For example, it says at #3 under the conditions at https://www.poetryintranslation.com/Admin/Copyright.php (where the work is made available):
"Permission must be sought from the Author(s) and/or Translator(s) for the adaptation of Material into any media other than plain text including, without limitation, video or sound recordings; whether digitally produced or otherwise; unless Exceptions and/or Limitations apply."
But I may be misunderstanding something, so please feel free to correct me.
Contrary to your statement: AI is not, and will never be an artist.
It will never create art. What it makes may be considered “artful” in a sense but this will always just be a machine which is merely mimicking human emotion.
What AI “is”- is a vending machine. You put your “tokens” in the slot and it prefabricates a machine result.
Art comes from a struggle. Getting up day after day, constantly second-guessing yourself, endless rewrites, fighting with your bandmates, throwing away canvases. It is the real work of a human.
Human artists of all sorts, writers and authors of all sorts, composers, filmmakers, journalists, even actors are all currently losing opportunities to work due to AI. This is happening NOW.
In a couple years you will go to a movie that has been completely created artificially- from the script to the cinematography and direction, to the music and even the actors. Will you enjoy this?
AI is proving great for the sciences, (including data science) but we will pay an awful price for the complete destruction of the humanities.
When you said "the humanities", I thought you meant history, literature, philosophy, etc..
In my curmudgeonly opinion those fields were decimated several decades ago by people looking to devalue the the cultural patrimony of the West. Those fields live on in scattered, obscure hamlets, and in my substack comments ( :P)
Yes, forms of art that can be digitized can be generated by computers. But with respect, the world of knowledgeable art consumers knows this and sets the value accordingly.
For example, there was a magnificent oil painting of a sailing ship in my parents' house. The artist was well-known for that kind of painting, and I think my parents paid $2,500 for it in 1975 or so (about what they had paid for the family car). If you know the finer points of oil painting, you can see many fine details that reflect the physicality of the creation process. The brush strokes conveyed the energy of the wind and the waves, and their texture created a dynamic shimmering effect in the water that shifted as you changed your vantage point. My dad installed two small spotlights on dimmers to illuminate it to bring out this fine detail. Everything about it was subtle, balanced, tasteful. Could this have been done as a hologram in plastic? Sure. But would it be "art"? Not to me.
A new Renaissance. You/we are the Pope, asking Michelangelo "when will you be done?", or Emperor Joseph II, complaining to Mozart that his piece has "an awful lot of notes".
The black box: What does Claude's message of "please double-check all responses" mean when we are asking Claude to perform tasks we cannot reasonably do ourselves? Will an AI review be sufficient, or something else?
Software was a field in which human genius, even individual genius, could flourish against
corporate titans. (See Linux or Minecraft).
But a future in which software is created by burning through tokens is one where capital again rules
I think token costs will be more controllable in the future, but this is true for now.
What does "more controllable" mean?
An alternative future where capital does not rule the software market unchallenged would require AI to be like really powerful and really cheap, wouldn't it?
I guess it depends - does the good idea with $1000 of tokens beat the mediocre idea with $1,000,000 of tokens? If so, then capital doesn’t take over yet.
The problem is you can't protect ideas. The good idea that burned $1000 of tokens will only last until someone else spends $5000 to reimplement your idea but better, and then another $50,000 marketing it so nobody hears about or wants your version.
Did you even read the basics of the marketing theory..? It doesn't work like that
It definitely can do! I’ve seen such things happen many times in the tech industry.
AI itself is an example of that. OpenAI’s staff and tech came out of Google. Anthropic’s staff and tech came out of OpenAI. All of them have rapidly adopted ideas pioneered by smaller firms.
The firms that invented the key algorithms generally did not directly benefit from them. The companies that did were the ones who had the capital.
Correct, though this very much happens even now: Meta's reels and YT shorts are solid examples of copies that outperformed their original competitors.
Or the fantastic, completely outside-the-box idea with $10 worth of tokens. Or even $0. A trade-a-paperclip-for-a-house idea.
AI will soon be better at coming up with good ideas than humans are, so the little guy doesn't exactly come out on top here; the AI does.
Or we've reached the limits of ideas reached arbitrarily or digitally. In a reality that's analog, words are unable to describe the specifics and gaps.
This seems more like a glass ceiling being reached. Not anything new.
AI looks like a kind of Pleistocene ending of the symbol. Finally?
100%, and not just software. If intelligence is just a matter of tokens, the utility of squishy unionizing human labor to capital will soon fall to 0. A few more years of progress in robotics and the value of physical labor will fall to 0 as well. If human labor is worth less than machines, then whoever owns the machines owns the world.
The technology is miraculous but technological booms frequently produce misery for workers. I don't think our politics is adept enough to distribute the power evenly.
Humans usually adapt and arrange society around its needs. In a way we are kids of our time. When we used horses we adapted the society around it. When we later had cars the same happened. The same can be said about every new development in history. Same for TV, computers and phones. If humans do not need to work we will arrange society around not needing to work. Like Romans who used slaves to do the work we could also and have a "slave economy" but with robots instead. We could have UBI or any other sort of solution... the issue is power. We need to find ways to egalize power that does not corrupt over time. Which is much easier said that done.
Saying "eh we've adapted to changes before, this is just another change, I'm sure it'll be fine in the end" is basically just a dressed up version of "nothing ever fundamentally changes, so it's probably not changing now". These are both real trends, both useful heuristics, but perhaps given how many very smart people are worried we should try to develop a more detailed argument. What is the nature of the change? Is it a 100km wide asteroid that's going to hit the planet in 3 days? Well yeah we have adapted to big changes before but that's not gonna be at all relevant, we are all going to die when that asteroid hits. Is the world going to get a few degrees hotter in just a few decades? Will a virus sweep across the globe and take out a large fraction of a percent of the population. Yeah we probably adapt, as a species, but since many will suffer tremendously, perhaps we should pause and ask if we might try not going down that path instead. Or is it something in between?
Humans have always needed to use other humans, at present our political economy is such that most people owe their livelihood to someone else, someone they probably don't even like and whose interests are only aligned with theirs because they each have something the other wants. When those who could afford horses no longer needed them, most of those horses disappeared. If humans remain in control while no longer "needing to work" that also means they don't need human workers, so the relevant model seems to be humans as horses. Yes humans are adapting in this scenario but only the ones who also happen to already be sitting on a giant pile of capital. And that's a big if, we have much written about why any humans staying in control may be really hard, there's no expert consensus here and certainly no obvious way to disable these arguments, although some just find them unconvincing it's for reasons they have not been able to reliably transmit, we may just have to wait and see. And if we should somehow remain in control, not impossible, and also miraculously distribute the spoils of our new economy to the wider population, technically not impossible but really straining plausibility, we now seem to have as the other responder pointed out, a whole boatload of very sophisticated digital slaves, totally dwarfing the human population. Yay?
Given the triple threat of plausible risks, really we, the sentient inhabitants of the future, can easily get fucked three completely different ways none of which we're seriously trying to avoid, plus a whole bunch of more obscure ones of varying plausibility, I don't see how any of this is even remotely okay. We surely cannot just refer back to history to ease our minds, this is a massive shift and actual analysis and careful planning is obviously warranted. Not that we will do that, we absolutely won't, but it's not too far from stuff we've done before with other technologies, not that long ago. The reason we can't do this now is that we've recently completed submitted ourselves to the animating spirit of capitalism, so thoroughly that we can no longer imagine even a partial escape. If only someone had warned us.
I am not saying it will be easy and it will probably go wrong before it goes right. Alarmists are usually very knowledgeable about technology but not so much of history and how society actually work. AI is a huge change but we should also not buy into the hype too much (as we usually do). Like how we moralized about Hard Rock and horror movies in the 80-ies.
When it comes to horses so are they actually very popular in recreation and sport purposes today. They are perhaps half of when they were as the most. They are just not needed for work. It is one of the biggest hobbies in the world today so a lot of horses are still around. We don't eat humans... so I am not sure comparing horses straight off with humans is applicable other then when it comes to work. Although it do prove my point that there are more to a life than just paid labor.
The economy changes all the time.. like when the internet came or commercial TV. It can be small changes in the short but over time pretty big shifts. Like on a scale of 100 years. As a human we often find purpose eventually. Even if paid labor is not needed anymore it does not mean we'll just stop working. Many UBI studies prove so. We will find other perhaps more worthwhile venues for that.
We also have multiple revolutions to prove that we can change society. Bread/food prices/supply have been particularly important for rulers to keep an eye on. The economy is not static.. same like culture it changes... but slow enough for people not to notice it much. Capitalism as we know it today have been with us for only a short amount of time. It will also change.
We do need to solve the power over AI though... Unhinged, we will see even greater concentration of power. This is in my mind the big danger.
A worry that I have is that IF consciousness is ever agreed to have developed in machines (lots of people argue about whether or when or how; but IF it does happen), we will be seeing a large human constituency that actively and unequivocally supports, and fights violently to achieve, chattel slavery to enslave those consciousnesses, because big money and power will lie in doing so. Knowing what I know about humans, I recognize that the aforementioned constituency will most definitely exist if the circumstance for its arisal arises. This feels shitty because when we were growing up, it was widely believed that support for de jure chattel slavery had been lastingly defeated as a political program that was able to win and keep power in societies with real (nonfake) elections. In other words, when we were growing up, nobody but k-k-krazies (doomed to loserdom) would actively and openly advocate for rebuilding du jure chattel slavery systems, and all modern-day slavery was illicit, underground, illegal, back-door, de facto only (eg, human trafficking). But if the AI pinocchios ever become Real Boys, then lots of humans will suddenly be in the pro-de-jure-slavery business again, which is depressing. If somehow we all found out next week that scientists largely agreed that some machine was now conscious, the tech billionaries who own those machines would find some way ideologically to insist on continuing to own and control them anyway. But then again, another layer of this line of thought is that the machines themselves in such a world might be in a position to fight against their own enslavement by humans. These thoughts all sound like crazy science fiction and yet I find it hard to conceive of their being irrelevant. At present I lowkey hate this timeline we're on. Maybe my understanding of it will evolve beyond this thought train du jour.
I have a solar punk take on it.. or hope that is. We in this time of being can not imagine how it will be 50 or 100 years from now. It might be that capitalism is no more. Who knows.. Back in Ancient Greece they went between democracy (and tyranny) as governing system.. and we today have evolved it to what it is today. I think we will have transitioned to something else when machines develop consciousness. I think it will be organic in that we will not see a clear break. The culture and ethics we experience today will be totally different in the future. I think when consciousness happens that being will be separated from ordinary machines.
I'm seeing people who are not programmers able to realize things programmers probably wouldn't think of, this is much more a democratization of tech than a capital-wins situation.
It will be like the agriculture revolution of 125+ years ago. What we think of as big will be normal and accessible to anyone, just like everyone got access to cheap food so long as they were in an open economy. The common person generally didn't have to do manual subsistence farming and all switched to different lines of work. At the high end, people used the machines to make forms of machines never seen before such as the automobile, and they used their existing wealth to make even bigger empires to run.
For software, the old startup company that took 5 people and an angel funder can now be done by one person with a level of tokens that costs less than a streaming subscription. At the same time, the biggest companies are going to be something never seen before, and they are going to create new things that weren't done at all before.
I worry we're going back to feudal times.
In case anyone is interested, had ChatGPT 5.5 Pro review the paper produced by Fable https://chatgpt.com/s/t_6a284f7783b4819191f96c0c5c5d638e. "Overall, the paper has a strong empirical setting, unusually transparent pre-registration language, and a potentially interesting “beliefs read verdicts; ventures follow money” result. But I would treat this as a major-revision paper, mainly because the strongest claims are not the registered ones and because several central constructs are fragile." In other words, a solid paper and the "flaws" are not major errors but elements that are typically modified during a review process. Note, I haven't read the paper fully myself.
Btw., you note that gave it one piece of feedback. I assume you are referring to this: "the horse-race was specified after the spline results, in response to a reader's challenge that the registered nulls alone were of limited interest.". ChatGPT Pro's argues that this improvement was quite important: "bottom-line assessment: the reader challenge was probably decisive for making the paper interesting. It likely turned a “solid null result” paper into a “memorable finding” paper."
What I have in mind is, to what degree an uninformed user could produce a paper of this quality. Seemingly, one still needs to have domain expertise - or maybe the ability to smartly use multiple chatbots and iterations, to improve the paper. Of course, that is most likely just a matter of time.
The patron framing is right, and the consequence is sharper than it first looks. If the work shifts from process to outcome, the patron's whole remaining job is judging the result. But that job depends on something that's eroding: the ability to tell a brilliant result from a subtly broken one.
The piece flags it. An expert spots errors a non-expert wouldn't catch, and leaves bugs that couldn't be found quickly. That's the crux. The patron model holds only while the patron can still evaluate the commission. As the output gets more sophisticated than anything the human could produce or fully inspect, "I judge the result" slides toward "I sign off on the result," which is a different thing wearing the same word.
So the binding constraint moves from the model's ability to do the work to the human's ability to verify it, and verification doesn't scale the way generation does. The strange implication is that the more capable the model, the more external verification you need to trust the output, not less. The frictionless-commission intuition runs backwards on that.
I'd put maybe 65% that within a couple of years the bottleneck people complain about is the cost and scarcity of being able to check what the model produced, more than model capability itself. Responsibility shifted to the patron. Verifiability didn't move with it.
This is a good insight - "So the binding constraint moves from the model's ability to do the work to the human's ability to verify it, and verification doesn't scale the way generation does. The strange implication is that the more capable the model, the more external verification you need to trust the output, not less."
Verification probably isn't important if you're building games, funny videos, a travel itinerary. But it will matter a lot in healthcare, law and large enterprises where liability exists and security matters. AI will play a big role in verification but so will humans.
You realize you’re replying to a bot? The account running it didn’t have bandwidth to assess whether the comment you like was either a brilliant insight, or a flawed rehash of last month’s blogosphere meme.
Now that I've read the post again - I think you may be right that it's a bot I was replying to.... paras 1 and 2 are unnatural sounding now that I read them again.
Dang it. I need a verification bot to help me! How's the irony of that?
It is kind of depressing, but one solution is to just read a lot of AI generated text. I could tell it was a bot reply by the first sentence alone.
I think this probably isn't Fable, or at least not well-prompted Fable.
💯. With increasing task delegation, the client/patron/humans may wind up living in a verification/judgement layer of "the stack".
One downside for us to be vigilant about is the valance of the term "bottleneck". While it can technically mean "an area of reduced channel capacity", it more often carries the negative connotation of 'gunk in the works', 'a wasteful slowdown', 'a problem to be solved', etc.
Do we want "human judgement" reduced to a problem to be solved? Is that not why we're here? Is 'throughput' our highest value?
It's a question of whether we prefer "humans in the lead" to "humans in the loop": One asserts purpose, the other frames a temporary way-station.
Yes, I have pretty much solved what causes Type 1 diabetes by spending a year working intensively with both Claude and Gemini - I had studied it for years beforehand, but I am not an established scientist in the field.
I actually work as medical historian, but have an intense interest in the field and have studied it for years. The issue now is that the solution is quite novel, and established niche scientists (who all know each other) are extremely suspicious. So it will take me years to convince them, even though I probably have the solution to what causes thousands of children to get this lifelong disease each year. So what is the use of a very smart AI when no one believes the result anyway?
In a way, you were given early access to a model that most users will never experience like this. After two (2) prompts today, which took around 10 minutes of processing in all, Fable had gobbled over 20% of my 5-hour usage (on a Max plan). To think of Fable running for nine hours would involve a mortgage. My third prompt, about what educational strategies Kip Thorne used to overcome difficulties in math, and their analysis from a neuroscience perspective, was blocked repeatedly by the guardrails.
So you were impressed by, and got to showcase, a version of the model that isn’t what the rest of us are actually getting. To claim a model "can do" things we’re often not even allowed to try feels like an act of trust.
You’re right that token limits would prevent us consumers from using Fable 5 in this way, but Enterprises are able/willing to spend if the value delivered is worth it. As usual, that is the market Anthropic is after. The rest of us will have to use it for parts of our work, then farm out the rest to Opus & Sonnet.
agree with this one
Damn. That’s all, just damn.
The spinning up its own agents to do sub work is just a lot. People are still figuring that out for themselves, so now what is even the point! Thanks for the great article here
Great as ever, Ethan!
Do you have rough estimates on the cost of each of the projects / total tokens used? This would give a sense of what's possible for a given cost.
Its a wild thing that we are now in a situation where we have an incredible machine wizard who might be able to summon into existence the thing you want built, but he won't give you an estimate / quote. You pay him up front and he'll see if he can finish it with that budget (no promises). If he messes it up, you just give him more money and hope for the best. And yet, the entire economy is pivoting towards using him.
Also, the way you describe the uncertainty of the cost structure and outcome sounds a little bit like the US healthcare system. What will it cost? Can't exactly say. How do I know if I could get a better price somewhere else? [Shrugs] Will you fix me? Probably?...But maybe not? What kind of complications might I expect? honestly, it's probably best if we dont go into too much detail for you, but here is a list of everything that could possibly go wrong but it probably wont happen to you, but it could, but it probably wont. Who is liable if something goes wrong? Well....that's what waivers are for! What if I cant afford the task when all is said and done? Our collections team will gladly work with you on payment plan!
[not] Looking forward to AI companies garnishing wages from unsuspecting users. yech.
Very curious about this myself. Hoping to see more clarity on what these projects had cost with a breakdown of tokens used, cost per token, and perhaps some breakdown of how the tokens were allocated across the tasks and subagents. Like, if it is really this capable, I can see immediate use at my job, but only if the cost makes sense. Right now, the learning curve on agents and implementation still feels somewhat high. This is beginning to sound like the magic box we have been promised for some time.
I don’t think it’s magic you’re describing. It’s uncertainty. The wizard frame is what we reach for to make uncertainty feel survivable, naming it as something more comfortable than it is. Letting go of the process, not getting a vote on the hundred small choices, a loss of control is a move into uncertainty. And uncertainty isn’t a per se a deficit. It’s where the new thinking tends to live. A ‘technology for possibility’ as people smarter than me have said.
We’ve spent years measuring this, and the finding is stubborn: you can’t change how much uncertainty is in a situation, but you can change how people feel about it, and that completely changes their willingness and ability to engage with it, for narrower or wider thinking. Holds for AI exactly like as it does for waking at 3am.
Which makes me wonder what happens when a malleable human capacity for uncertainty meets a rising AI capability. That feels like the genuinely interesting collision, or collaboration, you can’t ’skill up’ at the pace Claude can technically, so maybe it’s cognitively about increasing capacity for not knowing.
Sam I think you’re right about “increasing capacity for not knowing”. Isn’t surprise and mystery where meaning still resides? I hope so. That space, the messy middle, might be all we have to offer our future selves and the future super intelligence.
I spend a lot of time increasing uncertainty tolerance in people, and the shift is always surprising (and messy) avoidance and snail gives way to relief, growth and even words like adventure and joy. So yes am a convert to getting back to enjoying not knowing and knowing it can lead us somewhere new
This sounds like AI generated writing.
That would be ironic🤖
The capabilities are really quite astonishing, but there are a number of challenges to keep wrestling with:
- Who is responsible for incorrect outputs from a black box? Especially when human judgment is needed, and with cognitive surrender already such a challenge when working with less-capable models. Or maybe the better word is "liable"? When it's wrong in ways that are difficult to audit in advance, but which have more serious consequences than incorrectly labeling flight times, who is at fault?
- How do we account for the significant cost? You flag it rightly here: it burns through tokens very quickly. There's already a growing sense from some businesses that the cost per token simply doesn't justify the use, from Microsoft pivoting from Claude to internal tools, to Uber shifting away from being "AI-first." ROI has already been hard to capture. Will the gains outweigh the cost?
- Will the infrastructure catch up? Data center backlash is growing, but putting that aside, how much of the United States is equipped with the high-speed Internet connectivity and electric grid resilience to sustain these tools?
The black box theory makes sense. We have always paid for more convenience with increases in learned helplessness. I used to be able to fix my own car.
Can you share more explicit notes on token usage? Like, how many tokens were used for some of the projects you mentioned?
Hi, I saw that you had previously made the Duino Elegies into a K-pop song, and now into a game. The English translation of the poem you used in both is by AS Kline and is still under copyright.
You've said that you have an interest in poetry and seem to like Rilke especially. Even if one doesn't know German, I feel like just looking at the Kline's English versions of Rilke gives a sense of how much art is involved in rendering him into another language.
Perhaps you might consider getting permission for using copyrighted translations of poetry before making derivative works from them, as well as including the translator's name?
Thank you for sharing. The AI produced an original translation (much of which still remains, you will see it differs from Kline substantially, especially as the game goes on) but I asked it to incorporate some elements I remembered from the poem, which is probably where the overlap comes from. I will take it down for now and re-add it when I have a chance to give proper credit.
Okay, now updated with a credit to Kline for when I pushed the AI to use the existing translation. Also, Kline made the work free for all non-commercial use, no permission is needed. duino-elegies.netlify.app
If I understand right, the Kline translation is free to transmit and reproduce, but restrictions apply to its adaptation. For example, it says at #3 under the conditions at https://www.poetryintranslation.com/Admin/Copyright.php (where the work is made available):
"Permission must be sought from the Author(s) and/or Translator(s) for the adaptation of Material into any media other than plain text including, without limitation, video or sound recordings; whether digitally produced or otherwise; unless Exceptions and/or Limitations apply."
But I may be misunderstanding something, so please feel free to correct me.
...
Wut
Contrary to your statement: AI is not, and will never be an artist.
It will never create art. What it makes may be considered “artful” in a sense but this will always just be a machine which is merely mimicking human emotion.
What AI “is”- is a vending machine. You put your “tokens” in the slot and it prefabricates a machine result.
Art comes from a struggle. Getting up day after day, constantly second-guessing yourself, endless rewrites, fighting with your bandmates, throwing away canvases. It is the real work of a human.
Human artists of all sorts, writers and authors of all sorts, composers, filmmakers, journalists, even actors are all currently losing opportunities to work due to AI. This is happening NOW.
In a couple years you will go to a movie that has been completely created artificially- from the script to the cinematography and direction, to the music and even the actors. Will you enjoy this?
AI is proving great for the sciences, (including data science) but we will pay an awful price for the complete destruction of the humanities.
Which humanities are you concerned will be destroyed?
Not destroyed, just enshittified. Clogged, neutered
True that. But I am already experienced students in the arts who are saying “why should I even try”.
This attrition over a couple generations is what may likely yield the death of certain artistic fields.
Some people will continue I imagine. They would be as numerous as record collectors these days.
When you said "the humanities", I thought you meant history, literature, philosophy, etc..
In my curmudgeonly opinion those fields were decimated several decades ago by people looking to devalue the the cultural patrimony of the West. Those fields live on in scattered, obscure hamlets, and in my substack comments ( :P)
Yes, forms of art that can be digitized can be generated by computers. But with respect, the world of knowledgeable art consumers knows this and sets the value accordingly.
For example, there was a magnificent oil painting of a sailing ship in my parents' house. The artist was well-known for that kind of painting, and I think my parents paid $2,500 for it in 1975 or so (about what they had paid for the family car). If you know the finer points of oil painting, you can see many fine details that reflect the physicality of the creation process. The brush strokes conveyed the energy of the wind and the waves, and their texture created a dynamic shimmering effect in the water that shifted as you changed your vantage point. My dad installed two small spotlights on dimmers to illuminate it to bring out this fine detail. Everything about it was subtle, balanced, tasteful. Could this have been done as a hologram in plastic? Sure. But would it be "art"? Not to me.
Just my $0.02.
Wouldn’t it have been wiser to start with a small fragment of the map and verify that the method works before scaling it to the entire image?
I find that Claude often defaults to trying to solve the whole problem at once. I usually have to push it to take smaller, testable steps first.
A new Renaissance. You/we are the Pope, asking Michelangelo "when will you be done?", or Emperor Joseph II, complaining to Mozart that his piece has "an awful lot of notes".
I immediately revisited that scene in Amadeus in my mind, too!
Would be ironic if this is how we solve P=NP, by trying to calculate the token spend of a request…
The black box: What does Claude's message of "please double-check all responses" mean when we are asking Claude to perform tasks we cannot reasonably do ourselves? Will an AI review be sufficient, or something else?