My takeaway is that we're paying an awful lot of people an awful lot of money to do very low-value things (which is mostly what current AI can do). AI can (help) write a webserver because lots of webservers have already been written -- why would you be paying someone to write yet another one? Ask it to write some truly novel code and see what you get
Or the other view, realise that we have overpaid management consultants, but will continue to - not because AI can't do it. But because "no one ever got fired for using McKinseys". https://twitter.com/emollick/status/1639329763197333505
AI tools are certainly having an impact on productivity. I've been using it to boost my coding speed and it's been great.
But I'm not completely convinced that the stats in these papers are useful.
The Github Copilot study was tested with a task that almost no one will ever do in a real world situation, so it makes sense that the developers needed more time to work on it. There are likely some examples of a simple HTTP server on Github, so Copilot unsurprisingly knew how to do it. If we tried giving Copilot a problem that people actually solve at their companies, which requires specific knowledge of that company's codebase, I think the results would be significantly different.
The ChatGPT productivity study doesn't seem to have released the actual tasks done in the study, but in their own words "These comments point to an important (and inherent) limitation of our experiment: it involves relatively small, self-contained tasks that lack much context-specific knowledge beyond what we stipulate in the task prompts." Just like the Copilot example, whatever tasks they used were specifically optimized to be tasks that ChatGPT could do, and doesn't necessarily reflect real world tasks.
They do follow that up with "However, our core result, that ChatGPT can
increase productivity on many mid-level professional writing tasks, is supported by the fact
that many respondents choose to use it in their real jobs.", but that doesn't really give us stats on how productive it's making people in the real world. Perhaps it's just making people *feel* more productive and cool, and that's why they're using it at their jobs.
One consequence of what you note: companies that use less legacy code and more generic components (especially open source) can benefit the most from existing AI tools. There might also be an opportunity to work out how to effectively finetune generic code completion systems for the codebase of each organization, if the amount of code is large enough.
Thank you for pointing out what’s been bothering me about all this hype. I’m completely convinced that this is the future but it’s needs some way of proving that it’s correct (or at least some tools to make verifying it output easier).
Most of the work is in verifying the code than writing it. As anyone who has worked in software industry can attest to, anyone can write code but only a few will actually have the knowledge and take the time to make sure it’s correct and scalable.
But then again there was that news story about a university writing an apology letter using ChatGPT where the generated apology had the actual words “made by ChatGPT” and no one bothered to proofread the damn thing. People just suck and maybe this just becomes “one of those things” like how we all say we care about human rights, being a good person, and whatever but will happily and gleeful give up for clout, convenience, or wealth.
Pretty wide CI on the study results, but, yeah, increasing time efficiency for using the vast arsenal of canned heat makes sense.
We need, however, to consider the difference between “efficiency” (doing things right) and “effectiveness” (doing the right things). Frustration, not innovation, is the mother of invention. By greasing the skids on the same old same old, AI, might also remove the friction leading to the “there must be a better way!” driven breakthroughs l
Thank you for this post - it drives home a critical point - ChatGPT is a tool.
The question is - how long will it take to learn to use the tool correctly. As CK pointed out - you better "read" the results when you generate an apology letter (unless the reality is that you really don't care).
This is a little like walking into a wood-working shop - seeing a rack of bright shiny wood gouges next to a beautiful, polished wood bowl - and thinking you could do that.... Yep, you can..... (I'm using the ChatGPT autofill technique here)
With that in mind - I submitted this prompt to Chat:
"Consider that ChatGPT is a tool. A chainsaw is also a tool. Compare ChatGPT with a chainsaw."
Chat produced 3 paragraphs and a summary . But never got the point.
I'm reminded of the old story about a prison that had a list of jokes. Each joke had a number. The prisoners had heard the jokes so often that in conversation they would just call out the number. The other prisoners would break out in laughter.
Try prompting that joke line to Chat and see if you aren't ROFLOL at the response.
Prompt:I have a question about a joke. It concerns a backwoodsman that bought a chainsaw and didn't know how to use it when he returned it the store owner pull the card when it made noise. He jumped back and asked what's that noise if you heard that one it was in a Peter Coy column in the New York Times
Search….
Bing Response: I think I have heard a similar joke before. It goes like this:
A man goes to a hardware store and asks for a chainsaw that can cut down six trees in one hour. The salesman recommends the top-of-the-line model and says it can cut down ten trees in an hour. The man is impressed and buys it.
The next day, he comes back with the chainsaw and complains that it only cut down one tree in an hour. He asks for a refund. The salesman takes the chainsaw and pulls the cord to start it. The chainsaw makes a loud noise and the man jumps back in shock.
He says, “What’s that noise?”
The joke is supposed to be funny because the man did not know how to use the chainsaw properly and did not realize that it needed to be turned on1. It is also ironic that he bought the most expensive model but did not get any benefit from it.
I hope this answers your question about the joke. 😊
Yes - that answers it. I put this into ChatGPT - it couldn't find the Peter Coy reference. When I removed the reference to Peter and added this "Please complete this joke" to the front - it produced the whole joke.
Looking back, I was trying to get ChatGPT to connect that without training ChatGPT is like a chainsaw, which will cut your leg off if you don't know what you're doing. Probably a stretch.
My experience is that, unless you are very specific the bots get confused. And whenever that happens, I start over.
This time I worked with ChatGPT. My first attempt was unsuccessful but twice in a row I got good answers. In my first attempt, ChatGPT was clearly hallucinating. I used the exact same prompt in all attempts.
I may have been hallucinating as well about my success in January at finding the joke.
Here is my last successful Attempt:
Model: Legacy
Mike Randolph
I am trying to recall a joke. It is about a back woodsman who comes in to a store complaining about how long it takes to cut down trees. The salesman sells him a chainsaw. He comes back a week later complaining that it doesn’t do any better and his old saw. The salesman started it up and he jumped back at the noise saying what was that. Have you heard that joke?
Yes, I am familiar with that joke. It's a classic example of a humor that is based on a misunderstanding. The backwoodsman was expecting a silent chainsaw, but the salesman explained to him that all chainsaws make noise when they are in use.
Mike Randolph
Can you tell the whole joke?
Sure! Here's the full joke:
A backwoodsman walks into a store and complains to the salesman about how long it takes him to cut down a tree with his old saw. The salesman suggests that he try using a chainsaw, which will make the job much easier and quicker. The backwoodsman is skeptical but decides to give it a try, so he buys a chainsaw and leaves the store.
A week later, the backwoodsman walks back into the store, looking even more frustrated than before. He tells the salesman that the chainsaw doesn't seem to be doing any better than his old saw. The salesman offers to start up the chainsaw to show the man how it works. When the salesman starts the chainsaw, the backwoodsman jumps back in surprise at the noise.
"What was that?" the backwoodsman exclaims.
"That's the sound of progress," the salesman replies with a smile.
Mike Randolph
What is the source of the joke?
The origin of this joke is unclear and it's possible that it has been told and retold many times over the years, with variations. This type of joke, with a backwoodsman or rural character coming into a store and being surprised by modern technology, is a common theme in American humor and has likely been a part of the oral tradition for many decades. The joke is often told as a one-liner or a short story, and it's a classic example of humor that is based on a misunderstanding or cultural gap.
Well, at least this joke is funnier. There may be a lot of versions floating out there - or the LLM may have fudged.
I tried "Write a sequel to about the movie "Hero" with Dustin Hoffman."
Chat confused the main characters and their roles. I pointed out its errors and asked it to rewrite 4 times. Each time it was willing to throw one of the characters under the bus so that the other could bask in "virtue".
Its conclusion was always along these lines:
"The sequel to "Hero" would be a heartwarming and inspirational story of redemption and the power of selflessness. It would show that anyone can be a hero, and that even small acts of kindness can have a big impact on the world around us."
Who knows, maybe the movie industry has had access to ChatGPT longer than anyone will admit.
There are some basic mathematical errors in the paper you cite: by Felten, Raj and Seamans (March 2023). Check out the equation A_ij = sum{x_ij : i = 1,...,10} and it goes on. This is very undercooked research folks.
Another implication not noted here (perhaps not part of the study, which I have yet to read), is that by putting the lower skilled workers on quasi-par with veterans, it devalues the latter's expertise. When this narrowing of skills gap translates into a narrowing of wage gap, which direction do you think wages will go? That's the obvious huge issue, which unfortunately I find is seriously not given enough attention to.
Similar to the miners taken as example here, we can look at taxi drivers, whose legendary knowledge of the city no longer has any value. As a result, overall wages have dropped 10% for London cabbies (as per research conducted by Carl Frey).
In short, no, AI does not automate work, a human is still necessary in the loop for now. But just about any human can now do the job, hence it will go to the lowest bidder, and that could get very low. Surprised or even disgusted by the lack of interest in this obvious fact, I woite about this in my book "AI Battle Royale: how to protect your job from disruption in the 4th Industrial Revolution".
You make a good point about people being dishonest about using AI and I think the more these programs automate jobs, the the lazier and dishonest people will become. Humans always look for a way to just fake the metrics but absorb the merited credit.
I love the comparison to the productivity increase brought about by steam power. Something that was alluded to but not specifically mentioned was the impact on KNOWLEDGE WORKERS. Now we have AI, knowledge flows freely across all industries. It was the information age that brought about the knowledge worker and now that we move into the knowledge age, the optimist in me hopes that we see the dawn of the Wisdom worker.
does this drive a new era of equality in society where work that needs to be done face to face starts to gain more value and remote/ knowledge work reduces in value... Maybe carers and plumbers start getting paid more or maybe we need to think about universal wages to level out the impact?
I am dictating this to my iPad, so please forgive any mistakes.
Here’s a prompt I just put into Bing:
I have a question about a joke. It concerns a backwoodsman that bought a chainsaw and didn't know how to use it when he returned it the store owner pull the card when it made noise. He jumped back and asked what's that noise if you heard that one it was in a Peter Coy column in the New York Times
Also dictated, I corrected the spelling of Coy.
It did search the Internet
Here’s your response:
I think I have heard a similar joke before. It goes like this:
A man goes to a hardware store and asks for a chainsaw that can cut down six trees in one hour. The salesman recommends the top-of-the-line model and says it can cut down ten trees in an hour. The man is impressed and buys it.
The next day, he comes back with the chainsaw and complains that it only cut down one tree in an hour. He asks for a refund. The salesman takes the chainsaw and pulls the cord to start it. The chainsaw makes a loud noise and the man jumps back in shock.
He says, “What’s that noise?”
The joke is supposed to be funny because the man did not know how to use the chainsaw properly and did not realize that it needed to be turned on1. It is also ironic that he bought the most expensive model but did not get any benefit from it.
I hope this answers your question about the joke. 😊
Love the blog! I just wanted to point out a *slight* error:
“A study of programmers found a increase of 55.8% in productivity when using AI”
The tasks took the teams 161 minutes vs 71 minutes on average. That represents a productivity *increase* of (161 / 71) - 1, or roughly +127% compared to baseline, which is to say that developers using copilot performed the task *~2.27x* faster than those without it.
Another great analysis, thank you. My hope is that it becomes a badge of honor (oh, how many times I've suggested to an organization that they reward bottom-up change). However, I'd give people a font-spinning, multi-colored, flashing, badge if they also shared the other changes they made to make the AI use work in their setting. A slightly less flashy badge will go to whoever at APA, MLA, etc. helps us know how to effectively cite our new (and old) collaborators. https://terrigriffith.com/blog/t-is-for-talent
My takeaway is that we're paying an awful lot of people an awful lot of money to do very low-value things (which is mostly what current AI can do). AI can (help) write a webserver because lots of webservers have already been written -- why would you be paying someone to write yet another one? Ask it to write some truly novel code and see what you get
Or the other view, realise that we have overpaid management consultants, but will continue to - not because AI can't do it. But because "no one ever got fired for using McKinseys". https://twitter.com/emollick/status/1639329763197333505
my takeaway here is that 20%+ of developers today can't even create a web server, despite having ChatGPT
AI tools are certainly having an impact on productivity. I've been using it to boost my coding speed and it's been great.
But I'm not completely convinced that the stats in these papers are useful.
The Github Copilot study was tested with a task that almost no one will ever do in a real world situation, so it makes sense that the developers needed more time to work on it. There are likely some examples of a simple HTTP server on Github, so Copilot unsurprisingly knew how to do it. If we tried giving Copilot a problem that people actually solve at their companies, which requires specific knowledge of that company's codebase, I think the results would be significantly different.
The ChatGPT productivity study doesn't seem to have released the actual tasks done in the study, but in their own words "These comments point to an important (and inherent) limitation of our experiment: it involves relatively small, self-contained tasks that lack much context-specific knowledge beyond what we stipulate in the task prompts." Just like the Copilot example, whatever tasks they used were specifically optimized to be tasks that ChatGPT could do, and doesn't necessarily reflect real world tasks.
They do follow that up with "However, our core result, that ChatGPT can
increase productivity on many mid-level professional writing tasks, is supported by the fact
that many respondents choose to use it in their real jobs.", but that doesn't really give us stats on how productive it's making people in the real world. Perhaps it's just making people *feel* more productive and cool, and that's why they're using it at their jobs.
One consequence of what you note: companies that use less legacy code and more generic components (especially open source) can benefit the most from existing AI tools. There might also be an opportunity to work out how to effectively finetune generic code completion systems for the codebase of each organization, if the amount of code is large enough.
Thank you for pointing out what’s been bothering me about all this hype. I’m completely convinced that this is the future but it’s needs some way of proving that it’s correct (or at least some tools to make verifying it output easier).
Most of the work is in verifying the code than writing it. As anyone who has worked in software industry can attest to, anyone can write code but only a few will actually have the knowledge and take the time to make sure it’s correct and scalable.
But then again there was that news story about a university writing an apology letter using ChatGPT where the generated apology had the actual words “made by ChatGPT” and no one bothered to proofread the damn thing. People just suck and maybe this just becomes “one of those things” like how we all say we care about human rights, being a good person, and whatever but will happily and gleeful give up for clout, convenience, or wealth.
Pretty wide CI on the study results, but, yeah, increasing time efficiency for using the vast arsenal of canned heat makes sense.
We need, however, to consider the difference between “efficiency” (doing things right) and “effectiveness” (doing the right things). Frustration, not innovation, is the mother of invention. By greasing the skids on the same old same old, AI, might also remove the friction leading to the “there must be a better way!” driven breakthroughs l
Thank you for this post - it drives home a critical point - ChatGPT is a tool.
The question is - how long will it take to learn to use the tool correctly. As CK pointed out - you better "read" the results when you generate an apology letter (unless the reality is that you really don't care).
This is a little like walking into a wood-working shop - seeing a rack of bright shiny wood gouges next to a beautiful, polished wood bowl - and thinking you could do that.... Yep, you can..... (I'm using the ChatGPT autofill technique here)
With that in mind - I submitted this prompt to Chat:
"Consider that ChatGPT is a tool. A chainsaw is also a tool. Compare ChatGPT with a chainsaw."
Chat produced 3 paragraphs and a summary . But never got the point.
I'm reminded of the old story about a prison that had a list of jokes. Each joke had a number. The prisoners had heard the jokes so often that in conversation they would just call out the number. The other prisoners would break out in laughter.
Try prompting that joke line to Chat and see if you aren't ROFLOL at the response.
I asked about the chainsaw joke a few months ago. It took a couple of tries, but Chat got the point eventually.
Today. I think I could do it on my first try. Takes time to learn how to use the tool.
Sure would appreciate it if you could post that prompt. I'm first to admit that I'm learning - slowly, but I'm learning.
Thanks.
Prompt:I have a question about a joke. It concerns a backwoodsman that bought a chainsaw and didn't know how to use it when he returned it the store owner pull the card when it made noise. He jumped back and asked what's that noise if you heard that one it was in a Peter Coy column in the New York Times
Search….
Bing Response: I think I have heard a similar joke before. It goes like this:
A man goes to a hardware store and asks for a chainsaw that can cut down six trees in one hour. The salesman recommends the top-of-the-line model and says it can cut down ten trees in an hour. The man is impressed and buys it.
The next day, he comes back with the chainsaw and complains that it only cut down one tree in an hour. He asks for a refund. The salesman takes the chainsaw and pulls the cord to start it. The chainsaw makes a loud noise and the man jumps back in shock.
He says, “What’s that noise?”
The joke is supposed to be funny because the man did not know how to use the chainsaw properly and did not realize that it needed to be turned on1. It is also ironic that he bought the most expensive model but did not get any benefit from it.
I hope this answers your question about the joke. 😊
Yes - that answers it. I put this into ChatGPT - it couldn't find the Peter Coy reference. When I removed the reference to Peter and added this "Please complete this joke" to the front - it produced the whole joke.
Looking back, I was trying to get ChatGPT to connect that without training ChatGPT is like a chainsaw, which will cut your leg off if you don't know what you're doing. Probably a stretch.
My experience is that, unless you are very specific the bots get confused. And whenever that happens, I start over.
This time I worked with ChatGPT. My first attempt was unsuccessful but twice in a row I got good answers. In my first attempt, ChatGPT was clearly hallucinating. I used the exact same prompt in all attempts.
I may have been hallucinating as well about my success in January at finding the joke.
Here is my last successful Attempt:
Model: Legacy
Mike Randolph
I am trying to recall a joke. It is about a back woodsman who comes in to a store complaining about how long it takes to cut down trees. The salesman sells him a chainsaw. He comes back a week later complaining that it doesn’t do any better and his old saw. The salesman started it up and he jumped back at the noise saying what was that. Have you heard that joke?
Yes, I am familiar with that joke. It's a classic example of a humor that is based on a misunderstanding. The backwoodsman was expecting a silent chainsaw, but the salesman explained to him that all chainsaws make noise when they are in use.
Mike Randolph
Can you tell the whole joke?
Sure! Here's the full joke:
A backwoodsman walks into a store and complains to the salesman about how long it takes him to cut down a tree with his old saw. The salesman suggests that he try using a chainsaw, which will make the job much easier and quicker. The backwoodsman is skeptical but decides to give it a try, so he buys a chainsaw and leaves the store.
A week later, the backwoodsman walks back into the store, looking even more frustrated than before. He tells the salesman that the chainsaw doesn't seem to be doing any better than his old saw. The salesman offers to start up the chainsaw to show the man how it works. When the salesman starts the chainsaw, the backwoodsman jumps back in surprise at the noise.
"What was that?" the backwoodsman exclaims.
"That's the sound of progress," the salesman replies with a smile.
Mike Randolph
What is the source of the joke?
The origin of this joke is unclear and it's possible that it has been told and retold many times over the years, with variations. This type of joke, with a backwoodsman or rural character coming into a store and being surprised by modern technology, is a common theme in American humor and has likely been a part of the oral tradition for many decades. The joke is often told as a one-liner or a short story, and it's a classic example of humor that is based on a misunderstanding or cultural gap.
Well, at least this joke is funnier. There may be a lot of versions floating out there - or the LLM may have fudged.
I tried "Write a sequel to about the movie "Hero" with Dustin Hoffman."
Chat confused the main characters and their roles. I pointed out its errors and asked it to rewrite 4 times. Each time it was willing to throw one of the characters under the bus so that the other could bask in "virtue".
Its conclusion was always along these lines:
"The sequel to "Hero" would be a heartwarming and inspirational story of redemption and the power of selflessness. It would show that anyone can be a hero, and that even small acts of kindness can have a big impact on the world around us."
Who knows, maybe the movie industry has had access to ChatGPT longer than anyone will admit.
There are some basic mathematical errors in the paper you cite: by Felten, Raj and Seamans (March 2023). Check out the equation A_ij = sum{x_ij : i = 1,...,10} and it goes on. This is very undercooked research folks.
Another implication not noted here (perhaps not part of the study, which I have yet to read), is that by putting the lower skilled workers on quasi-par with veterans, it devalues the latter's expertise. When this narrowing of skills gap translates into a narrowing of wage gap, which direction do you think wages will go? That's the obvious huge issue, which unfortunately I find is seriously not given enough attention to.
Similar to the miners taken as example here, we can look at taxi drivers, whose legendary knowledge of the city no longer has any value. As a result, overall wages have dropped 10% for London cabbies (as per research conducted by Carl Frey).
In short, no, AI does not automate work, a human is still necessary in the loop for now. But just about any human can now do the job, hence it will go to the lowest bidder, and that could get very low. Surprised or even disgusted by the lack of interest in this obvious fact, I woite about this in my book "AI Battle Royale: how to protect your job from disruption in the 4th Industrial Revolution".
You make a good point about people being dishonest about using AI and I think the more these programs automate jobs, the the lazier and dishonest people will become. Humans always look for a way to just fake the metrics but absorb the merited credit.
I love the comparison to the productivity increase brought about by steam power. Something that was alluded to but not specifically mentioned was the impact on KNOWLEDGE WORKERS. Now we have AI, knowledge flows freely across all industries. It was the information age that brought about the knowledge worker and now that we move into the knowledge age, the optimist in me hopes that we see the dawn of the Wisdom worker.
"The key is to learn fast." True.
ChatGPT = Motorbike https://www.netizen.page/2023/03/chatgpt-motorbike.html
does this drive a new era of equality in society where work that needs to be done face to face starts to gain more value and remote/ knowledge work reduces in value... Maybe carers and plumbers start getting paid more or maybe we need to think about universal wages to level out the impact?
I am dictating this to my iPad, so please forgive any mistakes.
Here’s a prompt I just put into Bing:
I have a question about a joke. It concerns a backwoodsman that bought a chainsaw and didn't know how to use it when he returned it the store owner pull the card when it made noise. He jumped back and asked what's that noise if you heard that one it was in a Peter Coy column in the New York Times
Also dictated, I corrected the spelling of Coy.
It did search the Internet
Here’s your response:
I think I have heard a similar joke before. It goes like this:
A man goes to a hardware store and asks for a chainsaw that can cut down six trees in one hour. The salesman recommends the top-of-the-line model and says it can cut down ten trees in an hour. The man is impressed and buys it.
The next day, he comes back with the chainsaw and complains that it only cut down one tree in an hour. He asks for a refund. The salesman takes the chainsaw and pulls the cord to start it. The chainsaw makes a loud noise and the man jumps back in shock.
He says, “What’s that noise?”
The joke is supposed to be funny because the man did not know how to use the chainsaw properly and did not realize that it needed to be turned on1. It is also ironic that he bought the most expensive model but did not get any benefit from it.
I hope this answers your question about the joke. 😊
If these figures are accurate, we should start seeing an effect in GDP data soon. If we don't, we can conversely surmise the figures are not accurate.
Love the blog! I just wanted to point out a *slight* error:
“A study of programmers found a increase of 55.8% in productivity when using AI”
The tasks took the teams 161 minutes vs 71 minutes on average. That represents a productivity *increase* of (161 / 71) - 1, or roughly +127% compared to baseline, which is to say that developers using copilot performed the task *~2.27x* faster than those without it.
Another great analysis, thank you. My hope is that it becomes a badge of honor (oh, how many times I've suggested to an organization that they reward bottom-up change). However, I'd give people a font-spinning, multi-colored, flashing, badge if they also shared the other changes they made to make the AI use work in their setting. A slightly less flashy badge will go to whoever at APA, MLA, etc. helps us know how to effectively cite our new (and old) collaborators. https://terrigriffith.com/blog/t-is-for-talent
We have been warned by the brilliant Tim Urban about the mindboggling capabilities of soon-to-arrive AI back in 2015: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html
11/10 recommendation to read his blog for insights and laughs about AI.