26 Comments

My takeaway is that we're paying an awful lot of people an awful lot of money to do very low-value things (which is mostly what current AI can do). AI can (help) write a webserver because lots of webservers have already been written -- why would you be paying someone to write yet another one? Ask it to write some truly novel code and see what you get

Expand full comment

my takeaway here is that 20%+ of developers today can't even create a web server, despite having ChatGPT

Expand full comment

AI tools are certainly having an impact on productivity. I've been using it to boost my coding speed and it's been great.

But I'm not completely convinced that the stats in these papers are useful.

The Github Copilot study was tested with a task that almost no one will ever do in a real world situation, so it makes sense that the developers needed more time to work on it. There are likely some examples of a simple HTTP server on Github, so Copilot unsurprisingly knew how to do it. If we tried giving Copilot a problem that people actually solve at their companies, which requires specific knowledge of that company's codebase, I think the results would be significantly different.

The ChatGPT productivity study doesn't seem to have released the actual tasks done in the study, but in their own words "These comments point to an important (and inherent) limitation of our experiment: it involves relatively small, self-contained tasks that lack much context-specific knowledge beyond what we stipulate in the task prompts." Just like the Copilot example, whatever tasks they used were specifically optimized to be tasks that ChatGPT could do, and doesn't necessarily reflect real world tasks.

They do follow that up with "However, our core result, that ChatGPT can

increase productivity on many mid-level professional writing tasks, is supported by the fact

that many respondents choose to use it in their real jobs.", but that doesn't really give us stats on how productive it's making people in the real world. Perhaps it's just making people *feel* more productive and cool, and that's why they're using it at their jobs.

Expand full comment

Pretty wide CI on the study results, but, yeah, increasing time efficiency for using the vast arsenal of canned heat makes sense.

We need, however, to consider the difference between “efficiency” (doing things right) and “effectiveness” (doing the right things). Frustration, not innovation, is the mother of invention. By greasing the skids on the same old same old, AI, might also remove the friction leading to the “there must be a better way!” driven breakthroughs l

Expand full comment

Thank you for this post - it drives home a critical point - ChatGPT is a tool.

The question is - how long will it take to learn to use the tool correctly. As CK pointed out - you better "read" the results when you generate an apology letter (unless the reality is that you really don't care).

This is a little like walking into a wood-working shop - seeing a rack of bright shiny wood gouges next to a beautiful, polished wood bowl - and thinking you could do that.... Yep, you can..... (I'm using the ChatGPT autofill technique here)

With that in mind - I submitted this prompt to Chat:

"Consider that ChatGPT is a tool. A chainsaw is also a tool. Compare ChatGPT with a chainsaw."

Chat produced 3 paragraphs and a summary . But never got the point.

I'm reminded of the old story about a prison that had a list of jokes. Each joke had a number. The prisoners had heard the jokes so often that in conversation they would just call out the number. The other prisoners would break out in laughter.

Try prompting that joke line to Chat and see if you aren't ROFLOL at the response.

Expand full comment

There are some basic mathematical errors in the paper you cite: by Felten, Raj and Seamans (March 2023). Check out the equation A_ij = sum{x_ij : i = 1,...,10} and it goes on. This is very undercooked research folks.

Expand full comment

Another implication not noted here (perhaps not part of the study, which I have yet to read), is that by putting the lower skilled workers on quasi-par with veterans, it devalues the latter's expertise. When this narrowing of skills gap translates into a narrowing of wage gap, which direction do you think wages will go? That's the obvious huge issue, which unfortunately I find is seriously not given enough attention to.

Similar to the miners taken as example here, we can look at taxi drivers, whose legendary knowledge of the city no longer has any value. As a result, overall wages have dropped 10% for London cabbies (as per research conducted by Carl Frey).

In short, no, AI does not automate work, a human is still necessary in the loop for now. But just about any human can now do the job, hence it will go to the lowest bidder, and that could get very low. Surprised or even disgusted by the lack of interest in this obvious fact, I woite about this in my book "AI Battle Royale: how to protect your job from disruption in the 4th Industrial Revolution".

Expand full comment

You make a good point about people being dishonest about using AI and I think the more these programs automate jobs, the the lazier and dishonest people will become. Humans always look for a way to just fake the metrics but absorb the merited credit.

Expand full comment

I love the comparison to the productivity increase brought about by steam power. Something that was alluded to but not specifically mentioned was the impact on KNOWLEDGE WORKERS. Now we have AI, knowledge flows freely across all industries. It was the information age that brought about the knowledge worker and now that we move into the knowledge age, the optimist in me hopes that we see the dawn of the Wisdom worker.

Expand full comment

does this drive a new era of equality in society where work that needs to be done face to face starts to gain more value and remote/ knowledge work reduces in value... Maybe carers and plumbers start getting paid more or maybe we need to think about universal wages to level out the impact?

Expand full comment

I am dictating this to my iPad, so please forgive any mistakes.

Here’s a prompt I just put into Bing:

I have a question about a joke. It concerns a backwoodsman that bought a chainsaw and didn't know how to use it when he returned it the store owner pull the card when it made noise. He jumped back and asked what's that noise if you heard that one it was in a Peter Coy column in the New York Times

Also dictated, I corrected the spelling of Coy.

It did search the Internet

Here’s your response:

I think I have heard a similar joke before. It goes like this:

A man goes to a hardware store and asks for a chainsaw that can cut down six trees in one hour. The salesman recommends the top-of-the-line model and says it can cut down ten trees in an hour. The man is impressed and buys it.

The next day, he comes back with the chainsaw and complains that it only cut down one tree in an hour. He asks for a refund. The salesman takes the chainsaw and pulls the cord to start it. The chainsaw makes a loud noise and the man jumps back in shock.

He says, “What’s that noise?”

The joke is supposed to be funny because the man did not know how to use the chainsaw properly and did not realize that it needed to be turned on1. It is also ironic that he bought the most expensive model but did not get any benefit from it.

I hope this answers your question about the joke. 😊

Expand full comment

If these figures are accurate, we should start seeing an effect in GDP data soon. If we don't, we can conversely surmise the figures are not accurate.

Expand full comment

Love the blog! I just wanted to point out a *slight* error:

“A study of programmers found a increase of 55.8% in productivity when using AI”

The tasks took the teams 161 minutes vs 71 minutes on average. That represents a productivity *increase* of (161 / 71) - 1, or roughly +127% compared to baseline, which is to say that developers using copilot performed the task *~2.27x* faster than those without it.

Expand full comment

Another great analysis, thank you. My hope is that it becomes a badge of honor (oh, how many times I've suggested to an organization that they reward bottom-up change). However, I'd give people a font-spinning, multi-colored, flashing, badge if they also shared the other changes they made to make the AI use work in their setting. A slightly less flashy badge will go to whoever at APA, MLA, etc. helps us know how to effectively cite our new (and old) collaborators. https://terrigriffith.com/blog/t-is-for-talent

Expand full comment

We have been warned by the brilliant Tim Urban about the mindboggling capabilities of soon-to-arrive AI back in 2015: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html

11/10 recommendation to read his blog for insights and laughs about AI.

Expand full comment