Everyone is above average
Is AI a Leveler, King Maker, or Escalator?
Congratulations - you are now above average!
It may sound like an old, bad statistics joke, but I mean it quite literally. We now have very strong evidence that AI elevates the skills of the lowest performers across a wide range of fields to, or even far above, what was previously average performance.
This is a big deal because skill gaps among humans can be quite large. In many fields, the difference between top and bottom performers is profound, including programmers, middle managers, and radiologists, among many others. If you are able to find, train, and retain these top workers, you get tremendous benefits. A large part of schooling and work is focused on getting people to this highly skilled state.
However, people who are good at one skill may not be good at another - the top-performing salespeople are usually not also top-performing sales managers. Modern professional work consists of a wide range of activities, rather than a single specialization. For example, the job of a doctor may require many tasks, like diagnosing patients, providing treatment, offering advice, filling out expense reports, and overseeing the office staff. It is unlikely that any doctor is equally good at all of these tasks. Even the best workers have weak spots, requiring that they be part of larger organizations to ensure they can focus on what they do best.
Or they did have weak spots.
AI as Leveler
We already know one major effect of AI on the skills distribution: AI acts as a skills leveler for a huge range of professional work. If you were in the bottom half of the skill distribution for writing, idea generation, analyses, or any of a number of other professional tasks, you will likely find that, with the help of AI, you have become quite good.
The new paper that I, along with Fabrizio Dell’Acqua, Edward McFowland III, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim Lakhani, just released shows this works even for the highly trained knowledge workers at Boston Consulting Group. Across a set of 18 tasks designed to test a range of business skills - from analysis to idea generation to persuasion - consultants who had previously tested in the lower half of the group increased the quality of their outputs by 43% with AI help while the top half only gained 17%. Where previously the gap between the average performances of top and bottom performers was 22%, it shrunk to a mere 4% once the consultants used GPT-4.
We aren’t alone in this type of finding; a lot of work has been done on how AI levels different kinds of writing tasks. Another study, this one from Noy and Zhang of MIT, finds that half the performance gap between good and bad writers is erased when participants use ChatGPT. The same holds true for creative writing, as Doshi and Hauser find. They measured writers on creativity and then gave some of them access to AI generated story ideas. Again, skill differences were levelled, being given 5 ideas by GPT-4 “effectively equalizes the creativity scores across less and more creative writers.”
Beyond writing, LLMs appear to help lower performing specialists become better as well. Choi and Schwarcz gave law students access to GPT-4 and found that law students near the bottom of their class using AI equalized their performance with folks at the top of the class (who actually saw a slight decline when using AI). They conclude, “This suggests that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers.” And, in a real-world experiment, Brynjolfsson, Li, and Raymond discovered that LLMs helped customer service agents in the bottom quintile catch up with higher performers, with much smaller effects for better workers.
Taken together, the results are clear: AI acts as a leveler, raising everyone to a minimum level of performance. In some ways this may not be surprising - automation has always increased the bar for productivity. It used to matter whether workers were fast or slow at digging a ditch or planting seeds, but now a machine does that work much faster, no matter how good your mere human digging or planting skills are. But just as automation reshaped manual labor, it might do the same for intellectual work. The most extreme vision of this levelling effect was something I heard when I was on a panel with a CEO who had been experimenting with GPT-4. To audible gasps he said, “Most of our employees are engineers and we have a few hundred of them… and I think in 18 months we will need 20% of them, and we can start hiring them out of high school rather than four-year colleges. Same for sales and marketing functions.”
I think this prediction for the collapse of skilled work is wrong, however. As we discussed, jobs don’t consist of just one automatable task, but rather a set of complex tasks that still require human judgement. Plus, because AI has a “jagged frontier” (it is not good at everything, but rather has uneven abilities) it is unlikely to do every task that a worker is responsible for. Improving the performance in a few areas need not lead to replacement, but instead it will free up time to do more valuable, satisfying, and productive work. And this also assumes that levelling is the only effect of AI.
AI as Escalator, AI as Kingmaker
Just because early results for AI suggest that only lower performing people benefit does not mean that this is the only possible pattern. It may be that the reason only lower performers gain from AI currently is because the current AI systems are not good enough to help top performers. Or, alternately, it might be that top performers need more training and work to get benefits from AI. If either of these conditions prove true, and they certainly seem plausible, then AI might act more as an escalator, increasing the skills for everyone, from top to bottom performers. After an adjustment period, the relative skill positions stay similar, but everyone gets more done, faster.
Alternately, it might be that some people are just really good at working with AI. They can adopt Cyborg practices better than others and have a natural (or learned) gift for working with LLM systems. For them, AI is a huge blessing that changes their place in work and society. Other people may get a small gain from these systems, but these new Kings and Queens of AI get orders of magnitude improvements. If this scenario is true, they would be the new stars of our AI age, and are sought out by every company and institution, the way other top performers are recruited today.
Change is inevitable
We don’t know the ultimate shape of the new post-AI skills distribution, but we do absolutely know that things are changing. Even with the relatively primitive tools of our current, unspecialized AI systems, it is clear that we can become much more productive, and that less-skilled workers are now at much less of a disadvantage than they used to be.
Like so much else about AIs, the implications here are not completely clear, but we have agency over what happens next. Some of the highest-paid jobs are most impacted by AI - so what do companies do in response? Hire less skilled workers and have them boosted by AI? Expect more work out of all their employees? Focus on working with employees so that they become Cyborgs? Or are they tempted to cut wages or headcount? The answers to these questions are critical, and they will be made soon by companies, with influence from government, their employees, their stakeholders, and their customers. We should be thinking right now about how we want this new world of work to look.
And it isn’t just work. If AI turns poor employees into good ones, it might do the same for other fields. I suspect that it will have a major effect on entrepreneurship, giving every entrepreneur a personalized cofounder to fill in the gaps in their skills. This suggests a potential for an expansion of entrepreneurship, especially given that GPT-4 is widely available for free around the world, helping potential founders far from the usual technology hubs. However, without guardrails, AI may also do the same for criminals, increasing the competence of the worst actors in society.
A lot is changing, quickly. For the first time, we have a broad-based tool that boosts human intellectual abilities, but we are still limited to human wisdom. We will need to draw on that to make good choices even as change continues to accelerate.
Something I’ve discovered about myself after using AI to help me write and prep for interviews is that it’s operating much like GPS has for me. I used to use maps and learned to memorize streets and addresses (similarly with phone numbers), but now as I use GPS for navigating ALL THE TIME, I have no idea where anything is. I don’t remember what freeway number is my turnoff or what’s on the corner that cued my turn before. I’ve become dependent and MUST use it now. Same with my writing...I’ve become lazy about even trying to write anything. I look at notes as I give interviews rather than relying on my own knowledge of my history.
So this is the danger. We become dependent and lazy, using the tools to do all our work and probably making us all more similar and far less creative.
Thanks for the great read, Ethan :)
I believe both elevator and kingmakers models are simultaneously true, personally.
GPT-4 is an intellectual tool, but since that is a bit more abstract, let me compare it to a physical tool for a second - a chainsaw.
Prior to the invention of the chainsaw, when high-skill (HS) and low-skill (LS) axe-wielders wanted to chop down a tree, they had to bring a lot more to bear: the angle of the swing, the grip of the heft, optimal pacing to make decent progress without burning through their energetic resources. In other words, a mix of knowledge and muscle memory, a set of skills.
After the chainsaw's invention, HS axe-wielders learned that some of their strategies were transferable to slightly increase their effectiveness, but overall, the benefit was just in the job getting done more easily. For LS woodcutters, the chainsaw was a godsend. Most of the challenging aspects they lacked skills for were essentially automated away by the steady chug of the engine.
But on the whole, both HS and LS axe wielders became LS chainsaw wielders. What does a HS chainsaw wielder look like? Just google "chainsaw sculpture" for an idea of what's possible. Or "chainsaw ice sculpture" for a twist.
It's easy to pick up a chainsaw and learn the basic uses of the tool to get a quick benefit from it, but harder to master its use on challenging, even previously impossible challenges. Please correct me if I'm wrong, Ethan, but I don't suppose many of the high-performers in this study had specifically studied how to leverage the tools even more effectively (i.e. prompt engineering).
I'd be very interested in the results of a follow-up study comparing the gains from AI-assisted professionals against a similar group that receives a couple of weeks of instruction in prompt engineering.
---
Anyway that's my whole elevator argument. As for the kingmaker scenario, I think that's reserved for "artists" - the michelin-chef level executives who aren't just interested in productivity, but passionate with a dual expertise in their subject matter AND working with LLMs.