The Machines of Mastery

"Anyone can learn anything they want..." and how technology can help

Look at these graphs from a new paper tracking the test results of 7,000 learners across a wide variety of subjects. Each line shows the test scores of a student practicing a skill over a number of practice attempts (“opportunities”). You will notice something that even the paper’s authors found surprising.

Almost all of these lines have the exact same slope! That means that every student, regardless of their starting expertise or their rank within a class, gains roughly the same amount of skill and knowledge from practicing. In fact, the average student needs to practice seven times in the average subject to achieve a “reasonable level of mastery”¹. Students who start out behind can catch up by practicing more, and more advanced students need to practice less, but everyone gets almost the same benefit from practice.

The paper concludes, excitingly: “Our evidence suggests that given favorable learning conditions for deliberate practice and given the learner invests effort in sufficient learning opportunities, indeed, anyone can learn anything they want."

Note that the key words here are deliberate practice, a technique expounded on by the late Professor Anders Ericsson about how to achieve expertise. You may have encountered this concept in Malcolm Gladwell’s 10,000 hour rule (it takes 10,000 hours to make an expert). As many people, including Ericsson, pointed out - it is not the number of hours that you spend, but the fact that you spend them in deliberate practice, that builds expertise. But these new findings go further: we know that deliberate practice seems to work equally well for everyone at the same rate, regardless of subject or ability, a fascinating development that makes this technique even more important.

The pain and power of deliberate practice

Deliberate practice is powerful, but it also can be hard. In fact, it has to be. Unlike passive forms of learning, it requires a proactive and intentional approach to skill development. This process begins with breaking down a skill into its constituent parts, and building each part one step at a time. It is a gradual and systematic approach that ensures that each new skill is built upon a solid foundation of existing knowledge.

For deliberate practice to be truly effective, learners must bring their full focus and engagement to the task at hand. This can be challenging, as the work required is often difficult and demanding. Failure is common. In fact, one of the key aspects of deliberate practice is that it embraces failure as an opportunity for growth. When learners encounter obstacles or make mistakes, they are encouraged to reflect on what went wrong and what they can do differently next time. This process of continuous improvement is crucial to making lasting progress.

Like exercise, deliberate practice is designed to always be challenging, pushing learners out of their comfort zone and forcing them to stretch their abilities. This may feel difficult at first, but over time, learners develop a growth mindset and become more confident in their abilities. And deliberate practice benefits greatly from ongoing, personalized feedback from an expert instructor. This feedback helps learners identify their strengths and weaknesses, and provides actionable insights into how they can continue to improve.

Well-designed practice has to have all of these elements: a step-by-step approach, sustained engagement, a chance to make meaningful mistakes, continually increasing challenges, and ongoing feedback. Opportunities for deliberate practice are relatively easy to find in fields that have a lot of students: sports, math, musical instruments, and learning foreign languages, to name a few. In each case, there are many trained teachers, with good lesson plans, easy access to practice, and well-thought out approaches. But in most jobs, ranging from entrepreneurship to leadership, there are few opportunities to practice until you actually start doing things. And that can be a problem.

But here is where technology can help.

Deliberate practice at scale: simulations and games

Military trainers, as well as Tom Cruise fans, have long had a powerful example of the value of games and simulations for deliberate practice. The story begins in Vietnam, where, as the Vietnam War started, American fighter pilots were feeling extremely confident, and for good reason. During the war in Korea, American pilots had performed brilliantly against North Korean, Chinese, and Russian pilots in the first all-jet dogfights in history, achieving an exchange ratio of 10 to 1. That is, for every American plane shot down, the Americans managed to destroy ten enemy planes. Vietnam, however, turned into a very different experience. Due to a combination of capable pilots, shifting tactics, and changes in aircraft, the Vietnamese were much more successful against their American foes. By 1968, the exchange ratio had fallen to 3.7 to 1.

The fall of 1968 was marked by the beginning of a three-year halt in the bombing campaign over North Vietnam, and thus began a time when American pilots did not generally fly combat missions. In that year, the Navy commissioned a report to examine the relative failures of the air war of Vietnam, which identified training as a key gap. From this report, the Naval Fighter Weapons School, better known by movie fans as Top Gun. Top Gun changed the concept of pilot training, turning it into a competitive game, a simulation with real fighter jets between trainees and instructors who flew Vietnamese planes using Vietnamese tactics. All built around the concept of deliberate practice. And, while the Navy put its pilots through Top Gun, the Air Force also reviewed their performance in Vietnam. However, they ignored the training problem, choosing instead to invest in better missiles, improved planes, and high-technology cannons.

Top Gun

The two different strategies acted as a natural experiment: what would win, deliberate practice or better equipment? When the air war resumed in 1972, Top Gun-trained pilots were in every Navy unit. But they were still flying the same aircraft, while the Air Force had significantly improved their planes. Despite the inferior equipment, Navy’s exchange ratio went up to 13 to 1, while the Air Force’s briefly plunged to an even 1 to 1 exchange. It turns out that deliberate practice in simulated sessions really does help.

This same technique can be applied to other forms of training. In fact, it needs to be. There are so many critical life skills that we do not train nearly enough. Most people work in teams, but we don’t train working with teams through deliberate practice. Leadership matters in every human endeavor, but we don’t train leadership through deliberate practice. Entrepreneurship can be taught, but we don’t train it through deliberate practice.

But we can.

As someone who has long worked at the intersection of games and teaching, I know we can solve the deliberate practice problem at scale, because I co-founded Wharton Interactive a few years ago. We took on the mission of democratizing access to business education through simulations and games. To do that, we built a “Top Gun for business” where students run fake companies in real time using simulated tools (we built simulated versions of email, Slack, and Zoom into our games) experiencing deliberate practice. Students go through a learning loop, being taught how to do something, and then immediately deciding how to apply that skill, getting detailed feedback as a result. If players do well (say, mastering a delicate client negotiation), the next negotiation is harder, otherwise, the difficulty drops to keep the challenge level appropriate

This isn’t theoretical: you can play a free game that teaches you entrepreneurial mindsets right now, or run an entire entrepreneurship-class-in-a-game We also have games that teach leadership on a doomed mission to Saturn, and ones that teach you to analyze real data in Python, among others. With tens of thousands of players, and early evidence that suggests powerful outcomes, I think these simulations hint at a future where high-quality deliberate practice is delivered at scale through dedicated gaming platforms.

But we still need to carefully build each simulation, which requires a team of game designers, programmers, experts, and others. This means that we can only create a limited number of simulations, leaving many skills without easy ways to practice.

That is where AI comes in.

Deliberate practice for all: AI

Even in its current form, ChatGPT is shockingly close to being able to help anyone, anywhere learn via deliberate practice.

As an experiment, lets say I want to learn negotiation skills. I gave ChatGPT this prompt: I want to do deliberate practice about how to conduct negotiations. You will be my negotiation teacher. You will simulate a detailed scenario in which I have to engage in a negotiation. You will fill the role of one party, I will fill the role of the other. You will ask for my response to in each step of the scenario and wait until you receive it. After getting my response, you will give me details of what the other party does and says. You will grade my response and give me detailed feedback about what to do better using the science of negotiation. You will give me a harder scenario if I do well, and an easier one if I fail.

Here’s what happened next:

That is a remarkably good job for a first attempt with a general AI. It successfully simulated the step-by-step approach, meaningful errors, and adaptive feedback necessary for deliberate practice. Of course, the known issues of ChatGPT and other LLMs, including its tendency to lie, make me nervous about trusting it too much, as does the fact that scenarios are not repeatable. Every time I “run” the simulation, the experience can be radically different. But both of these problems are solvable, and these early results point the way to something extraordinary.

If deliberate practice means “anyone can learn anything they want” - then a machine that can create meaningful deliberate practice, on demand, is a machine that can help humans achieve mastery. Far from being skill-destroying, AI can be skill-building at a scale that we had never thought possible.

1. Note that “a reasonable level of mastery” means a student has 80% scores on a subject. That is quite different from our conception of mastery as being the best of the best. Innate skills and talents play an increasingly large role as you consider the top performers. At amateur levels in sports, for example, practice accounts for 10-30% of the difference in performance, quite a large proportion, but at the most elite level, it is only 1%.