And the great gears begin to turn again...

Progress in two vital areas had slowed, AI could change that

With powerful cellphones connected to all the information in the world, you would think we would be in a scientific golden age, but, in reality, the march of progress has been stalling. The two engines of economic growth — productivity and innovation — have been mysteriously grinding to halt in recent decades, with long-term consequences for all of us. But the rise of useful AI may be exactly what is needed to get the machine working again. And even at this early date, it is possible to hear the creaking of the great gears of progress starting to turn, as AI offers new solution to otherwise intractable problems.

Science and Innovation

Innovation has been slowing alarmingly. In fact, a recent, convincing, and depressing paper found that the pace of invention is dropping in every field, from agriculture to cancer research. More researchers are required to advance the state of the art. In fact, the speed of innovation appears to be dropping by 50% every 13 years. The paper ends: “to sustain constant growth in GDP per person, the US must double the amount of research effort every 13 years to offset the increased difficulty of finding new ideas.

Medical research progress has slowed even as the number of researchers has increased

Part of the issue appears to be a growing problem with scientific research itself: there is too much of it. The burden of knowledge is increasing, in that there is too much to know before a new scientist has enough expertise to start doing research themselves. That appears to be why the age at which scientists or inventors achieve their moment of genius increasing. Half of all pioneering contributions in science now happen after age 40, but it used to be younger scientists who achieved breakthroughs. To see the burden of knowledge at work, consider how much there is to learn about the existing state of even a narrow topic in science, like human metabolic pathways, as illustrated below (high quality version here).

Part of why science is slowing is that there is a lot you need to know to make a contribution, as this chart of metabolic pathways shows!

Similarly, startup rates of STEM PhDs are down 38% in the last 20 years. This paper finds that a reason, again, is the burden of knowledge: the nature of science is growing so complex that PhD founders now need large teams and administrative support to make progress, so they go to big firms instead!

Thus, we have the paradox of our Golden Age of science, as illustrated in this paper. More research is being published by more scientists than ever, but the result is actually slowing progress! With too much to read and absorb, papers in more crowded fields are citing new work less, and canonizing highly-cited articles more. The paper ominously concludes: “These findings suggest troubling implications…. If too many papers are published in short order, new ideas cannot be carefully considered against old, and processes of cumulative advantage cannot work to select valuable innovations.”

But there are already signs that AI can help (even though some of the early efforts to summarize research data have resulted in backlash due to the way they were implemented). A new paper has successfully demonstrated that it is possible to correctly determine the most promising directions in science by analyzing past papers with AI, ideally combining human filtering with the AI software. And other work has found that AI shows considerable promise in performing literature reviews.

While specialized tools will likely do much more to advance the use of AI in research, many academics are already experimenting with AI as a way to automate tasks. In fact, part of the reason I am confident that AI has power as summation and analysis tool is that I am using it myself. For example, I trusted ChatGPT to generate a draft abstract to a new whitepaper (which, any academic will tell you, is a surprisingly annoying task). After feeding it the entire first part of our new paper, it did a very solid first pass and summarizing the important points.

Productivity

While the slowdown in science is worrying, there have been few trends as frustrating as the long-term slowdown in productivity. Productivity, the amount of output per worker in a given period of time, is the ultimate driver of progress - it lets us create more stuff in less time. This both boosts the economy and our quality of life. In 1865, the average British worker worked 124,000 hours over their life (similar to the US & Japan). By 1980, the average British worker put in only 69,000 hours of work over their life, despite living longer with a higher standard of living. We went from spending 50% of our lives working to 20%, and got richer besides, thanks to productivity growth.

But since 1980, the rate of productivity gains has stalled, becoming especially bad recently. Despite the rise of software and internet access, the last decade has seen incredibly slow growth in productivity. Those numbers were briefly reversed during a productivity boom during the pandemic, but have fallen back to an even slower long-term trajectory this year. And, frustratingly, no one has a clear explanation as to the root cause.

Even without knowing the cause, there is something that might change the trajectory of productivity growth, and, if you have been reading this Substack, you will already guess that I am referring to generative AI. While the full dimensions of the value of this sort of AI system are unclear, early indications are that it can act as a shortcut for many different kinds of tasks. Many aspects of white collar and managerial work have been unchanged for years, involving lots of time on Excel or Powerpoint or Micosoft Word creating reports and documents, using tools that have been fundamentally unchanged for decades. AI offers the possibility of changing the nature of this work in a widespread and lasting way. Documents can be prepared quickly, designs can be rapidly prototyped, slide decks can be generated with a few clicks, and more. The old constraints on managerial work might change very quickly.

It is very early to say whether this is enough to break the productivity slowdown, but our clearest sign that it might is in programming. AI tools have used extensively in programming over the past year, and it seems to make a big difference. In a recent randomized controlled trial of Copilot, an AI programming companion (conducted by Microsoft, who owns the tool), using AI cut the time required to complete a programming task in half.

We are still very much at the start of any AI revolution, but early signs are encouraging. I am especially excited by the fact that AI helps with the pain points associated with the twin slowdowns in science (too much information) and productivity (too many analytical writing and programming tasks have been untouched by technological advances). If AI does prove to be helpful in innovation and productivity, it would help restart and sustain the engines of global growth, which would be good news for all of us.