There is still plenty of low-hanging fruit in science

The case for scientific optimism.

The most common explanation for the slowdown in scientific progress is that we’re running out of low-hanging fruit.

Most easy discoveries have already been made. What’s left is increasingly complex and requires steadily increasing investments in equipment, personnel, and research infrastructure.

For example, physicist Leo Kadanoff states:¹

"The truth is, there is nothing — there is nothing— of the same order of magnitude as the accomplishments of the invention of quantum mechanics or of the double helix or of relativity. Just nothing like that has happened in the last few decades. [...] Once you have proven that the world is lawful to the satisfaction of many human beings, you can’t do that again."

It’s a plausible explanation. It’s definitely a convenient one.

If the slowdown is inevitable, there is no one to blame and everyone can just keep going without making any changes.

But I think it’s completely wrong.

The idea that there are few easy discoveries that still can be made hinges on a faulty mental model of knowledge and a misleading interpretation of data.

First of all, knowledge is not finite but fractal. The idea that discoveries necessarily get harder over time because we’ve already explored all easily accessible areas is incorrect. Each new discovery opens up vast new areas, each bringing its own set of low-hanging fruit.

Secondly, scientific discoveries are rarely made as soon as they’re theoretically possible. Hence, it’s nonsense to conclude that data showing a lack of progress despite increased investments implies we’ve run out of low-hanging fruit. There are plenty of alternative interpretations that are just as plausible.

Let’s talk about these two points one after another.

Knowledge is fractal, not finite

There is a popular mental model comparing scientific progress to the exploration of a new continent.

For the first explorers, it was easy to make significant discoveries because everything was unexplored.

But as explorers gradually fill in all the blank spots on the map, it becomes increasingly harder to make significant discoveries.

You have to go to ever more remote areas, climb ever higher mountains, or dive ever deeper canyons.

Nowadays, our map is nearly complete.

Only highly trained explorers with huge amounts of funding for expensive equipment are able to discover anything at all.

This model is fundamentally flawed.

Knowledge isn't like a finite landmass waiting to be mapped - it's more like a fractal pattern that reveals new areas the closer you look.

Each discovery doesn't just fill in a blank spot, it opens up entirely new territories of investigation.

If anything, the blank spots are only getting larger as our scientific understanding expands.

Maxwell’s theory of electrodynamics filled out a huge blank spot in our understanding of physics, but it also contained little bridges to a vast new areas called Special Relativity and Quantum Field Theory.

Our understanding of fundamental physics has simultaneously never been greater and more incomplete.

A century ago, people were pretty sure that the fundamental constituents of matter were atoms. Then we discovered protons, neutrons, and electrons. This led to a whole zoo of “elementary particles”.

Nowadays, we aren’t even sure whether “there are no particles, there are only fields” or only particles and no fields.

Unlike a century ago, there is no coherent framework describing all known fundamental “forces.” This hints at a giant new framework waiting to be discovered.

94% of the universe’s energy content is “dark stuff.” Aptly named because we currently have absolutely no idea what it is.

What once seemed like a fairly well-understood smooth territory has turned into a highly fragmented landscape of poorly understood areas.²

And yet newly discovered theories aren’t necessarily getting more complicated. There are crazy complicated things in Classical Mechanics just as there are crazy complicated stuff in Quantum Field Theory. But at their heart, both theories are similarly simple.

So I don’t see any reason why the fruit in not yet explored territories should be hanging any higher.

And that’s, of course, just fundamental science.

The expansion of areas that can be explored by scientists is even more obvious when we talk about applied science.

Quantum mechanics alone spawned dozens of new fields like quantum computing and quantum cryptography.

Each newly discovered area in the landscape of scientific knowledge contains its own set of unexplored territories and low-hanging fruit.

In addition, if anything, discoveries should become easier thanks to technological progress.

Nowadays, every human with a smartphone has access to all of humanity's accumulated knowledge. With just a few clicks you can access virtually any paper and book ever written. You can run giant simulations and complex calculations on laptops that cost just a few hundred dollars. Anyone willing to invest $20/month now has access to their own personal Marcel Grossman to point them in the right direction whenever they get stuck.

So if it was really true that discoveries are rarer because they are getting harder to find, we should have seen an avalanche of progress as a result of this technological progress.

To stick to our analogy, think of explorers who complained for decades that they couldn't make major discoveries because they lacked proper resources to reach remote areas. Then, suddenly, they got access to drones, advanced mapping technology, and satellite imagery – and they still failed to make any significant discoveries.

This would clearly suggest that it wasn’t difficulties in reaching remote areas that was holding them back in the first place.

Hindsight Bias

Throughout human history, it always took shockingly long to make “obvious” discoveries.

It took thousands of years after the invention of the wheel before someone thought to attach wheels to luggage or before someone invented the bicycle.

Quantum mechanics could have been discovered much earlier by someone attempting to generalize probability theory.

It took thousands of years before someone dared to explore geometry without Euclid's fifth postulate seriously.

Fruit picked in the past always seems a lot more low-hanging than it really was.

Many calculations that are nowadays done by students in a few hours took the first researchers who did them many weeks, if not months.

Knowing that there is a solution, that it's worthwhile to carry out the calculation at all, and a few hints how to approach it go a long way.

Hindsight bias is a well-documented psychological phenomenon.

In one famous experiment, Daphna Baratz (1983) gave college students pairs of "discoveries", one true (for example, "In prosperous times people spend a larger proportion of their income than during a recession" or "People who go to church regularly tend to have more children than people who go to church infrequently"), the other its opposite ("In recession times people spend a larger proportion of their income than during prosperity" and "People who go to church infrequently tend to have more children than people who go to church regularly.")

The result? Whether given the true discovery or its opposite, most students rated it as something “I would have predicted.”

So it’s very much possible that there are plenty of discoveries waiting to be made right now that will seem perfectly obvious in a few decades.

Reality of scientific progress

A second reason why most discoveries were made much later than you would have expected is that progress is not a straightforward process.

Societal factors play a huge role.

This is illustrated, for example, by Max Planck’s observation that “a new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it”.

In other words, “science progresses one funeral at a time”.

If scientific progress was a straightforward, rational process, new theories should be widely accepted as soon as sufficient evidence is presented. Instead, established scientists often resist new ideas, regardless of their merit.

There is a fundamental inertia that makes scientific progress slower than it could be.

Just consider how long it took for the mathematical concept of zero to be accepted.³

A more optimistic version of Planck’s principle is Graubard’s Principle: Science progresses one birth at a time.

Often we have to wait for the right person to enter the field with a healthy dose of naivety and unwarranted confidence before the next breakthrough can be achieved.

The following old joke illustrates this nicely:

A finance professor and a student are walking across campus. The student sees a $20 bill on the ground and says, "Look, there's a $20 bill!"

The professor keeps walking and replies, "That can't be a real $20 bill; if it were, someone would have picked it up already."

There are plenty of $20 bills lying around in science today, waiting for the right person to pick them up.

How else can we understand the regular occurrence of anni mirablis where multiple groundbreaking discoveries are made in rapid succession by the same person?

In each instance, there was clearly plenty of low-hanging fruit, but it took the right person to come along to pick it.

For example, Galileo Galilei’s discovery that objects of different weights fall at the same rate or that constant motion is indistinguishable from rest could have been made (or accepted) centuries earlier.

Alternative explanations

With this in mind, it’s clear why an observed slowdown in progress despite increased investments by no means implies that we’ve run out of low-hanging fruit.

There’s no reason to believe that scientific inertia as a cultural phenomenon remains equally strong over time.

What if recent changes in the culture of science, like the introduction of peer review and the fixation on measurable authority increased scientific inertia significantly?

What if we made it harder and harder for new people to participate by professionalizing the job of a scientist, decoupling science from the rest of society, and only funding older researchers instead of giving young researchers opportunities to pursue their own ideas?⁴

What if attempts to make research funding more efficient made it impossible to explore the most promising stepping stones?

What if the current academic system only supports hill climbers and kicks out all valley crossers?

What if modern technology and increases in bureaucratic processes made it virtually impossible for researchers to dedicate significant time to deep, focused work required for breakthrough insights?⁵

Pessimism vs. optimism

One of the biggest turning points in my life was the realization that intelligence and talent are vastly overrated.

Through many years of brainwashing in the school system, I believed I wasn’t ever able to do anything meaningful because I was never the smartest or most talented in the room.

But one day I understood how silly believing this is.

All you’re accomplishing is turning it into a self-fulfilling prophecy.

You always have a choice between being optimistic or pessimistic.

Pessimism means you will definitely be correct because you won't even try.

Only by being optimistic are you giving yourself a chance to succeed.

The same applies to scientific progress.

If we believe we've almost reached the end of science or that all the easy discoveries have been made, our prediction almost certainly becomes true.

We will keep putting all focus and resources onto increasingly specialized research and mega experiments. Hence, all discoveries will be increasingly difficult and expensive to make. This self-fulfilling prophecy will reinforce the belief that science has reached its limits.

But if we remain optimistic and keep searching for those "$20 bills" lying around, we might just find them.

Crucially, this is not an exercise in wishful thinking.

As I have laid out above, there is plenty of good reason to believe that major low-hanging scientific breakthroughs are still possible.

Researchers thought they had reached the end of the road already a century ago.

Max Planck was told by a professor around 1890 that “the system as a whole stood there fairly secured, and theoretical physics approached visibly that degree of perfection which, for example, geometry has had already for centuries.”

Or as physicist Albert A. Michelson famously noted, “it seems probable that most of the grand underlying principles have been firmly established and that further advances are to be sought chiefly in the rigorous application of these principles to all the phenomena which come under our notice. It is here that the science of measurement shows its importance — where quantitative work is more to be desired than qualitative work. An eminent physicist remarked that the future truths of physical science are to be looked for in the sixth place of decimals.”

Then guys like Albert Einstein, Werner Heisenberg, Erwin Schrödinger, and Austin Bradford Hill came along and discovered plenty of low-hanging fruit.

It seems naive to believe that this time it's any different.

Do I know for sure? Of course not.

But looking at the evidence and given the choice between optimism and pessimism, I choose optimism every time.

1. Quoted in The End of Science by John Horgan

2. Maybe it will even turn out that we might have to travel (metaphorically) to a completely new planet to make further progress: maybe math isn’t the right language?

As Sabine Hossenfelder argues in her book Existential Physics, "it is presumptuous" to assume that "humans have already discovered the language in which nature speaks, basically on the first try."

Remember that Darwin spelled out his theory of evolution in On the Origin of Species without using a single equation.

Gödel's incompleteness theorems might be a hint in that direction.

Another hint is that there is never a unique mathematical formulation of each fundamental theory. Different formulations use completely different mathematical frameworks.

Also Iain McGilchrist’s ideas as spelled out, for example, in Ways of Attending seem relevant.

Or to quote John A. Wheeler: "To my mind there must be, at the bottom of it all, not an equation, but an utterly simple idea. And to me that idea, when we discover it, will be so compelling, so inevitable, that we will say to one another, 'Oh, how beautiful.' How could it have been otherwise?'"

3. Zero by Charles Seife is a fantastic read.

4. The word “scientist” didn't exist until 1833 (h/t Adam Mastroianni)

5. Stolen Focus by Johann Hari is an eye-opening read on the impact of modern technology on attention spans. The impact of increasingly complex bureaucratic process on the time researchers can spend on research is well documented, for example, here.