Technology is not stagnating, but science is

To overcome scientific stagnation, we need to embrace the anomaly

The debate over whether technology is stagnating has received much news coverage and attention from the academic and think tank communities. However, such a suggestion seems implausible in light of the biennial doubling of the number of transistors on computer chips (dubbed Moore’s Law), steadily increasing crop yields across agricultural products, decadal cost declines and efficiency gains in solar photovoltaic cells, and much much much more.

The debate reached a height in response to a 2017 National Bureau of Economic Research working paper, in which the authors argue that, while many technology categories are progressing according to various domain-specific metrics of performance, the number of researchers and the amount of financial resources required to maintain growth is rising—implying a decline in research productivity. They explain that new ideas are getting harder to find. This conclusion—that research gets harder over time as low-hanging fruit are picked—has been expounded by many. Matt Clancy presents further evidence in his blog, What’s New Under the Sun, including a decline in the number of creative patents defined by the use of novel technical terms and an overall slow-down in the number of unique topics—termed the cognitive extent—investigated across the scientific enterprise.

Various alternative theories have also tried to account for these data, including how declining research productivity should be the null hypothesis: implications otherwise would result in growth figures that are at least unrealistic, if not outright implausible. Others have argued instead that progress is, by and large, the product of outputs from a select few top-performing researchers and that new entrants in a growing field contribute comparatively less. Further examination of this hypothesis suggests the full story is more complicated.

The empirical evidence across a range of industries and technologies seems convincing enough: many of our most important technologies are making substantive progress. But perhaps we are simply not asking the right question.

That is, critics of so-called technological stagnation rightly point to the obvious progress in existing technologies, represented by exponential improvement in the figures of merit across various technologies of interest over time. But what they do not look at is how many discoveries warrant the creation of entirely new fields. In other words, how many new Moore’s law-like exponential curves have been started, and how might science find more? So, the better question is whether science itself—not technology—is stagnating.

Tracking progress in science

Quantifying scientific progress is a particularly onerous task. Whereas technologies have clearly defined and well-established figures of merit, science is far more enigmatic. Plus, its measures of progress are often based on the judgment of researchers.

Many attempts by social scientists to make facile measurements of scientific progress are wanting. Metrics or indices of scientific innovation inevitably rely far too much on patents per capita, citations per researcher, or similarly cursory measures. It is not that those metrics don't measure anything—surely they do—but it is unclear why one should expect those kinds of metrics to capture the essence of genuinely innovative scientific progress.

Some researchers, however, do better than others. A 2023 article in Nature by Park, Leahey, and Funk makes a compelling quantitative argument that patents and papers—even those published in highly regarded journals such as Science and Nature or those that would later be the basis for a Nobel Prize—were found to be continually less disruptive over time since at least the 1940s. By disruptive, they mean a scientific advance that alters a scientific paradigm or stimulates a significant change of course in a field.

The authors’ favored measure of disruptiveness is an index (the CD index) ranging from -1 (consolidating) to 1 (disruptive). A disruptive paper is one in which the later articles that cite it do not share its citations, indicating that the paper in question represents a substantial break from prior work and is potentially foundational in a new field. A consolidating paper is one where later articles that cite it also cite the articles cited by the paper of interest itself. This indicates that the paper—while innovative, to be sure—served more so as a consolidation of ideas. The utility of this relatively new metric, which can be measured at different time scales from the point of publication (5 years or CD₅ was the preferred time period of the authors), is certainly worthy of scrutiny. Still, the data show seemingly significant results, which appear robust to several empirical tests. And critiques of the work to date—specifically pertaining to patents and ignoring research papers—have fallen flat as the physicist Sabine Hossenfelder explained earlier this year.

The headline results, in my mind, show a steep drop in the CD₅ index of scientific papers over time, even in high-prestige journals and Nobel prize-winning articles. Interestingly, the absolute number of highly disruptive (i.e., CD₅ > 0.25) research papers actually remained fairly constant since 1945. So, while the proportion of highly disruptive papers is declining—despite significant growth in the share of the population involved in research—such articles are still being published and at a nominally constant output; there is still very good science being done.

So perhaps everything is fine?

The authors point out that the respective share of these disruptive papers changes over time between different fields, suggesting an untapped, latent capacity for more disruptive work. In other words, there is room for better innovation policy and maybe even a more ambitious scientific culture to raise the bar and stimulate more disruptive research efforts.

Average CD5 index of papers in different fields over time (left) and CD5 index of Nobel Prize-winning papers by field and of papers in the journals Science, Nature, and PNAS in the inset figure (right).

These CD₅ data can be interpreted very differently by scholars with different perspectives. Namely, some will be inclined to see the decline in disruptiveness—and that the decline is sharper earlier in the time series—as evidence for a ‘low-hanging fruit’ hypothesis. The authors point out, however, that it seems unlikely that different fields would pick their low-hanging fruit at the same rate (i.e., the rates of decline in CD₅ should vary across fields).

But in any event, are we really to believe that Bohr’s quantum mechanics was a low-hanging fruit?

Low-hanging fruit arguments are almost tautologically true for a particular technology that is progressing along a temporal trend and approaching physical limits (e.g., Moore’s Law). But it seems quite difficult to seriously argue that Madame Wu’s parity violation experiments in the 1950s were easier than, say, the latest advances in quantum optics or that the Dirac equation, Feynamn’s path integrals, or Einsteinian general relativity were any less challenging than today’s focus on string theories or loop quantum gravity.

The authors instead suggest that an increasingly siloed scientific community that specializes in narrow areas of expertise lacks the diversity of knowledge needed to make progress on the frontier of science. Indeed, many scientific revolutions arose out of scientists branching out into new areas of research (e.g., the role of quantum physicists in the foundation of molecular biology) or gaining expertise in and working at the intersection of otherwise segregated disciplines (e.g., the emergent invention of the transistor from integrating advances in conduction theories based on quantum mechanics, high-purity single crystal growth, and device tinkering by radio engineers). They argue that “narrower slices of knowledge benefits individual careers, but not scientific progress more generally.”

But again, one should have some skepticism that such quantitative analyses necessarily elucidates—rather than obscures—the qualitative assessment of researchers. On the flip side, there is no doubt that experts can be biased by the dominant theories of the day and lose perspective. Both modes of investigation are necessary.

Not all Nobels are alike

More evidence that science is not producing breakthroughs of the same order as, say, a century ago can be found in survey data presented by Patrick Collison and Michael Nielsen in The Atlantic. Namely, Collison and Nielsen share data on a survey of experts in physics, chemistry, and biology who were asked to choose the more significant advance between two Nobel-prize-winning discoveries in their respective fields. The most startling results are in physics, where plotting the probability that a discovery deemed more significant was made in a given decade peaked in the 1920s—around the advent of quantum mechanics. A smaller local maximum occurred in the 1960s due to the discovery of the cosmic microwave background radiation and progress towards the standard model of physics (e.g., Weinberg and Salam’s incorporation of the Higgs mechanism into the electro-weak force, unified in 1959 by Glashow), which continued into the 1970s with quantum chromodynamics.

Probability that physics expert survey respondents select a Nobel prize-winning discovery made in a particular decade as more important than a discovery from another decade.

While trends in chemistry and biology slightly increased in the first half of the 20th century before flatlining, what is most notable across all three disciplines is that the 1980s is the last decade plotted in the figures. The Atlantic piece was published in 2018; the gap is not a mistake! Compared to the 20th century, Nobel prizes in science have systematically been awarded for work done further in the past, which has been dubbed the Nobel Prize time gap, and the data for prizes awarded for work done in the 1990s and into the 21st century is so sparse that they were not plotted. Such a finding requires that one takes seriously the idea that contemporary innovations in science are qualitatively less consequential than those from earlier generations.

In a sense, the research question one might ask shifts from: ‘has a particular technology improved over time’ to ‘how many new scientific fields have emerged over time?’ The latter is more indicative of scientific progress.

With such a shift in frame, the more qualitative and sociological study of what leads to scientific revolutions or new scientific fields or fundamentally new technology disciplines becomes far more instructive. Where do fundamentally new technologies—and the science that undergirds them—come from?

Where does great science come from?

A natural place to start is Thomas Kuhn’s The Structure of Scientific Revolutions. The key insight of Kuhn’s canonical book is the importance of the anomaly in the process of scientific discovery and the way in which it interfaces with the scientific paradigm of the time.

Cover of Thomas Kuhn’s book, The Structure of Scientific Revolutions.

Kuhn rightly identifies how the ‘normal science’ of any given paradigm is ill-equipped to recognize data as anomalous and even suggests such dynamics as having more fundamental origins in human psychology and the study of pattern perception. The way in which normal science pursues progress aligns much closer to how quantitative social scientists expect scientific progress to manifest: as incremental advances, amending existing scientific understanding. In contrast, the study of paradigm-shifting anomalies inherently lends itself to a more messy—as opposed to normal—kind of science. Indeed, Kuhn documents numerous instances (e.g., the discoveries of oxygen and X-rays) in the history of science where the lack of context or a familiar theoretical framework hampers the ability of scientists to recognize what is, in hindsight, an obviously anomalous observation with the potential to disrupt a paradigm or give birth to a new field.

It seems entirely reasonable then to wonder if such anomalies have simply gone unrecognized in the past—and whether that ignorance persists today. That is, not only should we expect future anomalies to be difficult for the scientific community to digest, but it is entirely plausible, if not likely, that there should exist anomalies in the literature—even in well-explored fields, which either went unnoticed or were thought to be without merit due to a lack of reproducibility, experimental error, etc. An operative lesson from Kuhn, even if not stated explicitly, is that one should be skeptical of any historical scientific consensus that some purported anomaly has been decisively put to bed without daring to embark on messy science.

In The Golem, Collins and Pinch explore case studies of controversies in science in which anomalies, particularly those with ambiguity in the underlying data, feature prominently. What is most interesting about Collins and Pinch’s account is that they elucidate how the anomalies that underpin both scientific controversies, now widely understood—perhaps incorrectly—as erroneous and sloppy (e.g., cold fusion), as well as foundational experiments central to today’s scientific paradigm (e.g., early—not more recent—experimental evidence for Einstein’s relativity) have, at their origins, far more ambiguity than is widely understood, certainly by historians of science, but most surprisingly by scientists themselves.

But, of course, this should not come as a surprise at all.

Indeed, the ability or comfortability to constructively engage with such ambiguous, purportedly anomalous data is the work of messy science and is incompatible with Kuhn’s normal science. In a sense, normal science mistakes messy science as an unserious or uncritical endeavor. It should go without saying—but of course, nothing ever does—that scrutiny and skepticism are central to the progress of normal science and are meritorious qualities. The ability to do messy science is not the revocation of the scientist’s skepticism or the unwitting embrace of all that purports to be anomalous; it is the ability to explore the purported anomaly, grapple with ambiguity, and operate in murky waters while keeping one’s head above the waterline.

Cover of Harry Collins’s and Trevor Pinch’s book, The Golem: What you should know about science.

New fields emerge from probing anomalies, even if—especially if—such an anomaly has not been recognized, digested, or appreciated by the broader scientific community. The stagnation in scientific discovery is a vicious feedback loop. It has produced a scientific culture often incapable of practicing messy science outside the Kuhnian paradigm. Examples abound of experimental phenomena—the veracity of which may or may not ultimately have merit—that were adjudicated too quickly, without sufficient investigation, and have been largely lost to history or left unresolved. Indeed, there is nothing to resolve if no anomaly is thought to exist. This culture hampers the scientific community’s ability to recognize contemporary anomalies or identify those from the past.

This blog is titled Seeking Scientific Revolutions. In it, I will connect many threads from the science and technology innovation literature to historiographic deep dives of consequential scientific work to the sociology and philosophy of science to case studies of contemporary research programmes on the scientific frontier. The goal is prodigious but clear: to elucidate the process of scientific discovery, to foster a more ambitious scientific community, and to catalyze scientific revolutions.

---