Making progress on metascience
Making progress on metascience
Metascience at the Progress Conference and how progress studies should be thinking about science
I recently attended the inaugural 2024 Progress Conference: Toward Abundant Futures in Berkeley, California. Organized by the Roots of Progress Institute—which also runs a blog building fellowship, of which I was a part of this year’s cohort—the purpose of the meeting is to connect members of the progress studies community, bring them together in the same place, and see what happens. Progress studies has grown a lot since its inception in a 2018 Atlantic article by Tyler Cowen, Professor of Economics at George Mason University, and Patrick Collison, Co-founder of Stripe and maybe the most important patron of the progress movement. An annual Progress Conference serves to formalize the movement and acts as a meeting point to hash out different perspectives, learn and teach new ideas, catalyze ambitious collaborations, launch exciting projects, and more.
This was a genuinely generative and intellectually stimulating conference. The progress studies community has retained a healthy level of debate and quibbling. That is a good thing! Embracing and leaning into disagreement is necessary for a successful intellectual community and a cultural movement with broad attraction.
I have observed something similar in the Ecomodernist movement. For example, at the annual Dialogue conference series—which concluded this year after over a decade and was the flagship Ecomodernist conference put on by the Breakthrough Institute where I have a non-resident appointment—one was likely to find full-fledged socialists, free-market evangelists, and milk toast centrists. In both cases, intellectual diversity is not a gimmick; folks really do have clashes of ideas. It is much more fun this way.
Benjamin Parry, proprietor of the Skillful Notes Substack, recently summarized his thoughts on the event. He observed that the formal academic discipline of progress studies originally intended by Collison and Cowen has not quite materialized but has emerged as “a vibe.” Instead, it is a sort of mood or cultural moment that is unabashedly pro-human and keen on technological advancement. I think that is right.
But from the academic perspective, I think it is good that progress studies has not become a formal academic discipline of its own and one can only hope it will not become one. There is a sense in which formalizing too much and erecting the fences endemic to fields of academia means that progress studies would inevitably consolidate, rigidify, and stagnate. It would cease to be a dynamic melting pot of dozens of academic disciplines from the humanities to physical and social science departments, along with startup technologists, think tank policy wonks, essayists, filmmakers, and the works.
In comparing progress studies with effective altruism (there is considerable crossover), Tyler Cowen made a related observation in a fireside conversation with Dwarkesh Patel, host of the Dwarkesh Podcast (an excellent addition to your regular podcast rotation). Cowen had predicted ‘peak EA’ just before the Sam Bankman Fried scandal. He explained that at the time, he felt that EA had lost its dynamism; the group had become too self-enamored and had grown insular. As I think most everyone at the meeting felt, Cowen observed the conference and movement feels different.
Metascience in the Progress Studies Community
What interests me most in progress studies, more broadly, and what I found most intellectually stimulating at this conference, in particular, is the metascience discourse.
Metascience, put simply, is the study of science. Metascience includes applying scientific methods to evaluate scientific research and its institutions and studying the social processes in science. The Great Stagnation and the plethora of socioeconomic indicators showing slow economic growth post-1970s have been a unifying motivator for the progress studies community. The way out of stagnation that the progress movement focuses on most is technological innovation and scientific advancement. As such, the progress studies community has come to care a great deal about metascience. And rightly so.
One often notices a desire in this community for science and technology breakthroughs that have been sorely lacking for decades. Peter Thiel’s aphorism “we wanted flying cars, instead we got 140 characters” has become an animating mantra. But there is a paradox: many in the progress movement yearn for breakthroughs but pursue a metascience that lends itself to incremental progress. To be clear, incremental progress is good, but if scientific discoveries on par with DNA, the transistor, the laser, nuclear fission, genetic modification, etc., are what we seek, we need a better metascience to get there. This year’s Progress Conference makes me more confident that the progress movement is moving in the right direction.
The different flavors of metascience
Metascience comes in different flavors. I suggest four categories on a spectrum from quantitative to qualitative modes of analysis: scientometrics, history of science, sociology of science, and philosophy of science.
Scientometrics:
Scientometrics emphasizes the quantitative analysis of the scientific literature. Using empirical datasets and corresponding indices, scientometrics—heavily influenced by macroeconomics—seeks core insights into the scientific process by quantifying researcher productivity, measuring the outcomes of grant programs, charting the frequency a topic is mentioned in the literature, etc.
History of science:
The history of science seeks to uncover the fact pattern of how specific scientific advances actually occurred or were missed. There is no greater prize in the history of science than finding a significant episode in science that has, for one reason or another, been told or understood incorrectly or in a misleading fashion. It is thus inherently more granular and qualitative than scientometrics.
Sociology of science:
The sociology of science places a core emphasis on the social processes of science. Often closely connected to the history of science, the focus is on the scientists and not only on the science. Namely, the sociology of science is very interested in the behavior of a scientific community and its major players and in understanding how scientific institutions are organized. In this field, the social processes of science, even more qualitative than the historical fact pattern of a scientific discovery, are the guide to scientific progress.
Philosophy of science:
The Philosophy of science is more qualitative still. This field is interested in more fundamental and meta-questions, such as how scientific knowledge is established, what qualifies as scientific evidence, or how semantics and rhetoric are employed to describe scientific phenomena and adjudicate their validity.
The overemphasis on scientometrics seems to me to be a classic example of physics envy and misses the point entirely. There is a false comfort derived from well-defined metrics and quantitative analysis. No doubt there is very good scientometrics work out there. But there are also many examples of studies that, for all practical purposes, do not provide much insight into the scientific discovery process.
The primary modalities for pursuing metascience in the progress movement are still being adjudicated. The issue laid bare is this: if the progress movement desires flying cars—flying cars being rightly understood as a stand-in for science fiction-level discoveries—almost tautologically, we are speaking of discoveries at the tail of the distribution. And yet, the basic premise of scientometrics is to quantify the literature—and being a good empirical field, this is done with large, robust datasets. But of course, in doing so, one is not studying the tail of the distribution but modal science instead. You will not find the proverbial flying cars there.
There are various means of getting around this. Scientometricians will point out that one can develop metrics applied only to the most cited papers or those published in premier journals or articles that went on to be the basis for a Nobel Prize. I am only convinced of the latter. However, examples abound of Nobel Prize winning scientific work with unremarkable citation counts or articles in Nature or Science that are far more indicative of the latest fad in science, not a paradigm-changing advance.
The progress studies metascience community should embrace the qualitative rather than the quantitative study of science. Perhaps surprising coming from a scientist, but if one wants to understand the tail of the distribution, deep and rigorous case studies laden with interviews and primary sources are the most promising approach.
Other communities interested in metascience, such as science and technology studies, have traditionally made primary the quantitative analysis of scientific literature. One should understand this by the fact that it is much easier to justify a conclusion with a quantitative, reproducible methodology, even if it proves not to be valuable in stimulating new scientific discoveries. I was worried that progress studies would tend toward this path; I have been pleasantly surprised.
Progressing Metascience
Metascience was front and center at the Progress Conference, starting with Patrick Collison, who also sat down for a fireside chat with Dwarkesh Patel. Not only did Collison essentially launch the progress studies movement, but he and Michael Nielsen wrote one of the most incisive and widely read progress-oriented pieces on metascience also in The Atlantic, which I have previously written about. And since then, Collison has co-founded the ARC Institute to push next-generation advances in biotechnology and life sciences and recently joined the board of the latest fusion outfit, Pacific Fusion.
Collison emphasized a curious observation that spans both the history and sociology of science: most Nobel Prize winners in physical science come from an academic lineage of Nobel Prize winners. In particular, Benjamin Parry specified that 96% of Nobel Prizes in physics, chemistry, medicine, and economics are awarded to researchers who come from an academic tree of mentors who themselves won a Nobel Prize. That is a remarkable fact!
It suggests a tacit knowledge and a way of practicing science that is passed down via social interaction as opposed to explicit documentation. Understanding this social process of training students or mentees seems critical for Nobel Prize-level discovery. One of the conference sessions was on just this: tacit knowledge. Parry led the session, and a fascinating discussion of how tacit knowledge plays a major role in scientific discovery ensued. Everything from how one prepares and executes experiments to the effort level exerted by prominent scientists to how one chooses which scientific questions to work on is of critical importance. It also has implications for advocates of AI scientists: if the training for scientific discovery is inherently tacit as opposed to explicit, it would seem difficult to train AI models on this critical component of scientific discovery.
Another relevant session, called Bell Labs 2.0, focused on why the 20th-century industrial labs, such as Bell Labs, Xerox PARC, etc., were so revolutionary. The transistor, solar cell, laser, charge-coupled device, personal computer, and much more were invented or developed at these institutions. Why, the session organizers—Laura Ryan, Co-founder of the Better Science Project at the Tony Blair Institute, and Jeff Tsao, Senior Scientist at Sandia National Labs—asked, were these labs so effective at producing hugely significant scientific and technological innovations and how can we achieve this level of progress today?
Inspired by Ada Lovelace, the pair argued for a Lovelace R&D organization model, whereby this organization would serve to insulate research institutes from funders. This approach enables long-term funding for research outfits—each of which can experiment with different research models—with a well-defined, time-bound progress review to decide if the institute should continue or be replaced by a new model.
It was encouraging to see others engaging with these alternative models for research. Of course, the advent of the Focused Research Organization, pioneered by Convergent Research, and Ben Reinhardt’s Speculative Technologies industrial research programs are good examples of the exciting times for new research models.
Where does metascience go from here?
The progress studies movement needs to focus on the tail of the distribution of science. If metascience is left to the scientometricians or science and technology studies departments, there is little hope for progress of the same order as the breakthroughs made in the 20th century.
It is as important as ever for the progress studies community to fund truly ambitious scientific work, which conventional institutions such as the National Science Foundation will likely never have the risk appetite to support. This requires continued experimentation with different research models, some of which will be better suited for different types of science. Progress studies should also lean into the qualitative study of discovery, recognizing the social dynamics very much at play in the history and present of science.
The good news is a guide already exists for the progress studies community: A Vision of Metascience. In their brilliant manifesto on metascience, Michael Nielsen and Kanjun Qiu lay out an extensive argument for prioritizing the social processes of science and the need for experimentation in scientific institutions. I will likely dedicate a future blog post to unpacking their hugely important publication. My hope is that Michael’s and Kanjun’s vision can be a launch pad for the metascience-inclined members of the progress studies community interested in scientific discovery and technological invention at the tail of the distribution.
Many thanks to Jason Crawford, Heike Larson, and Emma McAleavy for organizing the Progress Conference and the blog building fellowship program.
Loved this! The break down of the 4 categories of: Scientometrics, History of science, Sociology of science, and Philosophy of science especially helpful.