The first major project of The Metascience Observatory is to build a publicly accessible database of experimental replications across all of science.

There have already been many great studies assessing reproducibility, such as the tour de force studies in psychology and cancer biology spearheaded by Brian Nosek, the "Many Labs" initiatives in psychology, and the Transparent Replications project run by Spencer Greenberg’s team at clearerthinking.org. These were large, concerted efforts requiring many hours of work and lots of funding.

However, there are also one-off “garden variety” replication attempts. They are rare (perhaps 1% of papers), but there are thousands out there. Using both manual curation and AI/LLMs, we will create a database of replications and statistics on how reproducibility varies across fields. Unlike the FORRT replication database, which covers only psychology, we are covering all of science, including fields like materials science and engineering where there is not a lot of hard data on reproducibility.

A database of replications will enable many interesting avenues of metascience research looking at "correlates of reproducibility". The database will also help people quickly determine if a finding has replicated. After all, it is only after independent experiments consistently demonstrate a claim that we can really start to believe it strongly. (This is true regardless of whether you are a verificationist, falsificationist, Bayesian, Popperian, etc.)

Longer term, we may be able to create a reproducibility ranking for different journals. Reproducibility rankings for journals could improve science by shifting focus away from impact factor (citation counts) towards the actual quality and rigor of scientific work. Previous work shows that citation numbers are not correlated with reproducibility and that citation rates are positively correlated with the chance that a paper will be retracted.

The Metascience Observatory is currently fiscally sponsored by Mind First Foundation, a 501c3 nonprofit based in Massachusetts.

As of 2026 our scientific advisory board consists of Stuart Buck, Gavin Leech and Preston Estep III.

We are also informally advised by James Heathers, Paul Litvak, and Alex Holcombe.

Why am I (Dan Elton) qualified to work on this?

I am quite familiar with academic research, as I was a PhD student in physics for six years, followed by two years as a postdoc and three years as a Staff Scientist at NIH. I am a co-author on about 45 peer-reviewed papers in physics, AI for molecular discovery, and AI for medical imaging. I've written eight articles about metascience topics on my Substack. The most notable is "A Defense of Peer Review", an article I did for Asimov Press which required over 100 hours of research.

Update - October 2026

In August I published a report on what I've learned so far from the database project.

I am now seeking funding to continue work on this project. More specifically:

• Support the merger of our database with the FORRT FReD and FLoRA projects.

• Do more thorough evaluation work on the AI extraction system, which has changed since it was last evaluated in February, 2026. 

• Make improvements to the pipeline, especially to improve the extraction of statistical data. 

• Run the extraction system on a backlog of about 2,000 papers that have been screened by AI (Haiku) as containing replications.

• Improve the way I search for replications to find more replications in a more diverse set of fields. The current approach I use is pretty basic (a keyword set, then LLM filter).

• Improve the pipeline code and pipeline dashboard. I would like to be able to run the pipeline regularly (ie once a month) to pick up new replications as they are published. Improving the code will streamline this process, so the entire pipeline can be run easily without much human involvement (search -> download -> screen -> extract -> ingest).

• Ingest data from replications initiatives I haven’t ingested from yet. At a minimum this includes data on education replications and data from reproducibility challenges in machine learning. 

• See if I can get data on replications that were conducted but never published from Replication Games and MATS. 

• Create a “did this replicate” educational game (similar to this prototype for psychology replications). Drawing on “calibration games” that are played in the rationalist community, I will ask users to provide a probability and supply a calibration curve at the end. If I get many users playing our game, then we can study interesting questions like whether including the journal helps or hurts human guessers. 

Appendix: longer term project we are exploring : developing a system for finding and grouping papers with similar claims 

A key difficulty with the replications dataset is that direct replications are very rare (< 1% of papers), and in some fields like physics they are extremely rare. “Conceptual replications” that attempt to replicate a specific finding but using different methods are a bit more common, but are still quite rare. 

What is more common, however, are studies that all address a general claim from different angles. 

LLMs can easily break a paper into a discrete set of core claims, with each claim being a single natural language sentence. Consider the following claims: 

• “Nattokinase prevents coronary plaques in mice.”

• “Nattokinase supplementation has no effect on coronary plaque development in the serial CT scans of elderly patients.”

• “Nattokinase supplementation has no effect on stroke risk when taken for a year by Japanese adults.” 

All three claims attempt to address the same general question (“Does nattokinase supplementation prevent heart disease?”), but they do so in different ways. 

By embedding claims into sentence vectors, it might be possible to rapidly find claims that are closely related. (Finding negations is apparently tricky, and not as simple as "the vector points in the opposite direction".)

Researchers have published work on claim extraction and using embeddings for search before, although I haven't yet seen the exact combination I'm describing here. As part of my work I would review the literature. A recent work appears to have implemented something similar to this for Chemistry, although it uses verbatim quotes as the claims, not LLM summaries.