Economics and Political Science Studies Hard to Repeat, New Research Shows

Only about half of published papers in social sciences, including economics and political science, can be independently repeated by other researchers. This is a big problem for science.
By Newsroom | Social Sciences, Research |

Major replication efforts reveal widespread issues with replicating published findings.

A large-scale project re-analyzing 110 articles from top economics and political science journals found that while 84% of claims could be computationally reproduced, the robustness of these findings is a separate, and more concerning, matter. This effort, involving six independent research teams examining 12 prespecified hypotheses, suggests that experience in replication can paradoxically lead to finding lower levels of robustness, with no clear link between robustness and author characteristics or data availability.

Further contributing to the discourse, a broader study across social sciences, including economics and political science, indicates that findings from roughly half of all published papers are not independently replicable. This project, known as Systematizing Confidence in Open Research and Evidence (SCORE), investigated over 100 papers across numerous leading journals. The availability of data and computer code emerged as a crucial factor, with papers making this information readily accessible showing significantly higher reproduction rates. Conversely, only one-third of papers in the SCORE sample had their data and code publicly available.

Read More: New tools may help diagnose endometriosis faster for women

The replication initiatives encountered significant hurdles. The lack of raw data severely restricted the scope of re-analyses across various categories. While some results were reproducible, the study highlighted a high prevalence of coding errors. These re-analyses, which involved altering weighting schemes, control variables, estimation methods, or employing new data, often led to a decrease in the statistical significance of originally published test statistics. The robustness checks, categorized into eight groups, underscored the fragility of many published conclusions.

These findings emerge from a confluence of recent scholarly outputs, including a report in Nature, discussions on blogs such as Nevin Manimala's and Ali Niazi's, and a feature in AoI (Articles of Interest) on the replication network. The research highlights a systemic challenge within these fields, where the initial publication of results, even with mandatory data and code sharing policies, does not guarantee their long-term reliability or independent verification. The implication is that a significant portion of the scholarly record in economics and political science may not withstand rigorous scrutiny, presenting a sobering reflection on the foundations of knowledge in these disciplines.

Frequently Asked Questions

Q: Why are many economics and political science studies hard to repeat?
New research shows that findings from about half of published papers in these fields are not independently replicable. This is because data and computer code are often not shared, making it difficult for others to check the work.
Q: What did the large project re-analyzing 110 articles find?
A big project looked at 110 articles from top economics and political science journals. While most claims could be checked using computers, their strength and reliability were often weaker when checked by others.
Q: What is the SCORE project and what did it find?
The SCORE project studied over 100 papers in social sciences. It found that only about half of the papers had their data and code shared publicly, which made it much harder for other researchers to repeat their findings.
Q: What problems did researchers face when trying to repeat studies?
Researchers often could not repeat studies because the raw data was missing. When they did try, they found many coding errors and that the original results were less strong when checked again.
Q: What does this mean for the future of economics and political science research?
These findings suggest that many published results in these fields may not be reliable. This is a serious issue that makes us question the trust we place in the knowledge from these areas.