Predicting Replicability Challenge

Predicting Replicability is a challenge to advance automated, rapid assessment of the credibility of research claims. The aim is to develop methods that approximate expert and replication-based confidence judgments in seconds, enabling readers, researchers, reviewers, funders, and policymakers to focus attention and resources on high-importance, uncertain findings. This would scale trustworthiness assessment as new evidence arrives and improve the allocation of replication and review efforts.

Evidence from Altmedj’s work on predicting the replicability of social science lab experiments shows that simple black-box models can achieve performance comparable to market-aggregated expert beliefs: approx. 70% cross-validated accuracy (AUC ~0.77) on binary replication and Spearman ρ ~0.38 for relative effect sizes, with preregistered out-of-sample validation (approx. 71% accuracy, AUC ~0.73; effect size ρ ~0.25). Predictive features include sample and effect sizes and whether effects are main effects versus interactions. Such models can provide cheap, prognostic replicability metrics to help institutionalize evaluation workflows and target replications where they are most informative.