Recently, through DARPA's Systematizing Confidence in Open Research and Evidence (SCORE) program, we developed artificial prediction markets to evaluate the replicability of published research claims in the social and behavioral science literatures. Markets were populated by artificial agents; they will be trained and updated within human-expert prediction markets, but deployable offline. This line of work aims to improve our understanding of the robustness of the existing social science literature and highlight paths forward for a more credible, interpretable scholarly record.

Building on the SCORE work and some of the data and methods developed there, we are now working with Open Philanthropy and the Center for Open Science to develop benchmarks for LLM agents (and to develop LLM agents) who can replicate published work. 

Through other NSF-funded and seed-funded efforts, we continue to try to understand the research ecosystem beyond replication, including the role of institutions, cultures, and norms in shaping science. 

​​​​​

• R. Obadage, S. Rajtmajer, J. Wu. CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis. Joint Conference on Digital Libraries (JCDL), December 2025.

• T. Chakravorti, S. Gautam, S. Rajtmajer. Embracing Transparency: A Study of Open Science Practices Among Early Career HCI Researchers. PLoS ONE, November 2025.

• T. Chakravorti, X. Wang, P. Narayanan Venkit, S. Koneru, K. Munger, S. Rajtmajer. Social Scientists on the Role of AI in Research. AAAI/ACM Conference on AI, Ethics, and Society (AIES), October 2025.

• R. R. Obadage, K. Thennakoon, S. Rajtmajer, and J. Wu. (2025). [Re] Network Deconvolution. ReScience C 10, 1, #4.

• T. Chakravorti, S. Koneru, S. Rajtmajer. Reproducibility and replicability in research: What 452 professors think in Universities across the USA and India. PLoS ONE 20(3): e0319334, March 2025.

• R. Obradage, S. Rajtmajer, J. Wu. Can citations tell us about a paper's reproducibility? A case study of machine learning papers. ACM Conference on Reproducibility and Replicability (ACM REP), June 2024. Short paper.

• S. Koneru, J. Wu, S. Rajtmajer. Can LLMs discern evidence for scientific hypotheses? Case studies in the social sciences. Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), May 2024.

• C. Wu, X. Wang, J. Carroll, S. Rajtmajer. Reacting to Generative AI: Insights from Student and Faculty Discussions on Reddit. ACM Conference on Web Science (WebSci), May 2024.

• C. Wu, T. Chakravorti, J. Carroll, S. Rajtmajer. Integrating measures of replicability into scholarly search: Challenges and opportunities. ACM Conference on Human Factors in Computing Systems (CHI), May 2024.

• S. Koneru, X. Wei, J. Wu, S. Rajtmajer. Can machine learning algorithms predict publication outcomes? A case study of COVID-19 preprints. AI4SciSci Workshop at the International Conference on Data Mining (ICDM), December 2023.

• K. Ajayi, M. Hasan Choudhury, S. Rajtmajer, J. Wu.  A Study on Reproducibility and Replicability of Table Structure Recognition Methods. International Conference on Document Analysis and Recognition (ICDAR), August 2023.

• ​S. Koneru, M. Smith, D. Guarrera, J. Robinson, S. Rajtmajer. The evolution of scientific literature as metastable knowledge states. PLoS ONE, July 2023.

• K. Ajayi, M. Hasan Choudhury, S. Rajtmajer, J. Wu.  A Study on Reproducibility and Replicability of Table Structure Recognition Methods. International Conference on Document Analysis and Recognition (ICDAR), August 2023.

• T. Chakravorti, R. Fraleigh, T. Fritton, M. McLaughlin, V. Singh, C. Griffin, A. Kwasnica, C. Pennock, C. Lee Giles and S. Rajtmajer. A prototype hybrid prediction market for estimating replicability of published work. The Second International Conference on Hybrid Human-Artificial Intelligence (HHAI), June 2023.

• T. Chakravorti, V. Singh, M. McLaughlin, R. Fraleigh, C. Griffin, A. Kwasnica, D. Pennock, C. Lee Giles and S. Rajtmajer. Artificial prediction markets present a novel opportunity for human-AI collaboration. 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2023. Extended Abstract.

• D. Priestley, J. Staph, S. Koneru, S. Rajtmajer, A. Cwiek, S. Vervoordt, F. Hillary. Establishing ground truth in the traumatic brain injury literature: if replication is the answer, then what are the questions? Brain Communications, December 2022.

• S. Rajtmajer, T. Errington, F. Hillary. How failure to falsify in high-volume science contributes to the replication crisis. eLife, August 2022.

• E. Cruz Cortes, S. Rajtmajer, D. Ghosh. Structural Interventions on Automated Decision Making Systems. ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT), June 2022.

• L. Salasbil, J. Wu, M. Choudhury, W. Ingram, E. Fox, S. Rajtmajer and C. Lee Giles. A Study of Computational Reproducibility using URLs Linking to Open Access Datasets and Software. Sci-K @ The Web Conf, April 2022.

• S. Rajtmajer, C. Griffin, J. Wu, R. Fraleigh, L. Balaji, A. Squicciarini, A. Kwasnica, D. Pennock, M. McLaughlin, T. Fritton, N. Nakshatri, A. Menon, S.A. Modukuri, R. Nivargi, X. Wei, C.L. Giles. A synthetic prediction market for estimating confidence in published work. AAAI 2022 Demonstrations, February 2022.

• E. Cruz Cortes, S. Rajtmajer, D. Ghosh. Structural Interventions on Automated Decision Making Systems. NeurIPS workshop on Algorithmic Fairness through the Lens of Causality and Robustness (NeurIPS AFCR), December 2021.

• A. Cwiek, S. Rajtmajer, B. Wyble, V. Honavar, F. Hillary. Feeding the machine: challenges to reproducible predictive modeling in resting-state connectomics. Network Neuroscience, October 2021.

• S. Lanka, S. Rajtmajer, J. Wu, C. Lee Giles. Extraction and evaluation of statistical information from social and behavioral science papers. Workshop on Scientific Knowledge (Sci-K) at The Web Conference, April 2021.

• S. Modukuri, S. Rajtmajer, A. Squicciarini, J. Wu, L. Giles. Understanding and predicting retractions of published work. AAAI-21 Workshop on Scientific Document Understanding (AAAI-SDU), February 2021. 

• C. Grady, S. Rajtmajer, L. Dennis. When smart systems fail: the ethics of cyber-physical critical infrastructure risk. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

• L. Dennis, S. Rajtmajer, C. Grady. Analyzing cyber-physical threats to Pennsylvania dams through a lens of vulnerability. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

• J. Wu, P. Wang, X. Wei, S. Rajtmajer, C. Lee Giles, C. Griffin. Accurate Acknowledgement Entity Recognition in CORD-19 Papers. Workshop on Scholarly Document Processing and Shared Tasks at The Conference on Empirical Methods in Natural Language Processing (EMNLP SDP), November 2020. 

• S. Rajtmajer and D. Susser. Automated Influence and the Challenge of Cognitive Security. ACM Symposium on the Science of Security (HoTSoS), February 2020.