top of page

Through DARPA's Systematizing Confidence in Open Research and Evidence (SCORE) program, we are developing artificial prediction markets to evaluate the reproducibility of published research claims in the social and behavioral science literatures. Markets are populated by artificial agents; they will be trained and updated within human-expert prediction markets, but deployable offline. This line of work aims to improve our understanding of the robustness of the existing social science literature and highlight paths forward for a more credible, interpretable scholarly record.

In parallel, we are also working with the National Science Foundation's National Center for Science and Engineering Statistics to study trends in research and publishing, integrating data from their annual Survey of Doctoral Recipients. Our broader aim is data-driven policies on scientific processes and funding, with specific focus on underrepresented PhD recipients. 

Recent papers on science policy focus on the ethics of securing our increasingly interdependent cyber-physical critical infrastructure.

  • D. Priestley, J. Staph, S. Koneru, S. Rajtmajer, A. Cwiek, S. Vervoordt, F. Hillary. Establishing ground truth in the traumatic brain injury literature: if replication is the answer, then what are the questions? Brain Communications, December 2022.

  • S. Rajtmajer, T. Errington, F. Hillary. How failure to falsify in high-volume science contributes to the replication crisis. eLife, August 2022.

  • E. Cruz Cortes, S. Rajtmajer, D. Ghosh. Structural Interventions on Automated Decision Making Systems. ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT), June 2022.

  • L. Salasbil, J. Wu, M. Choudhury, W. Ingram, E. Fox, S. Rajtmajer and C. Lee Giles. A Study of Computational Reproducibility using URLs Linking to Open Access Datasets and Software. Sci-K @ The Web Conf, April 2022.

  • S. Rajtmajer, C. Griffin, J. Wu, R. Fraleigh, L. Balaji, A. Squicciarini, A. Kwasnica, D. Pennock, M. McLaughlin, T. Fritton, N. Nakshatri, A. Menon, S.A. Modukuri, R. Nivargi, X. Wei, C.L. Giles. A synthetic prediction market for estimating confidence in published work. AAAI 2022 Demonstrations, February 2022.

  • E. Cruz Cortes, S. Rajtmajer, D. Ghosh. Structural Interventions on Automated Decision Making Systems. NeurIPS workshop on Algorithmic Fairness through the Lens of Causality and Robustness (NeurIPS AFCR), December 2021.

  • A. Cwiek, S. Rajtmajer, B. Wyble, V. Honavar, F. Hillary. Feeding the machine: challenges to reproducible predictive modeling in resting-state connectomics. Network Neuroscience, October 2021.

  • S. Lanka, S. Rajtmajer, J. Wu, C. Lee Giles. Extraction and evaluation of statistical information from social and behavioral science papers. Workshop on Scientific Knowledge (Sci-K) at The Web Conference, April 2021.

  • S. Modukuri, S. Rajtmajer, A. Squicciarini, J. Wu, L. Giles. Understanding and predicting retractions of published work. AAAI-21 Workshop on Scientific Document Understanding (AAAI-SDU), February 2021. 

  • C. Grady, S. Rajtmajer, L. Dennis. When smart systems fail: the ethics of cyber-physical critical infrastructure risk. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

  • L. Dennis, S. Rajtmajer, C. Grady. Analyzing cyber-physical threats to Pennsylvania dams through a lens of vulnerability. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

  • J. Wu, P. Wang, X. Wei, S. Rajtmajer, C. Lee Giles, C. Griffin. Accurate Acknowledgement Entity Recognition in CORD-19 Papers. Workshop on Scholarly Document Processing and Shared Tasks at The Conference on Empirical Methods in Natural Language Processing (EMNLP SDP), November 2020. 

  • S. Rajtmajer and D. Susser. Automated Influence and the Challenge of Cognitive Security. ACM Symposium on the Science of Security (HoTSoS), February 2020.

bottom of page