Insights into the accuracy of social scientists’ forecasts of societal change

The Forecasting Collaborative; Grossmann, Igor; Rotella, Amanda; Hutcherson, Cendri A.; Sharpinskyi, Konstantyn; Varnum, Michael E. W.; Achter, Sebastian; Dhami, Mandeep K.; Guo, Xinqi Evie; Kara-Yakoubian, Mane; Mandel, David R.; Raes, Louis; Tay, Louis; Vie, Aymeric; Wagner, Lisa; Adamkovic, Matus; Arami, Arash; Arriaga, Patrícia; Bandara, Kasun; Baník, Gabriel; Bartoš, František; Baskin, Ernest; Bergmeir, Christoph; Białek, Michał; Børsting, Caroline K.; Browne, Dillon T.; Caruso, Eugene M.; Chen, Rong; Chie, Bin-Tzong; Chopik, William J.; Collins, Robert N.; Cong, Chin Wen; Conway, Lucian G.; Davis, Matthew; Day, Martin V.; Dhaliwal, Nathan A.; Durham, Justin D.; Dziekan, Martyna; Elbaek, Christian T.; Shuman, Eric; Fabrykant, Marharyta; Firat, Mustafa; Fong, Geoffrey T.; Frimer, Jeremy A.; Gallegos, Jonathan M.; Goldberg, Simon B.; Gollwitzer, Anton; Goyal, Julia; Graf-Vlachy, Lorenz; Gronlund, Scott D.; Hafenbrädl, Sebastian; Hartanto, Andree; Hirshberg, Matthew J.; Hornsey, Matt...

Abstract

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender–career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists’ forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.

• Journal Article Type: Article
• Acceptance Date: Dec 19, 2022
• Online Publication Date: Feb 9, 2023
• Publication Date: Apr 1, 2023
• Deposit Date: May 16, 2023
• Publicly Available Date: Aug 10, 2023
• Journal: Nature Human Behaviour
• Electronic ISSN: 2397-3374
• Publisher: Springer Science and Business Media LLC
• Peer Reviewed: Peer Reviewed
• Volume: 7
• Issue: 4
• Pages: 484-501
• DOI: https://doi.org/10.1038/s41562-022-01517-1
• Keywords: Behavioral Neuroscience; Experimental and Cognitive Psychology; Social Psychology
• Public URL: https://nottingham-repository.worktribe.com/output/17087610
• Publisher URL: https://www.nature.com/articles/s41562-022-01517-1
• Additional Information: Received: 26 June 2022; Accepted: 19 December 2022; First Online: 9 February 2023; : The authors declare no competing interests.; Free to read: This content has been made available to all.