Does Permitting Uncertain Estimates Help or Hinder the Wisdom of Crowds?

Ellerby, Zack; Wagner, Christian

Abstract

This paper adds to a growing body of research into the practical utility of using interval-valued (IV) response modes to efficiently capture richer quantitative data from people-e.g., through surveys. Specifically, IV responses offer a cohesive method of capturing uncertainty, vagueness, or range associated with individual quantitative responses. In turn, IV data provide a strong foundation for subsequent fuzzy set based modelling-e.g., using the Interval Agreement Approach. The present paper focuses on the impact of soliciting IV estimates upon accuracy of group perceptual judgements-the 'Wisdom of the Crowd'. We report results from two empirical studies, examining the utility of IV data in the context of estimating specific (i.e., discrete point) ground truths, and directly comparing IV perceptual judgements (quantity estimates) against more traditional point estimates. There were two main hypotheses concerning the potential impacts of permitting uncertain (i.e., IV) estimates. First, it is possible that when specific predictions are required, permitting deliberately imprecise responses may reduce (prediction) accuracy versus forcing each respondent to provide their single 'best guess'. Second, that capturing the uncertainty associated with individual predictions should permit improved aggregation of group estimates, through weighting individual estimates according to their certainty. We report findings from two studies designed to investigate these hypotheses, and outline proposals for future research in this area.

Citation

Ellerby, Z., & Wagner, C. (2022). Does Permitting Uncertain Estimates Help or Hinder the Wisdom of Crowds?. In 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). https://doi.org/10.1109/FUZZ-IEEE55066.2022.9882802

• Conference Name: 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)
• Conference Location: Padua, Italy
• Start Date: Jul 18, 2022
• End Date: Jul 23, 2022
• Acceptance Date: May 23, 2022
• Online Publication Date: Jul 18, 2022
• Publication Date: Jul 18, 2022
• Deposit Date: May 24, 2022
• Publicly Available Date: May 24, 2022
• Publisher: IEEE
• Volume: 2022-July
• Book Title: 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)
• ISBN: 978-1-6654-6711-7
• DOI: https://doi.org/10.1109/FUZZ-IEEE55066.2022.9882802
• Public URL: https://nottingham-repository.worktribe.com/output/8219687
• Publisher URL: https://ieeexplore.ieee.org/document/9882802
• Related Public URLs: https://wcci2022.org/
• Additional Information: © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.