Capturing richer information: On establishing the validity of an interval-valued survey response mode

Ellerby, Zack; Wagner, Christian; Broomell, Stephen B.

Abstract

Obtaining quantitative survey responses that are both accurate and informative is crucial to a wide range of fields. Traditional and ubiquitous response formats such as Likert and visual analogue scales require condensation of responses into discreteor point values—but sometimes a range of options may better represent the correct answer. In this paper, we propose an efficient interval-valued response mode, whereby responses are made by marking an ellipse along a continuous scale. We discuss its potential to capture and quantify valuable information that would be lost using conventional approaches, while preserving a high degree of response efficiency. The information captured by the response interval may represent a possible response range—i.e., a conjunctive set, such as the real numbers between 3 and 6. Alternatively, it may reflect uncertainty in respect to a distinct response—i.e., a disjunctive set, such as a confidence interval. We then report a validation study, utilizing our recently introduced open-source software (DECSYS), to explore how interval-valued survey responses reflect experimental manipulations of several factors hypothesised to influence interval width, across multiple contexts. Results consistently indicate that respondents used interval widths effectively, and subjective participant feedback was also positive. We present this as initial empirical evidence for the efficacy and value of interval-valued response capture. Interestingly, our results also provide insight into respondents’ reasoning about the different aforementioned types of intervals—we replicate a tendency towards overconfidence for those representing epistemic uncertainty (i.e., disjunctive sets), but find intervals representing inherent range (i.e., conjunctive sets) to be well-calibrated.

Citation

Ellerby, Z., Wagner, C., & Broomell, S. B. (2022). Capturing richer information: On establishing the validity of an interval-valued survey response mode. Behavior Research Methods, 54, 1240-1262. https://doi.org/10.3758/s13428-021-01635-0

• Journal Article Type: Article
• Acceptance Date: May 18, 2021
• Online Publication Date: Sep 7, 2021
• Publication Date: 2022
• Deposit Date: Jun 18, 2021
• Publicly Available Date: Sep 7, 2021
• Journal: Behavior Research Methods
• Print ISSN: 1554-351X
• Electronic ISSN: 1554-3528
• Publisher: Springer Verlag
• Peer Reviewed: Peer Reviewed
• Volume: 54
• Pages: 1240-1262
• DOI: https://doi.org/10.3758/s13428-021-01635-0
• Keywords: General Psychology; Psychology (miscellaneous); Arts and Humanities (miscellaneous); Developmental and Educational Psychology; Experimental and Cognitive Psychology
• Public URL: https://nottingham-repository.worktribe.com/output/5690661
• Publisher URL: https://link.springer.com/article/10.3758/s13428-021-01635-0