Objective Assessment of Subjective Tasks in Crowdsourcing Applications

Haralabopoulos, Giannis; Tsikandilakis, Myron; Torres, Mercedes Torres; Mcauley, Derek

Abstract

Labelling, or annotation, is the process by which we assign labels to an item with regards to a task. In some Artificial Intelligence problems, such as Computer Vision tasks, the goal is to obtain objective labels. However, in problems such as text and sentiment analysis, subjective labelling is often required. More so when the sentiment analysis deals with actual emotions instead of polarity (positive/negative). Scientists employ human experts to create these labels, but it is costly and time consuming. Crowdsourcing enables researchers to utilise non-expert knowledge for scientific tasks. From image analysis to semantic annotation, interested researchers can gather a large sample of answers via crowdsourcing platforms in a timely manner. However, non-expert contributions often need to be thoroughly assessed, particularly so when a task is subjective. Researchers have traditionally used 'Gold Standard', 'Thresholding' and 'Majority Voting' as methods to filter non-expert contributions. We argue that these methods are unsuitable for subjective tasks, such as lexicon acquisition and sentiment analysis. We discuss subjectivity in human centered tasks and present a filtering method that defines quality contributors, based on a set of objectively infused terms in a lexicon acquisition task. We evaluate our method against an established lexicon, the diversity of emotions-i.e. subjectivity-and the exclusion of contributions. Our proposed objective evaluation method can be used to assess contributors in subjective tasks that will provide domain agnostic, quality results, with at least 7% improvement over traditional methods.

Citation

Haralabopoulos, G., Tsikandilakis, M., Torres, M. T., & Mcauley, D. (2020, May). Objective Assessment of Subjective Tasks in Crowdsourcing Applications. Presented at Language Resources and Evaluation Conference (LREC 2020), Marseille, France

• Presentation Conference Type: Edited Proceedings
• Conference Name: Language Resources and Evaluation Conference (LREC 2020)
• Start Date: May 11, 2020
• End Date: May 16, 2020
• Acceptance Date: Mar 15, 2020
• Online Publication Date: May 11, 2020
• Publication Date: 2020-05
• Deposit Date: Mar 29, 2021
• Publicly Available Date: Apr 21, 2021
• Pages: 15-25
• Series Title: Language Resources and Evaluation Conference Proceedings
• Book Title: Proceedings of the 12th Language Resources and Evaluation Conference
• ISBN: 9791095546597
• Keywords: Natural Language Processing; Crowdsourcing; Lexicon; Subjectivity; Objectivity
• Public URL: https://nottingham-repository.worktribe.com/output/4554093
• Related Public URLs: http://www.lrec-conf.org/proceedings/lrec2020/workshops/cllrd2020/index.html
• Additional Information: Proceedings of the LREC 2020 Workshop "Citizen Linguistics in Language Resource Development"