Tension in big data using machine learning: Analysis and applications

Wang, Huamao; Yao, Yumei; Salhi, Said

Abstract

© 2020 Elsevier Inc. The access of machine learning techniques in popular programming languages and the exponentially expanding big data from social media, news, surveys, and markets provide exciting challenges and invaluable opportunities for organizations and individuals to explore implicit information for decision making. Nevertheless, the users of machine learning usually find that these sophisticated techniques could incur a high level of tensions caused by the selection of the appropriate size of the training data set among other factors. In this paper, we provide a systematic way of resolving such tensions by examining practical examples of predicting popularity and sentiment of posts on Twitter and Facebook, blogs on Mashable, news on Google and Yahoo, the US house survey, and Bitcoin prices. Interesting results show that for the case of big data, using around 20% of the full sample often leads to a better prediction accuracy than opting for the full sample. Our conclusion is found to be consistent across a series of experiments. The managerial implication is that using more is not necessarily the best and users need to be cautious about such an important sensitivity as the simplistic approach may easily lead to inferior solutions with potentially detrimental consequences.

• Journal Article Type: Article
• Acceptance Date: Jun 17, 2020
• Online Publication Date: Jun 30, 2020
• Publication Date: Sep 1, 2020
• Deposit Date: Jun 30, 2020
• Publicly Available Date: Dec 31, 2021
• Journal: Technological Forecasting and Social Change
• Print ISSN: 0040-1625
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 158
• Article Number: 120175
• DOI: https://doi.org/10.1016/j.techfore.2020.120175
• Keywords: Big data; Machine learning; Data size; Prediction accuracy; Social media
• Public URL: https://nottingham-repository.worktribe.com/output/4739492
• Publisher URL: https://www.sciencedirect.com/science/article/pii/S0040162520310015
• Additional Information: This article is maintained by: Elsevier; Article Title: Tension in big data using machine learning: Analysis and applications; Journal Title: Technological Forecasting and Social Change; CrossRef DOI link to publisher maintained version: https://doi.org/10.1016/j.techfore.2020.120175; Content Type: article; Copyright: © 2020 Elsevier Inc. All rights reserved.