Skip to main content

Research Repository

Advanced Search

Tension in big data using machine learning: Analysis and applications

Wang, Huamao; Yao, Yumei; Salhi, Said

Tension in big data using machine learning: Analysis and applications Thumbnail


Yumei Yao

Said Salhi


© 2020 Elsevier Inc. The access of machine learning techniques in popular programming languages and the exponentially expanding big data from social media, news, surveys, and markets provide exciting challenges and invaluable opportunities for organizations and individuals to explore implicit information for decision making. Nevertheless, the users of machine learning usually find that these sophisticated techniques could incur a high level of tensions caused by the selection of the appropriate size of the training data set among other factors. In this paper, we provide a systematic way of resolving such tensions by examining practical examples of predicting popularity and sentiment of posts on Twitter and Facebook, blogs on Mashable, news on Google and Yahoo, the US house survey, and Bitcoin prices. Interesting results show that for the case of big data, using around 20% of the full sample often leads to a better prediction accuracy than opting for the full sample. Our conclusion is found to be consistent across a series of experiments. The managerial implication is that using more is not necessarily the best and users need to be cautious about such an important sensitivity as the simplistic approach may easily lead to inferior solutions with potentially detrimental consequences.

Journal Article Type Article
Acceptance Date Jun 17, 2020
Online Publication Date Jun 30, 2020
Publication Date Sep 1, 2020
Deposit Date Jun 30, 2020
Publicly Available Date Dec 31, 2021
Journal Technological Forecasting and Social Change
Print ISSN 0040-1625
Publisher Elsevier
Peer Reviewed Peer Reviewed
Volume 158
Article Number 120175
Keywords Big data; Machine learning; Data size; Prediction accuracy; Social media
Public URL
Publisher URL
Additional Information This article is maintained by: Elsevier; Article Title: Tension in big data using machine learning: Analysis and applications; Journal Title: Technological Forecasting and Social Change; CrossRef DOI link to publisher maintained version:; Content Type: article; Copyright: © 2020 Elsevier Inc. All rights reserved.


You might also like

Downloadable Citations