A preliminary study of the feasibility of global evolutionary feature selection for big datasets under Apache Spark

Galar, M.; Triguero, I.; Bustince, H.; Herrera, F.

Abstract

Designing efficient learning models capable of dealing with tons of data has become a reality in the era of big data. However, the amount of available data is too much for traditional data mining techniques to be applicable. This issue is even more serious when evolutionary algorithms are a key part of the learning algorithm. In this scenario, one typical approach is to follow a divide-and-conquer strategy, where data is divided into different chunks that are individually and independently addressed. Afterwards, the partial knowledge obtained from each chunk of data is combined in order to give a solution to the problem. Nevertheless, these kinds of local approaches do not look at data as a whole, missing a global view of the problem, which may result in less accurate models that also depend on how data is split. In this work, we focus on evolutionary feature selection algorithms. A divide-and-conquer approach to handle evolutionary feature selection in big data was already developed. We aim at designing its global counterpart, which looks at the feature selection problem from a global perspective, making use of the data as a whole to select the most appropriate features. In order to do so, we consider Apache Spark as a big data technology where our algorithm is implemented. We design a genetic algorithm capable of dealing with big datasets by selecting the proper parameters for our base algorithm (the well-known CHC) and adapting the evaluation procedure to take all the distributed data into account. Several preliminary results are discussed to study the feasibility of global evolutionary feature selection methods for big datasets.

Citation

Galar, M., Triguero, I., Bustince, H., & Herrera, F. (2018). A preliminary study of the feasibility of global evolutionary feature selection for big datasets under Apache Spark. In 2018 IEEE Congress on Evolutionary Computation (CEC). , (1-8). https://doi.org/10.1109/CEC.2018.8477878

• Conference Name: 2018 IEEE Congress on Evolutionary Computation (CEC)
• Start Date: Jul 8, 2018
• End Date: Jul 13, 2018
• Acceptance Date: Mar 15, 2018
• Online Publication Date: Oct 4, 2018
• Publication Date: Jul 12, 2018
• Deposit Date: Oct 18, 2018
• Publicly Available Date: Oct 18, 2018
• Publisher: Institute of Electrical and Electronics Engineers
• Pages: 1-8
• Book Title: 2018 IEEE Congress on Evolutionary Computation (CEC)
• Chapter Number: N/a
• ISBN: 9781509060177
• DOI: https://doi.org/10.1109/CEC.2018.8477878
• Public URL: https://nottingham-repository.worktribe.com/output/1174978
• Publisher URL: https://ieeexplore.ieee.org/document/8477878
• Additional Information: © 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.