The Automatic Design of Hyper-heuristic Framework with Gene Expression Programming for Combinatorial Optimization problems

 Abstract —Hyper-heuristic approaches aim to automate heuristic design in order to solve multiple problems instead of designing tailor-made methodologies for individual problems. Hyper-heuristics accomplish this through a high level heuristic (heuristic selection mechanism and an acceptance criterion). This automates heuristic selection, deciding whether to accept or reject the returned solution. The fact that different problems or even instances, have different landscape structures and complexity, the design of efficient high level heuristics can have a dramatic impact on hyper-heuristic performance. In this work, instead of using human knowledge to design the high level heuristic, we propose a gene expression programming algorithm to automatically generate, during the instance solving process, the high level heuristic of the hyper-heuristic framework. The generated heuristic takes information (such as the quality of the generated solution and the improvement made) from the current problem state as input and decides which low level heuristic should be selected and the acceptance or rejection of the resultant solution. The benefit of this framework is the ability to generate, for each instance, different high level heuristics during the problem solving process. Furthermore, in order to maintain solution diversity, we utilize a memory mechanism which contains a population of both high quality and diverse solutions that is updated during the problem solving process. The generality of the proposed hyper-heuristic is validated against six well known combinatorial optimization problem, with very different landscapes, provided by the HyFlex software. Empirical results comparing the proposed hyper-heuristic with state of the art hyper-heuristics, conclude that the proposed hyper-heuristic generalizes well across all domains and achieves competitive, if not superior, results for several instances on all domains.


I. INTRODUCTION
The growth in the complexity and constraints of optimization problems that can be found in many real world applications makes them not only an ongoing challenge but also implies that they cannot be solved using exact methods within tractable (or acceptable) computational time [1], [2].Alternatively, meta-heuristic approaches, which offer no guarantee of returning an optimal solution (or even near optimal solutions), becomes not only a suitable option but also the only available option, as they usually return reasonably good solutions within a reasonable time.Although the efficiency of meta-heuristic approaches has been demonstrated over several real world applications, their success is due to the use of domain-specific knowledge [3], [4], [5].As a consequence, to solve a given problem by a meta-heuristic algorithm, practitioners usually have to face the problem of configuring the selected metaheuristic such as selecting the appropriate problem specific structures, most suitable operators and fine tuning the parameters, which are non-trivial tasks [6], [7] .
Over the years, it has become evident that the decision of which problem specific structures, operators and parameter values to be included (or excluded) in a given metaheuristic algorithm has an impact on algorithm performance [3], [8], [9], [10].Thus, to obtain a good quality solution, meta-heuristic approaches have to be expertly crafted by incorporating problem-specific knowledge of the underlying problem instance [3], [11].Customization of a meta-heuristic can be problem or even instance dependent and consequently will decrease its generality.Moreover, according to the No Free Lunch Theorem [12] no single algorithm with a unique configuration is able to perform well over all problem instances.As a consequence, when new problems are considered, meta-heuristics need to be (re)developed, which is usually not only time consuming but also requires a deep understanding of both algorithm behavior and the instance structure.Broadly speaking, at the expense of generality, researchers and practitioners have concentrated their effort on outperforming existing methods on one, or a few instances, by tailoring a given algorithm to the problem at hand.Arguably, meta-heuristic configuration plays a crucial role on the algorithm performance [5], [6].Furthermore, different problems require different configurations, and even for different instances of the same problem using a different configuration during the solving process could improve algorithm performance [7].When a search becomes trapped in a local optima, adapting the algorithm, on the fly, could help the algorithm to escape.Therefore, one way to design an effective search methodology is to take advantage of several operators as well as different parameter values by combining them in one framework or adjusting them during the solving process [13].Automated heuristic design has proven to be an efficient and effective way in enhancing the search methodology by adjusting algorithm operators or parameter values in on-line fashion [7], [13].These methodologies should work well, not only across different instances of the same problem, but also across several problem domains.Hyper-heuristics [3], parameter tuning [7], reactive search [14], adaptive memetic algorithms [9] and multi-method [15] are some of examples of automated heuristic design.Recently proposed frameworks in the automatic heuristic design concern the self-adaptation of search methodologies by coadapting algorithm configuration through coevolutionary process such as coadapted memeplexes [16], a theoretic model of symbiotic evolution [17] and the coevolving memetic algorithms [18].This work focuses on the hyper-heuristic framework.Hyper-heuristics are search methodologies that explore the search space of a given set of heuristics, or heuristic components in order to select the most appropriate heuristic.Hyper-heuristics can also be utilized to evolve new heuristic by combining basic component of existing heuristics.These features distinguish hyper-heuristics from meta-heuristic methods, as they operate directly on the solution space.The key motivation behind hyper-heuristics is to raise the level of generality and to combine the strength of several heuristics or heuristic components into one framework [3].
A traditional hyper-heuristic framework has two levels.The higher level heuristic manages which low level heuristic to call (heuristic selection mechanism) and then decides whether to accept the resultant solution (the acceptance criterion).The lower level contains a set of problem specific heuristics which are different for each problem domain.Since each instance has certain characteristics and landscape complexity, high level heuristic components have a dramatic impact on the hyperheuristic performance and that is why there is considerable research interest in devolving either new heuristic selection mechanisms or different acceptance criteria [3], [4].The design of a good high level heuristic would increase the ability of the hyper-heuristic in selecting the correct heuristic at any particular point, and a good acceptance criterion can guide the search process toward promising regions [19], [20].
Although the high level heuristic of a heuristic to choose heuristic hyper-heuristic framework, has been properly designed, one can argue that most of them have one (or a few) sensitive parameters and they have been manually designed by human experts [19].In addition, a manually designed high level heuristic needs considerable expertise and experience, and they only represent a small fraction of the overall search space.Furthermore, as far as we are aware, previous hyper-heuristic frameworks that have been proposed in the scientific literature [4], [19] are single solution based method.Reliance on a single solution may restrict their ability in dealing with huge and heavily constrained search spaces [10].
Therefore, we address the challenges of designing the high level heuristic components and of using a population of solutions in a hyper-heuristic framework by proposing the following (see Fig. 1): i) Instead of manually designing the high level heuristic of a perturbative heuristic to choose heuristics in a hyper-heuristic framework, we propose an automatic programming generation framework to automatically design the heuristic selection mechanism and the acceptance criteria by using gene expression programming [21] (denoted as GEP-HH).The proposed gene expression programming framework, see Fig. 1, is implemented as an on-line heuristic or rule generation method, which evolves a population of individuals.Each individual represents a set of rules that is decoded into a selection mechanism and acceptance criteria to be used by the hyper-heuristic framework.The quality of the generated rule is evaluated by inserting it into the hyper-heuristic framework and using it on a given problem instance for a certain number of iterations.We use the idea of controlling the population size in an evolutionary algorithm to measure the performance of the generated heuristic [22].We utilize gene expression programming algorithm to automate the design of the high level heuristic of the hyper-heuristic framework instead of genetic programming, due to its ability in avoiding code bloat and the fact that it generates a solution that is syntactically correct [21].ii) We utilize a memory mechanism, which contain a set of both high quality and diverse solutions, see Fig. 1, which is updated as the search progresses in order to enhance the ability of the perturbative heuristic to choose heuristics when dealing with heavily constrained problems in a huge search space, and also to diversify the search.
To our knowledge, the high level heuristic components of the currently existing hyper-heuristic frameworks are all manually designed and they are also single based solution methods.Hence, the proposed framework represents a paradigm shift in using an automatic program generation method in automating the design of hyper-heuristics or meta-heuristic components, as well as using a population of solutions instead of a single solution within the set of low level heuristics.This could reduce the human expertise required in manually customizing the high level heuristic of the hyper-heuristic framework and could also enhance the performance of the hyper-heuristic framework.Our research questions are: "Can we use a gene expression programming algorithm framework to generate high level heuristic components (heuristic selection mechanism and the acceptance criteria) of the hyper-heuristic framework?Does the use of a population of solutions, instead of a single solution, within the hyper-heuristic framework enhance the performance of the hyper-heuristics?" Thus, our objectives are: -To propose an on-line gene expression programming (GEP-HH) framework to automatically generate the high level heuristic components (heuristic selection mechanism and the acceptance criteria) of the hyperheuristic framework.
-To propose a population based hyper-heuristic framework by incorporating a memory mechanism which contains a set of solutions updated during problem solving progress in order to effectively diversify the search.
-To test the generality and the performance of the proposed hyper-heuristic framework over six different problem domains, of very different natures and compare the results with the state of the art hyper-heuristics.
We demonstrate the generality and the consistency of the proposed hyper-heuristic framework using the HyFlex (Hyper-heuristics Flexible Framework) software [23], which provides access to six problem domains with very different landscape structures and complexity.The domains are: boolean satisfiability (MAX-SAT), one dimensional bin packing, permutation flow shop, personnel scheduling, traveling salesman and vehicle routing.This work is among the first attempts to apply a hyper-heuristic framework to tackle all these challenging problems.Although it is entirely appropriate to have a bespoke method that can produce the best known results for one (perhaps more) instance, having a methodology which is generally applicable to more than one problems domain would be more beneficial.Our ultimate goal is not to propose a hyper-heuristic framework that can outperform the best known methods but rather propose a methodology that generalizes well over different problem domains.However, the results demonstrate that the proposed hyper-heuristic is able to update the best known results for some instances.

DESIGNING
As we have mentioned earlier, given an optimization problem and a solution method, researchers or practitioners have to address the problem of which problem specific structures, operators and parameter values to be used within the given solution method in order to achieve good quality results.Although algorithm configuration is intuitively appealing, usually it is very difficult, if not impossible, to manually search through all possible configurations such as adding or removing specific operators or adjusting the parameter values [24].Therefore, exploring such an interactive and large search space using other search methods (i.e.GEP, GP or other meta-heuristic algorithms) might yield a better performance compared to manually designing an algorithm [6] and this is actually what the automated heuristic design usually does.
Recently, automatic program generation methods, such as genetic programming (GP), have paved the way for a paradigm of optimizing or evolving the components of search methodologies.For example, GP has been employed in [25] to evolve the cooling schedule in simulated annealing to solve quadratic assignment problems.Whilst, in [26] GP has been utilized to generate constructive heuristics for the hyper-heuristic framework.It is also used in [27] to evolve the equation that controls the movement of particles in particle optimization algorithms.In [28] GP has been used to evolve the pheromone updating strategy for an ant colony algorithm.Recently a grammatical evolution (GE) algorithm has been utilized in [29] to evolve low level heuristics for the bin packing problem.Whilst, GE is used in [30] to automatically combine the high level heuristic components of the hyper-heuristic framework.Please note that the main difference between the proposed gene expression framework and the framework introduced in [30] is that the framework proposed in this paper generates a set of rules to select the most suitable low level heuristic and then either accepts or rejects the generated solution, whilst the framework in [30] combines existing metaheuristic acceptance criteria with neighborhood structures.Furthermore, the utilized terminal and function sets are fundamentally different.
However, despite the success of GP based hyperheuristics, the same hyper-heuristic cannot be used to generate heuristics for other domains such as exam timetabling or vehicle routing.That is, the function and terminal sets that have been defined for one domain cannot be used on other domains.In this work we propose an automatic program generation framework to automatically generate the high level heuristic of the hyper-heuristic IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION framework.The novelty of our proposed framework is that it can tackle many optimization problems using the same set of functions and terminals.This feature distinguishes our framework from existing GP based hyper-heuristics.In practice, evolving or optimizing algorithm components will not only alleviate user intervention in finding the most effective configuration, but also facilitate algorithm configurations.
Thus, if the automatic program generation methods can optimize meta-heuristic components [25], [28] and evolve the constructive heuristic of the hyper-heuristic framework [26], then using the automatic program generation method (GEP in this work) to automatically design the high level heuristic of the hyper-heuristic framework in an on-line manner may produce an effective hyper-heuristic framework.

III. RELATED WORK
Hyper-heuristics are one of the automated heuristic design methodologies motivated by the fact that different heuristics impose different strength and weakness.Thus it makes sense to merge them into one framework.A recent definition of a hyper-heuristics framework is "an automated methodology for selecting or generating heuristics to solve hard computational search problems" [3].Over the years, hyper-heuristic frameworks have demonstrated success in solving various classes of real world applications.A generic hyper-heuristic framework is composed of two levels known as high level and low level heuristics [3] (see Fig. 2).The high level heuristic is problem independent and has no domain knowledge.Its role is to manage the selection or generation of which heuristic are to be applied at each decision point.The low level heuristic corresponds to a pool of heuristics or heuristic components.

Heuristics Repository
Apply heuristic Domain-independent information acquisition and processing: change in a candidate solution quality, number of low level heuristics, measuring the performance of the applied heuristics, statistics, etc.

Domain Barrier
Fig. 2. A generic hyper-heuristic framework [3] Recently, hyper-heuristic frameworks have been classified [3] based on the nature of the heuristic search space and the source of feedback during learning (see Fig. 3).The source of feedback can be either on-line, if the hyper-heuristic framework uses the feedback obtained during the problem solving in decision making, or off-line, if the hyperheuristic framework uses information gathered during the training phase in order to be used when solving other or unseen instances.The nature of the heuristic search space is also classified into two subclasses known as heuristics to choose heuristics and heuristics to generate heuristics.In either case, this is often further classified based on the employed low level heuristics into: constructive heuristics, which starts from scratch and keeps extending a partial solution step by step until a complete solution is generated, or perturbative heuristics, which starts with a complete solution and iteratively refines it to improve its quality.

Low level heuristics
Constructive Constructive Purtrbative Purtrbative

Source of feedback during learning
On-line

No learning
Off-line Fig. 3. Classifications of hyper-heuristic approaches, according to two dimensions: (i) the nature of the heuristic search space and (ii) the source of feedback during learning [3].

A. Heuristics to choose heuristics
Most of hyper-heuristic frameworks published are heuristics to choose heuristics.These operate on a set of human designed heuristics called low level heuristics [19].
The set of low level heuristics can be either constructive or perturbative.The role of the hyper-heuristic framework is to intelligently select, from a given set of low level heuristics, which heuristic to apply at a given time.The motivation behind heuristics to choose heuristics is that the strength of several heuristics can be included in one framework.A traditionally perturbative heuristic based hyper-heuristic framework has two components, known as the heuristic selection mechanism and the acceptance criteria.The role of the selection mechanism is to select the low level heuristic from the given set, whilst, the acceptance criteria is to decide whether to accept or reject the resultant solution after applying the selected low level heuristic.Both components play an important role and have significant impact on hyper-heuristic performance [19], [20].Examples of heuristic selection mechanisms are tabu search [31], genetic algorithm [32], iterated local search and variable neighborhood [33].Examples of acceptance criteria that have been used within hyper-heuristics are simulated annealing, great deluge and tabu search [19].More details of these hyper-heuristics can be found in recent surveys [19], [4].The cross-domain heuristic search (CHeSC) competition has been recently introduced, which provides a common software interface for investigating different (high level) hyper-heuristics and provides access to six problem domains where the low level heuristics are provided as part of the supplied framework [23].The algorithm designer only needs to provide the higher level component (heuristic selection and acceptance criterion).The adaptive hyperheuristic (AdapHH) proposed in [34] was the competition winner.Their heuristic selection mechanism uses an adaptive dynamic heuristic set or relay hybridization and an IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION adaptive acceptance criterion.Further details about the competition, including further results, are available in [23].
Recently, Chen [35] introduced an algorithm development environment (ADEP) to address metaheuristic design and configuration problems through an integrated framework that allows both manual and automated configuration of a variety meta-heuristic approaches.The main difference between [35] and proposed GEP-HH framework is that the proposed GEP-HH framework generates meta-heuristic components instead of combining and/or configuring existing ones.
Although several types of heuristic selection mechanisms and acceptance criteria exist, no heuristic selection mechanisms or acceptance criteria so far presented are the best, or the most suitable, across all domains [19].In practice, all of them face generalization issues.This is because the choice of which heuristic to apply does not depend only on the problem instances but also on the current stage of the solving process, since at each decision point the problem instance landscape is acquiescent to at least one low level heuristic.Most of the current heuristic selection mechanisms use simple rules to select the low level heuristic based on their past performance [19].However, to quickly respond to instance landscape changes, a sophisticated heuristic selection mechanism may be needed.Furthermore, some low level heuristics perform well only at the beginning of the search process while others could be good at the end of solving process [19], [13].For example, the application of a certain local search based low level heuristic would be unuseful if the solution is already trapped in a local optima.As a result, there is a need for a high level heuristic that is more general than those currently available, that can use the problem state in selecting the appropriate low level heuristic, and can cope with several problem domains or even different instances of the same problem.
In this work, we address this challenge by proposing a gene expression programming framework to generate, for each instance, the heuristic selection mechanism and the acceptance criteria for the perturbative heuristic to choose heuristic.What makes our proposed framework different from others is that, at every iteration, the generated selection mechanism and acceptance criteria favor different criteria or information in selecting the low level heuristic and the acceptance of the generated solution.For example, the heuristic selection mechanism generated at iteration i may favor the selection of the low level heuristic that has very good performance during the previous application, whilst, the heuristic selection mechanism generated at iteration i+1 may favor the selection of the low level heuristic that has been more frequently applied than those of very good performance.

B. Heuristics to generate heuristics
In contrast to the heuristics to choose heuristics hyperheuristic, where the hyper-heuristic starts with a set of low level heuristics provided manually, in a heuristics to generate heuristics hyper-heuristic the aim is to fabricate new low level heuristics by combining existing heuristic components [3].Genetic programming has been successfully used to evolve constructive heuristics for SAT [36], scheduling [37] and bin packing problems [26].
Despite the fact that genetic programming hyperheuristics have achieved good results, one can argue that most of them are tailored to solve specific problems (e.g.SAT and the bin packing problems) using a restricted constructive heuristic component.Another limitation is that they have been used in an off-line manner which may restrict their generality because they will be tailor made to the training instances unless the testing instances have the same features and complexity which usually does not reflect many real world applications.
Motivated by the achievements of the above work, in this work, we propose a gene expression programming framework to automatically generate the high level heuristic for the perturbative heuristics to choose heuristics hyper-heuristic framework.The proposed gene expression framework can be classified as an on-line generational hyper-heuristic and thus the same as a genetic programming hyper-heuristic.The benefit of the proposed gene expression programming framework is its ability to use the current problem state to generate, for each instance, different high level heuristic in an on-line manner which could help the search in coping with the changes that might happen during the instance solving process.

IV. THE PROPOSED FRAMEWORK
The proposed hyper-heuristic framework has two levels called high level and low level heuristics.The high level heuristic contains two components, a heuristic selection mechanism and an acceptance criterion.The low level heuristic contains a set of perturbative low level heuristics, the memory mechanism and the objective function.The proposed hyper-heuristic starts with an initial solution, randomly selected from the memory mechanism, and iteratively explores its neighborhood by applying a perturbative low level heuristic.Given a pool of perturbative low level heuristics, a complete solution (randomly selected from the memory mechanism) and the objective function, the proposed hyper-heuristic framework will successively invoke the following steps for a certain number of iterations (defined by the user): i) Call the heuristic selection mechanism to select, from a given pool, one perturbative low level heuristic.ii) Randomly selects one solution for the memory mechanism.iii) Apply the selected perturbative low level heuristic to the given solution to generate a new solution.iv) Call the objective function to evaluate the generated solution.If it is better than the incumbent solution, replace it with the incumbent solution and continue the search.If not, call the acceptance criterion to decide either to accept or reject the generated solution according to the acceptance criterion rules.v) Update the memory mechanism, the parameters and start a new iteration.

A. High level heuristic
In the high level heuristic, the role of the heuristic selection mechanism is to select, for a given instance, the suitable perturbative low level heuristic from those supplied.Usually, the choice of which perturbative low level heuristic to be applied is a crucial decision, as this would lead the search in order not to confine it to a localized region of the solution space.The aim of the acceptance criterion is to assist the search process in order to avoid local optima and at the same time explore different areas of the search space through the decision of accepting or rejecting the generated solution [4].A good acceptance criteria mechanism should be able to strike a balance between accepting improving solutions and also worse solutions if the search is trapped in a local optima [20].Therefore, this work proposes a program generation method using gene expression programming to adaptively select the suitable low level heuristic and to balance between accepting and rejecting the generated solution (see Fig. 4).
Fig. 4. The proposed high level heuristic

1) Basic gene expression programming algorithm
Gene expression programming (GEP) [21] is a program generation method that uses a linear representation instead of a tree representation that is often used in genetic programming (GP).Each individual in GEP comprises a set of strings with a fixed size, called genomes.The program in GEP is generated by converting the individual string into a parse tree utilizing breadth-first search.The parse tree is then executed against the given problem instance.To generate a new individual, GEP applies genetic algorithm operators (crossover, mutation, inversion and transposition) directly on the linear encoding instead of the parse tree.Thus, GEP merges the advantages of both a genetic algorithm and genetic programming in evolving a population of computer programs.This feature allows GEP to generate programs that are always syntactically correct while avoiding the problem of code bloat (a well-known problem in traditional GP).The evolutionary steps of GEP in generating population of individuals are shown in Fig 5.
First, GEP components are defined.These are the function set (F) (which manipulates the values returned by terminals, and they take one or more arguments), terminal set (T) (which represents a set of nodes that form the leaf nodes of the program tree; they take no arguments), fitness function, GEP parameters and stopping condition.
Next, we generate a population of individuals.An individual in GEP is composed of a set of symbols called genes.Each gene has two elements called head and tail.Head contains both terminals and functions and its length h is fixed by users.The tail only contains terminals and its length t is calculated by the formula t=h*(n-1) +1, where n represents the maximum number of function arguments [21].Thus, the individual length is equal to h+t.fitness values.The selected individuals will go through the following genetic operators: i) Crossover: exchanges elements between two randomly selected genes from the chosen parents (e.g., one-point and two point crossover).ii) Mutation: change any string in the generated individual while making sure that the string in the head part can be changed into both terminal and function and, string in the tail part can be changed into terminals only.iii) Inversion: reveres small sequence of strings within the head or tail.iv) Convert the created individuals (offsprings) to program trees and execute them on the underlying problem to calculate their fitness values.v) Following roulette wheel (or other selection operators) sampling with elitism, the fittest individuals are always copied into the next generation.
This process is executed until the stopping condition is satisfied (e.g. a given number of generations).

2) The proposed gene expression programming framework to generate the high level heuristic components
In this work, we propose a gene expression programming framework to automatically generate the high level heuristic selection mechanism and the acceptance criteria, based on a given problem instance, for the perturbative heuristic to choose heuristic hyper-heuristic framework.This is an on-line heuristic generation method based hyperheuristic which iteratively evolves a population of individuals through the evolution process.Each individual represents a set of rules which are decoded into a selection mechanism and acceptance criterion to be used by the hyper-heuristic framework.To simultaneously generate both selection mechanism and the acceptance criterion, each individual is divided into two parts of equal size to represent both components.For example, in a individual of m strings, strings 1 to m/2 will be used for the selection mechanism and strings m/2 to m will be used for the acceptance criterion.Each part has a head of a user defined length h (contains terminal and function) and a tail (contains terminal only) of length t=h*(n-1) +1, where n represent the maximum number of function arguments.Each part employs the head-tail encoding method which ensures the validly of the generated program which represents one expression tree for the selection mechanism and acceptance criterion, respectively.Except crossover, genetic operators (mutation and inversion) can occur at any point as long as the gene rules are respected, i.e., a head element can be changed into terminal or function, whilst, a tail element can be changed into terminal only.Crossover operators will exchange elements between two randomly selected genes from the chosen parents within the same parts.For example, if the selected genes are from the first part of the first individual, these genes will be replaced with those in the first part of the second individual.This will ensure that the exchanged genes are the same types, i.e., either for the selection mechanism or the acceptance criterion.
To run the proposed gene expression programming framework, one needs to define the following components:

1-Terminal and function sets
A crucial issue in the design of the proposed framework is the definition of the terminal set (T) and the function set (F).The terminal set (T) represents a set of variables which will express the state of the underlying problems.The function set (F) represents a set of arithmetic or logical operators that will be used to connect or compose the terminal set (T).To use the proposed framework across various problems, we keep the definition of the terminal set (T) and function set (F) as general and simple as possible.By doing so, the proposed framework can be used across other problem domains, in addition to those considered in this work.Since the purpose of the heuristic selection mechanism is fundamentally different from the acceptance criterion, we use two terminal sets.The first set represents the selection mechanism, whilst, the second represents the acceptance criterion.
To cope with instance changes that might happen during the instance solving process, the proposed framework utilizes several evaluation criteria to represent the terminal sets in such a way that their combination will favor one criterion among others and these evaluation criteria will be updated during instance solving.Each evaluation criterion favors the selection of the low level heuristic from a different perspective.The rationale behind this is that some low level heuristics perform well only at the beginning of the search process while others could be better at the end of the process.Therefore, the heuristic selection mechanism should be able to quickly respond to instance landscape changes by selecting the appropriate low level heuristic.The function (F) and terminal (T) sets of the selection mechanism that have been used in this work are presented in Table 1.The utilized terminals for the heuristic selection are: -Reward credit (RC): The main idea of this reward is that infrequently used low level heuristics which lead to a large improvement in the solution quality are preferred to be selected more than those that lead to a small improvement.Thus, as a result, the low level heuristic which brings frequent, but small improvements will get less reward and consequently has a lesser chance of being preferred [13].

Cbest
The number of times that the i-th low level heuristic has updated the best known solution.

Ccurrent
The number of times that the i-th low level heuristic has updated the current solution.

Caccept
The number of times that the generated solution by the i-th low level heuristic has been accepted by the acceptance criterion.

Cava
The average of the previous improvement strength of the i-th low level over the search process.

Cr
The number of times that the i-th low level heuristic has been ranked the first.

2-Fitness function
The aim of the fitness function is to evaluate the performance of the generated high level heuristics (population individual).In this work, we use the idea in [22] that was used to control the population size in an evolutionary algorithm to evaluate the fitness of the generated high level heuristics.The probability of selecting each high level heuristic (an individual in the GEP framework) is updated according to the quality of the best solution returned, after the stopping condition is satisfied.The quality of the returned solution is usually either better or worse than the one that has been used as an input solution for the hyper-heuristic framework.Formally, let Ah[] represent the array of the probability of selecting the high level heuristics (individual), f i and f b represent the fitness of the initial and returned solutions, NoH represents the number of high level heuristics (individuals) or the population size of GEP.Then, if the application of the i-th high level heuristic leads to an improvement in the solution quality, then reward the i-th high level heuristic (individual) as follows: Other high level heuristics, j {1,…, NoH} and j ≠ i, are penalized as Ah[j] = Ah[j] -(∆/(NoH-1)).Otherwise (if the solution cannot be improved), then penalize the i-th high level heuristic, where α= Current_Iteration / Total_Iteration and reward other high level heuristics, j {1,…, NoH} and j ≠ i, Ah[j] =Ah[j] + (|∆|*α/(NoH-1)).Please note that the main idea behind decreasing the probability of other high level heuristic is to decrease their chances of being selected.Initially, the probability of each high level heuristic (individual) is calculated by translating them into expression trees and executing the corresponding program.

3-The stopping condition
In this work, the maximum number of consecutive non improvement iterations is used as the stopping condition (see section V.A).When all elements are defined, the proposed framework is carried out as follows (see Fig. 6): i) Generate a population of individuals.ii) Calculate the fitness of the population by inserting them into the hyper-heuristic framework and using it to solve a given instance for a certain number of iterations.iii) Iteratively selects two parents, apply crossover and mutation operators to generate two offspring, evaluate the fitness of the generated offspring and update the population.This is executed for a certain number of generations.
The main role of GEP is to evolve a population of individuals, each encoding a high level heuristic (selection mechanism and acceptance criterion) which will be used by the hyper-heuristic framework.The hyper-heuristic framework will be called at every generation to evaluate the generated offspring.When the proposed hyper-heuristic is called the following steps will be carried out: i) Decoded the current individual into a heuristic selection mechanism and an acceptance criterion, i.e., translate it into two expression trees for the selection mechanism and the acceptance criterion, respectively.Then, use the terminal (T) set value of each low level heuristic as the input for the selection mechanism expression tree.ii) Execute the selection mechanism expression tree and rank the given set of low level heuristics from the highest to the lowest based on the value retuned from the expression tree.iii) Randomly select one solution for the memory mechanism.Apply the highest ranked low level heuristic to the given solution and calculate the quality of the generated solution.iv) If the generated solution is better than the current one, the current one is replaced.If not, the hyperheuristic will call the acceptance criterion expression tree and execute the corresponding program.Then, the generated solution by the low level heuristic is accepted if the exponential of the value retuned by the acceptance criterion expression tree is less or equal to 0.5 (the exp function returns values between 0 and 1).In the literature, a value of 0.5 was suggested [26], but for different domains.The value 0.5 was also determined based on preliminary testing.v) Repeatedly apply the current low level heuristic until no improvement is returned.vi) If no improvement is returned, the hyper-heuristic framework will stop applying the current low level heuristic and restarts from the local optimum obtained by current low level heuristic, but with next low level heuristic in the ranked list.vii) If the hyper-heuristic framework reaches the end of the low level heuristic ranked list, it executes the current heuristic selection mechanism expression tree again and rank the given set of low level heuristics and restart the search from the local optimum, but using the current highest ranked low level heuristic.viii) The proposed hyper-heuristic framework will keep using the utilized high level heuristic components (selection mechanism and acceptance criterion), which is generated by the GEP framework, for a predefined number of iterations (see section V. A).

B. Low level heuristics
The low level heuristic of the proposed hyper-heuristic framework has three components as follows:

1) A set of perturbative low level heuristics
In this work, a pool of problem-specific perturbative heuristics is used as low level heuristics.The aim of the low level heuristics is to explore the neighborhoods of the current solution by altering the current solution (perturbation).The generated neighborhood solution is accepted if it does not break the imposed hard constraints and also satisfies the acceptance criterion.Thus, the employed low level heuristic explores only the feasible search space.Details of these perturbative heuristics are presented in the problem description sections (see section V.C).

2) Memory mechanism
Most hyper-heuristic frameworks that have been proposed in the scientific literature operate on a single solution [4], [19].Reliance on a single solution may restrict their ability in dealing with a large and heavily constrained search space, as it is widely known that single solution based methods are not well suited to cope with the large search spaces and heavily constrained problems [10].In order to enhance the efficiency of the proposed hyper-heuristic framework and to diversify the search, we embed it with a memory mechanism as in [38] which contains a collection of both high quality and diverse solutions, updated as the Calculate the fitness of S`1 and S`2.

Satisfied? Yes No
Terminate and return the best solution

Update the population
Call the Hyper-heuristic in Fig. 1.
Call the Hyper-heuristic in Fig. 1.
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION algorithm progresses.The integrated memory mechanism interacts with the high level heuristic as follows: first initialize the memory mechanism by generating a set of diverse solutions (randomly or by using a heuristic method, see Section V).For each solution, associate a frequency matrix to measure solution diversity.The frequency matrix stores the frequency of an object assigned to the same location.At every iteration, the high level heuristic will randomly select one solution from the memory; apply the selected low level heuristic to this solution, update both the solution in memory and the solution frequency matrix.
The associated frequency matrix is represented by a two dimensional array where rows represent objects and columns represent locations.For example, in the bin packing problem, the frequency matrix stores how many times the item has been assigned to the same bin.Whilst, in the vehicle routing problem, it stores how many times a customer has been assigned to the same route.In this work, objects represent the items in the bin packing problem or customers in the vehicle routing problem, while locations represent bins in the bin packing problem and routes in the vehicle routing problems.Fig. 7 shows an example of a solution and its corresponding frequency matrix.The frequency matrix is initialized to zero.We can see five objects (represented by rows, items or customers) and there are five available locations (represented by columns, bins or routes).The solution on the left side of Fig. 7 can be read as follows: object 1 is assigned to location 1, object 2 is assigned to location 3, etc.The frequency matrix on the right side of Fig. 7 can be read as follows: object 1 has been assigned to location 1 twice, to location 2 three times, to location 3 once, to location 4 four times and to location 5 once; and so on for the other objects.If any solution is used by the hyper-heuristic framework, then we update the frequency matrix of this solution.Next we calculate the quality and the diversity of this solution.In this work, the quality represents the quality of the solution of a given instance (see section V).The diversity is measured using the entropy information theory (see Equations ( 1) and ( 2)) as follows [38]: (2) Where -eij is the frequency of allocating object i to location j.
-m is the number of objects.
εi is the entropy for object i.
Next, add the new solution to the memory mechanism by considering the solution quality and diversity.

3) Objective function
The objective function is problem dependent and it measures the quality of the generated solution (see section V).

V. EXPERIMENTAL SETUP
In this section, we will discuss the parameter settings of GEP-HH, problem description and the perturbative low level heuristics of the considered problems.

A. GEP-HH Parameter Settings
Fine tuning the algorithm parameters for optimal performance is usually a tedious task that needs considerable expertise and experience [6].Therefore, the parameter values of the GEP-HH are obtained by using Relevance Estimation and Value Calibration method (REVAC) [39].REVAC is a tool for parameter optimization, where a steady state genetic algorithm and entropy theory are used in defining algorithm parameter values.REVAC is utilized to find the generic values that can be used for all considered domains instead of finding the optimal one which is problem (if not instances) dependent.
Taking into consideration the solution quality and the computational time needed to achieve good quality solutions, the running time for each instance is fixed to 20 seconds and the number of iterations performed by REVAC is fixed at 100 iterations (see [39] for more details).To do so, we tuned GEP-HH for each domain separately and then used the average of the minimum value for each parameter obtained by REVAC for all tested instances.Then the average values over all tested instances for all domains for each parameter are set as the generic values for GEP-HH.Table 3 lists the parameter settings of GEP-HH that have been used for all problem domains.GEP-HH.HyFlex is a java framework which provides six problem domains (boolean satisfiability (MAX-SAT), one dimensional bin packing, permutation flow shop, personnel scheduling, traveling salesman and vehicle routing), the initial solution generation method, and a set of perturbative low level heuristics [23].HyFlex was used during the crossdomain heuristic search challenge competition (CHeSC) in order to compare the performance of hyper-heuristic methods and to support researchers in their efforts to develop generally applicable hyper-heuristics for various problem domains.In addition, we also report in the appendix, the results of testing GEP-HH on exam timetabling and dynamic vehicle routing problems (See the supplementary file).

1) Boolean Satisfiability (MAX-SAT) Problems
Boolean Satisfiability problems can be defined as follows [40]: given a formula of Boolean variables, determine the assignment of truth values to the variables that can make the formula true.MAX-SAT, which is an extension of Boolean Satisfiability, is an optimization problem where the aim is to determine the maximum number of true clauses of a given Boolean formula.In other words, the aim of the optimization process is to minimize the number of unsatisfied clauses in a given formula.The instances that were considered in this work are summarized in Table 4.
The set of initial solutions are randomly generated by assigning either true or false value to each variable.The quality of the solution is measured based on how many `broken' clauses in a given formula i.e., those which evaluate to false.See [40] for more details.

2) One Dimensional Bin Packing Problems
The one dimensional bin packing is a well-known combinatorial optimization problem.Given a set of items of a fixed weight and a finite number of bins of fixed capacity, the goal is to pack all items into as few bins as possible [41].The packing process should respect the following constraints: each item should be assigned to one bin only and the total weight of items in each bin should be less or equal to the bin capacity.The aim of the optimization process is to minimize the number of bins that are used.Table 5 shows the characteristic of the considered instances.The set of initial solutions are generated as follows: first, generate a random sequence of items and then pack them one by one into the first bin which they will fit, i.e. "first fit heuristic".The quality of solution is measured by quality= where n is the number of bins, fl is the sum of the sizes of all the pieces in bin i, and C the bin capacity.See [41] for more details.

3) Permutation Flow Shop Problems
The permutation flow shop problem is defined as, while respecting the imposed constraints, find the sequence for a set of jobs to be processed on a set of consecutive machines with the minimal completion time of the last job to exit the shop [42].Each job requires a processing time on a particular machine.One machine can only process one job at a time.Jobs can be processed by only one machine at a time.The job ordering process should be respected and machines are not allowed to remain idle when a job is ready for processing.Table 6 shows the characteristic of the considered instances.The set of initial solutions are generated by using the NEH [42] algorithm which works as follows: first generate a random permutation of jobs and an empty schedule.Then, assign the first job in the permutation sequence into the schedule, second job into places 1 and 2; third job into places 1, 2 and 3, and so on.Each assignment should be fixed where the partial schedule has the smallest makespan time, i.e. completion time of the last job.The quality of solution represents the completion time of the last job in the schedule.See [42] for more details.

4) Personnel Scheduling Problems
Personnel scheduling is a well-known NP-hard problem.Given a set of employees of specific categories, a set of predefined periods (shifts) on a working day, and a set of working days; the aim of the optimization process is to assign each employee to specific planning periods to meet the operational requirements and satisfying a range of preferences as much as possible [43].Due to the variety of hard and soft constraints, which are different from one organization to another, the modeling and implementation is challenging.A unique general mathematical model to accommodate all related constraints does not exist.Table 7 gives the characteristics of the considered instances.The set of initial solutions are created by using a neighborhood operator which incrementally adds new shifts to the roster until all employees have been scheduled.The quality of the generated solutions is assessed based on how many soft constraints are satisfied.See [43] for more details.

5) Traveling Salesman Problems
The traveling salesman problem is a very popular combinatorial optimization problem [44].In its classic form, given a set of cities and their positions (pairwise distances), the aim is to find the shortest path where each city is visited only once and the path ends at the starting city.The aim of the optimization process is to minimize the traveling distance.Table 8 gives the characteristics of the considered instances.The set of initial solutions are created by randomly generating permutation sequences.The quality of solution is represented by the total distance of the route.

6) Vehicle Routing Problems
The vehicle routing problem is a well-known challenging combinatorial optimization problem [45].Given a set of customers associated with demand and serving time, and a fleet of vehicles with a maximum capacity, the aim is to design a least cost set of routes to serve all customers, where each vehicle starts and ends at the depot, the total demand of each route does not exceed the vehicle capacity, each customer is visited exactly once by exactly one vehicle during its time window(s).Table 9 shows the characteristics of the considered instances.The set of initial solutions are generated as follows: first create an empty route, then loop through all customers and add any one to the current route that does not violate any constraints.If no customer can be added to the current route, create a new route.The process is repeated until all customers have been assigned to a route.The quality of solution represents the total travel distance.Table 10 shows the total number of each type of the perturbative low level heuristics for the six problem domains [23].

VI. COMPUTATIONAL RESULTS AND DISCUSSION
This section is devoted to assess the performance of GEP-HH against other hyper-heuristic methods in the literature.Our aims are: -To assess the benefit of integrating the memory mechanism within GEP-HH.-To test the generality and consistency of GEP-HH over six different problem domains and compare it to the state of the art of hyper-heuristic methods.
In this work, we have carried out, for each problem domain, two sets of experiments: i) The first one compares the performance of the GEP-HH with the memory mechanism (GEP-HH) against GEP-HH without the memory mechanism (denoted as GEP-HH*) using the same parameter values and computational resources.ii) The second evaluates the performance of GEP-HH against the top five hyper-heuristics of the first crossdomain heuristic search challenge (CHeSC) [23].These are: AdapHH [34], VNS-TW [46], ML [47], PHUNTER [48] and EPH [49].Following CHeSC, rules and in order to make the comparison as fair as possible, for both experimental tests, the execution time is used as the stopping condition.It is determined by using the benchmark software provided by the organizers to ensure fair comparisons between researchers using different platforms [23].We have used this software to determine the allowed execution time using our computer resources (i.e. 10 minutes on the benchmark machine).The best, average, standard deviation and median of GEP-HH and GEP-HH* over independent 31 runs (adhering to the CHeSC competition rules) are reported for each instance.In addition, the percentage deviation from the best known value found in the hyper-heuristic literature is also calculated for each instance as follows: % * * (%) best best best HH GEP (3) where best GEP-HH is the best result returned over 31 independent runs by GEP-HH and best* is the best result obtained by other hyper-heuristic methods.
To demonstrate the generality, consistency and the effectiveness of GEP-HH across all tested problem domains, we have compared the performance of GEP-HH against GEP-HH* and existing hyper-heuristic methods based on generality, consistency, efficiency, statistical test and formula one (see [30] for more details).

A. The computational results of GEP-HH compared to GEP-HH*
The first set of experiments presents the comparison between GEP-HH and GEP-HH* across all of the six considered problems.Each problem domain contains 5 instances and the total number of tested instances is 30.The computational results of GEP-HH and GEP-HH* over 31 independent runs for the six problems are summarized in Table 11.
Observing the results reported in Table 11, we can make the following observations: in terms of solution quality, GEP-HH outperformed GEP-HH* on 18, tieing with GEP-HH* on 11 and being inferior on 1 (MAX-SAT Instances 4) out of 30 instances of the considered problem domains.From the average results perspective, it is clear that, across all instances of the considered problems domains, GEP-HH is the overall best.
In addition to the solution quality and the average results, it is natural to ask how consistent GEP-HH is, i.e., how likely GEP-HH would perform well over multiple runs on each instance compared to GEP-HH*.This question can be answered by analyzing the standard deviation and the median over 31 runs as well as the box-plots of solution distributions.In general, the standard deviation produced by GEP-HH is smaller than those from GEP-HH* for all instances of the considered problem domains (except PS 2 in Table 11).From the median perspective, we can draw the following conclusion: GEP-HH obtained better median results for 19, tieing on 3 and being slightly worse than GEP-HH* on 8 out of 30 instances of the considered problem domains.To save space, the box-plot figures (Figs. 8 to 13) are presented in the supplementary file.Figs. 8 to 13 show the box-plot of results distribution of GEP-HH and GEP-HH* for all instances of the considered problem domains, where one can clearly see that, for most instances, GEP-HH is more consistent than GEP-HH*.This indicates that GEP-HH is more consistent than GEP-HH* across all tested problem domains.
In addition to the above results, it is worth drawing some statistical significant conclusions regarding the performance of GEP-HH and GEP-HH*.Therefore, the Wilcoxon test (pairwise comparisons) with significant level of 0.05 is performed.The p-value of the Wilcoxon test of GEP-HH versus GEP-HH* are presented in the last column of Table 11.Where "S+" indicate GEP-HH is statistically better than GEP-HH* (p-value <0.05), "S-" indicate GEP-HH outperformed by GEP-HH* (p-value >0.05) and "~" indicate both algorithms have the same performance (pvalue =0.05).The results in Table 11 (last column) show that GEP-HH is statistically better than GEP-HH* on 23 instances, not statistically better than GEP-HH* on 5 and perform the same as GEP-HH* on 2 instances out of 30 tested instances of the considered problem domains.
To summarize, the results demonstrate that GEP-HH is better than GEP-HH* in term of consistency, efficiency and generality (with regards to the tested instances of the considered problem domains).This is mainly due to the use of memory mechanism within GEP-HH which has a positive effect on the ability of GEP-HH in producing good quality and consistent results compared to GEP-HH*.

B. The computational results of GEP-HH compared to other hyper-heuristic methods
We now assess the performance GEP-HH versus the top five hyper-heuristic methods from the CHeSC competition [23] (AdapHH, VNS-TW, ML, PHUNTER and EPH) from the best and median results perspective.In addition, we have also included the results of GEP-HH* (without memory) in the comparison to assess its ability in producing good quality solutions compared to the top five hyper-heuristic methods from the CHeSC competition.Table 12 present the best, percentage deviation and instances ranking results for the six problems obtained by GEP-HH along with a comparison with respect to the best result of top five hyper-heuristic methods from the CHeSC competition.Please note that all the compared methods (GEP-HH, GEP-HH* and the top five hyper-heuristics) used the 10 minute execution time as the stopping condition which is determined by the benchmark software provided by the CHeSC organizers.
The results in Table 12 suggest that, out of 30 instances, GEP-HH outperformed the top five hyper-heuristic methods on 12 instances, match the best results on 12 instances and is inferior on 6 instances.We can also remark that GEP-HH without memory mechanism (GEP-HH*) manages to produce new best results for 6 instances and tieing on 12 out of 30 instances compared to the top five hyper-heuristic methods.
In Table 13, we provide the median, percentage deviation and instances ranking results achieved by GEP-HH in comparison with the median results obtained by the top five hyper-heuristic methods from the CHeSC competition as well as GEP-HH* median results.It is clear from Table 13 that, GEP-HH obtained better median results for 4 instances and tie with other hyper-heuristic methods on 8 out of 30 instances.Table 13 also show that GEP-HH without memory mechanism (GEP-HH*) obtained better median results for 1 instance and matched the best in 6 out of 30 instances of the considered problem domains.
To summarize, even though GEP-HH did not manage to obtain the best results for all instances, the percentage deviation of these instances is, however, relatively small and GEP-HH achieved the second best and third best results for other instances.One can clearly see that both GEP-HH and GEP-HH* have generalized well across all tested domains and produced good quality results compared to the top five hyper-heuristic methods in the existing literature.i) In the first comparison we used Formula one that was used in the CHeSC competition [23] to calculate the score of GEP-HH and the top five hyper-heuristic methods.Table 14 shows the overall rankings of GEP-HH and the top five hyper-heuristic methods (the higher the better).We also included GEP-HH* in the comparisons.It is interesting to note that GEP-HH obtained the first rank, whilst, GEP-HH* obtained the third rank compared to the top five hyper-heuristic methods.performed Friedman and Iman-Davenport tests with a critical level of 0.05 to detect whether there are statistical differences between the results of these methods.The p-value of Friedman (p-value = 0.000) and Iman-Davenport (p-value =0.000) are less than the critical level 0.05, which implies that there is a significant difference between the compared methods.
As a result, we conducted a Friedman test to calculate the average ranking of each method.Table 15 summarizes the average ranking (the lower the better) produced by the Friedman test for each method.It is obvious that, GEP-HH ranked the first, followed by AdapHH, GEP-HH*, ML, VNS-TW, PHUNTER and EPH.Overall, the advantages of the proposed framework are the ability to utilize the information about the current state during instance solving to automatically generate the heuristic selection mechanism and an acceptance criterion.
Results demonstrate that it provides a general mechanism regardless of the nature and complexity of the instances and can be applied to other domains without many changes (i.e. the user only needs to change the low level heuristics).Applying a methodology to other problem domains or even different instances of the same problem usually requires a considerable amount of modification (e.g.change algorithm parameters or structures).Our GEP-HH provides automated heuristic method that can cope with not only different instances of the same problem, but we have demonstrated its generality across six different problem domains.We would hope that the proposed methodology would also generalize to other domains.

VII. CONCLUSIONS
In this work, we have proposed a new hyper-heuristic framework for combinatorial optimization problems.At the higher level, we have introduced a gene expression programming framework to automatically generate the high level heuristic of the hyper-heuristic framework.The proposed gene expression programming framework evolves a population of individuals and each one is decoded into a heuristic selection mechanism and an acceptance criterion.
The evolved heuristic selection mechanism takes the current state as input (pervious performance) and decides which low level heuristic is to be applied.Then, the generated solution is accepted if it satisfies the evolved acceptance criterion.At the lower level, we employed a set of human designed perturbative low level heuristics to perturb the solution of a given instance.To diversify the search, we have embedded the proposed hyper-heuristic with a memory mechanism, which contains a set of high quality and diverse solutions, which are updated during the search.
We have shown that gene expression programming algorithm can be effectively used to automatically generate the high level heuristics of the perturbative hyper-heuristic framework.The efficiency, consistency and the generality of GEP-HH is demonstrated across six challenging problems using HyFlex software.The experimental results demonstrate that GEP-HH achieves highly competitive results, if not superior, and generalizes well over six problem domains (MAX-SAT, one dimensional bin packing, permutation flow shop, personnel scheduling, traveling salesman and vehicle routing problems) when compared to GEP-HH without a memory mechanism as well as the top five hyper-heuristic methods from the CHeSC competition.The main contributions of this work are: 1-The development of a GEP-HH hyper-heuristic framework that automatically generates, during instance solving process, the high level heuristic (heuristic selection mechanism and the acceptance criteria) of the improvement based hyper-heuristic framework.
2-The development of a population based hyper-heuristic framework that uses a memory mechanism of a set of solutions, which is updated during the solving process to effectively diversify the search.
3-The development of a hyper-heuristic framework that is not customized to specific problems class and can be applied to different problems without much development effort.
In our future work, we intend to investigate the effectiveness of the GEP-HH across other combinatorial optimization problems.

Fig. 5 .
Fig. 5. Basic gene expression programming flowchartThen, we calculate individual fitness as follows: following the breadth-first manner individuals are converted into expression trees.First, scan the individual string one by one from left to right.The first string will form the node of the tree and other strings are written in a left to right manner at each lower level.If the scanned string is a function (F) with n (n>=1) arguments, then the next n strings are attached below it as its n children.Otherwise, it will form a leaf of the corresponding tree (terminal (T)).The scanning process is repeated until all leaves in the corresponding tree are terminals only.Next, the program trees are executed on the underlying problem and their fitness values are calculated.Next, two individuals are selected by the selection mechanism (e.g.roulette wheel selection) according to their Select two solutions (S1 and S2)Apply crossover, mutation and inversion on S1 and S2 to generate S`1 and S`2.
The Automatic Design of Hyper-heuristic Framework with Gene Expression Programming for Combinatorial Optimization problems Nasser R. Sabar, Masri Ayob, Graham Kendall, Senior Member, IEEE and Rong Qu, Senior Member, IEEE IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION This terminal is good in reducing the heuristic search space by only favoring certain low level heuristics.-Update the best known solution counter (C best ): This terminal favors the low level heuristic that manage to update the best known results.This terminal is good in systematically improving the current local optima.TRANSACTIONS ON EVOLUTIONARY COMPUTATION -Update counter of accept solution (C accept ): This terminal favors the low level heuristic that produces a solution that is accepted by the acceptance criterion.This terminal is good in helping the search to escape from a local optima.-Update the average improvement counter (C ava ): This terminal favors the low level heuristic that has made a large improvement on average.This terminal is good at focusing the search on the current area in the search space.-Update the first rank counter (C r ): This terminal favors the low level heuristic that has been selected first.This terminal is good for applying the current low level heuristic.Please note that the terminal (T) set of the heuristic selection mechanism is used for the low level heuristic and their value together with function (F) set are used to rank the low level heuristics.
-Update the counter of accepting current solution (C current ): This terminal favors the low level heuristic that manages to update the current solution.This terminal is good in keeping the search focused around the current local solution.IEEE

TABLE 1 THE
TERMINAL AND FUNCTION SET OF THE SELECTION MECHANISM Terminals

set for the heuristics selection mechanism terminal description
RC The extreme value-based reward is used to calculate the credit (CA) for each low level heuristic.When the i-th low level heuristic is applied, its corresponding improvement to the current solution is computed.The improvement gained is then saved for the i-th low level heuristic in a sliding time window of size W, following the rule of FIFO.The credit of any low level heuristic is then set as the maximum value in its corresponding sliding window W.In this work, the improvement gained (PI) from the i-th low level heuristic is calculated as follows: PI(i) =(/f1-f2/f1)*100 if f2<f1.Where f1 is the quality of the current solution and f2 is the quality of the resultant solution after applying the i-th low level heuristic.

TABLE 3
B. Problem DescriptionIn this work, we used HyFlex (Hyper-heuristics Flexible Framework) to test the generality and the performance of 10.1109/TEVC.2014.2319051,IEEE Transactions on Evolutionary Computation IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION

TABLE 9 THE
VEHICLE ROUTING PROBLEMS INSTANCES

TABLE 11 THE
RESULT OF GEP-HH COMPARING TO GEP-HH* FOR ALL PROBLEM DOMAINS

TABLE 12 THE
BEST RESULT OF GEP-HH and GEP-HH* COMPARING TO THE TOP FIVE HYPER-HEURISTICS The numerical results presented throughout this work demonstrate that, across six very different combinatorial optimization problems, GEP-HH achieved favorable results compared to the top five hyper-heuristic methods from the CHeSC competition.More importantly, out of the 30 instances GEP-HH matched the best results for 12 instances and manages to obtain new best results for 12 instances.In all domains, the standard deviation and the percentage deviation of GEP-HH reveal that GEP-HH results are stable and very close to the best results obtained by other hyperheuristic methods.These results are also supported by statistical tests and box-plots of solution distribution.In order to compare the performance of GEP-HH against the top five hyper-heuristic methods from the CHeSC competition (AdapHH, VNS-TW, ML, PHUNTER and EPH) more accurately, we have conducted the following comparison:

TABLE 14
In the second comparison, we conducted a multiple comparison statistical tests between GEP-HH and the top five hyper-heuristic methods.To do so, we 10.1109/TEVC.2014.2319051,IEEE Transactions on Evolutionary Computation IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION