Dispersion of relative importance values contributes to the ranking uncertainty: sensitivity analysis of Multiple Criteria Decision-Making methods

Multiple Criteria Decision-Making (MCDM) methods are widely used in research and industrial applications. These methods rely heavily on expert perceptions and are often sensitive to the assumptions made. The reliability and robustness of MCDM analysis can be further tested and verified by a computer simulation and sensitivity analysis. In order to address this, five different MCDM approaches, including Weighted Sum Model (WSM), Weighted Product Model (WPM), revised Analytic Hierarchy Process (rAHP), Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and COmplex PRoportional ASsessment (COPRAS) are explored in the paper. Real data of the case study for assessing housing affordability are used for testing the robustness of alternative ranking and finding the most sensitive criteria to the change of criterion weight. We identify the most critical criteria for any and best ranking alternatives. The paper highlights the significance of sensitivity analysis in assessing the robustness and reliability of MCDM outcomes. Furthermore, randomly generated and model-based data sets are used to establish relationship between the dispersion of relative importance values of alternatives and ranking uncertainty. Our findings demonstrate that the dispersion of relative importance values of alternatives correlate with the Euclidian distances of aggregated values. We conclude that the dispersion of relative importance values contributes directly to the ranking uncertainty and can be used as a measure for identifying critical criteria.


Introduction
Decision-making determines the success and failure in many areas of human activity. It becomes a difficult task in the situation that requires handling large amounts of data and information. In the light of this problem, the Multi-Criteria Decision Analysis (MCDA) has widely emerged as an important domain of operations research aimed at assisting with the decision-making. Multiple Criteria Decision-Making (MCDM) methods have found their use in the evaluation of alternatives, involving many and often conflicting decision criteria, with the aim of establishing an optimal or best suited choice by ranking presented alternatives [1,2].
The MCDA has also been extended to estimate a numerical solution, e.g. the prediction of a value [3,4] and implemented to facilitate decisions in business, management, engineering and industry [5][6][7][8]. MCDM methods were identified as particularly suitable for decision-making in fields such as the urban regeneration [9], urban and housing market sustainability [10,11], sustainable housing affordability [12], real estate valuation [3,13], wind energy [14], waste and water resource management [15,16], building site planning and materials [17], inventory classification [18], mining industry [19] and many others. For this purpose, a great variety of MCDM methodologies has been developed in last few decades [20][21][22]. It has also been recognised that none of the methods can be considered as the "best" or "most suitable" for a single or diverse problems and applications [20,23].
MCDM problems can be divided into two categories, discrete and continuous [21]. The later usually involves more than one objective function, which are sought to be optimised simultaneously often using evolutionary algorithms and is named Multi-Objective Optimization (MOO) [24,25].
Whereas, the former described as a multi-attribute discrete option often consists of a modest number of alternatives, often is aimed at achieving one objective and is termed Multi-Attribute Decision Making (MADM) [21,26].
The key stages of MCDM methodologies include: 1) establishing the relevant criteria and alternatives; 2) determining the numerical values (weights) concerning the relative importance of the criteria; 3) determining the impacts of the alternative option on criteria; and 4) processing the numerical values in determining a ranking of each alternative [1].
The determination of the relative importance of different criteria is a complex problem and often becomes a very influential step in MCDA. The information for determining the weights of the relative importance for different criteria is difficult to obtain for the definition of weights itself and the precise input data are rarely if ever available [27]. Often, experts are employed in addressing this issue and to help determine the required parameters. Yet, the professional expert opinion and perception of the criteria importance can vary depending on the sector of industry and other similar factors [28].
Ambiguities in the input data cause variability (uncertainty) and imprecision of the results (output) [29].
It has been shown that different methods, if applied to the same problem, can return diverse results [30][31][32][33][34][35][36], suggesting that the choice of method may also contribute to the inconsistency of the output. Therefore, depending on the MCDM method, the importance of the criteria can have a quite varied influence on the output of the analysis.
Whereas the usefulness of any model depends on the precision and reliability of its results [29], it is highly desirable to develop MCDM methods, which are less sensitive to the influence of the subjective assumptions made through the determination of the relative importance of the criteria, or to build strategies helping to assess the sensitivity of the model and the uncertainty of the result. To assist with this, the use of the sensitivity analysis has become widespread in many areas of sciences and is now widely acknowledged as a critical step in verifying the reliability and accuracy of the methodology.
The sensitivity analysis is performed with a number of objectives: 1) to establish parameters, which require additional investigation to improve the knowledge base and thereby to reduce the result (output) variability (uncertainty); 2) to establish parameters that are insignificant and can be disregarded in the final model; 3) to determine data (inputs) that contribute most to the output uncertainty; 4) to identify parameters, which have most effect on the output; and 5) when the model is in operation, to calculate what the effect has on the change of a given input parameter on the output [37]. To date, a limited amount of research has been dedicated to the sensitivity analysis for MCDM models mostly reported in 1980s and 1990s [38][39][40][41][42]. Previous studies mainly explore the dependency of output on the input parameters by helping to identify criteria that are most sensitive to the change of weight and have significant impact on the MCDM outcomes. The statistics of input parameters has not been considered so far. This paper focuses on five commonly used MCDM models, the Weighted Sum Model (WSM), the Weighted Product Model (WPM), the revised Analytic Hierarchy Process (rAHP), Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), and Complex Proportional Assessment (COPRAS) [1,30,[43][44][45]. By building on previous research [12,46] and using real data from a case study on the assessment of housing affordability, the sensitivity analysis is performed identifying tolerable change of criteria weights and how change of criteria weights contribute to the ranking output of alternatives. The robustness and reliability of MCDM analysis results is evaluated. Moreover, the correlation between the dispersion of relative importance values of alternatives and either sensitivity of criteria, dissimilarity of aggregated values, or dissimilarity of ranking is investigated.

1. Data collection
The criteria were identified by conducting interviews and literature review as described previously [47].
A total of 18 decision criteria were utilized in the assessment of sustainable housing affordability, as an empirical case study. A questionnaire-centred survey was performed and criteria weights wj were calculated as previously [12]. The criteria weights enabled to express the relative importance of the criteria (Table 1). Eleven residential housing areas (alternatives A1 to A11) in Liverpool, UK, were selected for empirical data collection the eighteen decision criteria. Openly available data from various sources as reported in [46] were used to calculate the relative importance aij for each alternative Ai in terms of criterion Cj. Obtained values are presented in Table 1.
The initial data collection process can be summarised in the following steps: 1) establishing criteria for the comprehensive assessment of the affordability of sustainable housing, which was achieved through the literature review and interviews with professionals; 2) determining criteria weights to reflect their importance, which were determined by experts; 3) selecting alternatives to be compared; and 4) computing relative importance values for each alternative. Table 1. Initial matrix for MCDM *The sign (+/-) indicates that a greater/lesser criterion value satisfies sustainable housing affordability

Sensitivity analysis
The sensitivity analysis was performed by applying two approaches. In the first, the analysis was performed to quantify the level of crosstalk between criteria and ranking. The methodological principles of this sensitivity analysis approach were described in [48]. By using this approach, the objective was to determine independently the effect of each criteria on the MCDM outcomes. The weight of each criterion was independently changed by 5% (small change) or 50% (large change) by increasing or decreasing it. The remaining criteria weights were kept unchanged. Obtained specific values of relative sensitivity coefficients indicate a number of changes in alternative ranking due to the change of criterion weight (5 or 50% increase/decrease). Inverse application of this approach enabled to determine a tolerable change of criteria weights as a percentage of allowed change (increase or decrease) for each criterion, which has no effect of alternative ranking result. This approach, in combination with the computer simulation, was used to study how the alteration of criteria weights contributes to the ranking of alternatives.
In the second sensitivity analysis approach, the "most critical criterion" was defined as the criterion Cj for which the smallest relative change (in percentage), denoted as Dj, in its weight value Wj must occur to alter the existing ranking of the alternatives [38]. The sensitivity coefficient of criterion Cj, was denoted as SCj, and was used as a measure of the sensitivity to the change of criterion weight as follows: For simulation purposes, the uniformly distributed pseudorandom numbers were generated using MATLAB function rand. Since the number of simulations depends on the quantity of input parameters, the limited number of alternatives (11) and defined number (12) of random number intervals were used to reduce computational costs.

MCDM models
Since there is no single method considered to be the most appropriate for all situations of decisionmaking, a large number of MCDM models and their derivatives have been developed [21] and the search for the "best" method continues. However, if applied to the same problem, different MCDM methods can deliver dissimilar results. As the method that is suitable for all types of decision-making has not been yet developed [20], the paradox of the selection of a suitable MSDM method continues [1]. The identification and selection of an appropriate MCDM method is therefore not a simple task and a considerable consideration must be given to the choice of method [20,35,49].
The affordability of sustainable housing in eleven residential areas was ranked by applying the five most common MCDM models, Weighted Sum Model (WSM), Weighted Product Model (WPM), revised Analytic Hierarchy Process (rAHP), Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and COmplex PRoportional Assessment (COPRAS) [26,30,43,45]. These were applied to the data from a real case study and provide the initial decision-making matrix (Table 1).
Using each model, the aim was to determine the relative importance of each alternative (aij) in terms to each criterion and to compute the aggregated value for each alternative (Ai WSM , Ai WPM , Ai rAHP , Ai TOPSIS , Ai COPRAS ) establishing the overall ranking order of alternatives. Since input data (xij) often are expressed in different units of measurement (e.g. ratio, points, percentage, price), and in order to ensure that the data are non-dimensional, the data normalisation for WSM, rAHP, and COPRAS was performed following principles suggested in previous studies [50,51] and as follows: where, aij is a value of relative importance (weight) for the ith alternative in terms of the jth criterion.
For WSM, this transformation allowed to replace multidimensional criteria, which are not permitted for use with this model, with the measures that are both numerical and comparable, and expressed in the same unit [51].

WSM
The WSM is one of the simplest MCDM methods. This method is based on weights given by the decision-maker. Each criterion is given a non-negative weight, and the alternatives are ranked by evaluating the weighted sum of the criteria [43]. The method involves adding together criteria values for each alternative and applying the individual criteria weights, which should be expressed as nonnegative values. This is best expressed as a decision matrix: The decision matrix represents m alternatives (A1 to Am) in rows and n criteria (C1 to Cn) in columns, where aij (i = 1,…,m; j = 1,…,n) indicates the relative importance of alternative Ai with respect to criterion Cj.
Following this, each criterion value is multiplied by its corresponding weight. The aggregated value for each alternative is calculated using the following formula: The alternative with the highest value can be established as the best solution as follows: where, the matrix A has data entries aij corresponding to the relative importance value of the ith alternative in terms of the jth criterion. Ai WSM and Aoptimal WSM are the WSM aggregated values for any and optimal alternatives, respectively, wj is the weight (relative importance) of the jth criterion, and m and n are the numbers of alternatives and criteria, respectively.

WPM
The WPM is in principle similar to the WSM, except that it performs ranking of alternatives on the basis of a multiplicative measure instead of addition [1]. Therefore, when applying WPM in decisionmaking, the impact of zero values for the relative importance of criteria must be treated cautiously, since they might distort the multiplicative ranking measure. Similarly to the WSM, the WPM also requires cost criteria to be transformed into benefit ones prior to normalization. The aggregated value for each alternative is calculated by using the following formula: Then the alternative with the highest value can be selected as the best solution: where, Ai WSM and Aoptimal WSM are the WPM aggregated values for any and optimal alternative, respectively. Since WPM eliminates any units of measure, this method is considered as a dimensionless analysis and can be used for single and multi-dimensional problems [1].

rAHP
The rAHP employs, in a similar manner to the WSM and WPM, a matrix formulated from relative importance values of alternatives in terms of each criterion and the use of pair-wise comparisons, both to estimate criteria weights and to compare the alternatives with regard to the decision criteria [52].
However, instead of accumulated the sum of relative importance values of alternatives (a1j, …, aij) being equal to 1, proposed in the original AHP [44], in the rAHP the relative importance value (aij) of the jth criterion is divided by maximum value of aij (maxaij) of the jth criterion [53] resulting in calculation of aggregated value using following formula: The alternative with the highest value can be established as the best solution as follows: = max ∑ max =1 ; = 1,2,3, … , ; The rAHP can use both benefit and cost criteria [54].

TOPSIS
The TOPSIS is a model based on an aggregating function representing distances from the ideal solution [26]. TOPSIS approaches a MCDM problem by considering that the optimal alternative should have the shortest distance from the ideal solution and the greatest distance from the negative-ideal solution.
The distances are computed for normalized and weighted data and are measured using the Euclidean metrics. TOPSIS can be applied to both benefit and cost criteria [55]. TOPSIS commences with the normalization of relative importance values of alternatives, which are calculated as following: In the third step, the positive ideal (best) (V + ) and negative-ideal (worst) [56] solutions are determined as follows: = max ; = 1,2,3, … , ; It should be mentioned that TOPSIS does not consider the relative importance of the distances from the positive and negative ideal solution points [30]. The TOPSIS model uses squared terms in the evaluation of criteria and this should be emphasised. The consequence is that the most beneficial and most costly relative importance values in input data can have more of an impact on the final output, whereas average data points will have a lesser impact to the results if compared to the methods that do not utilise squared terms. Methods that utilise squared terms may not be particularly suitable where criteria values for different alternatives are similar, thus requiring further identification.

COPRAS
The COPRAS is, in principle, similar to the WSM. However, COPRAS allows for both benefit and cost criteria to be considered within the matrix [45].
After the normalization step as described in section 2.
The aggregated value for each alternative is calculated using the following formula: The alternative with the highest value is established as the best solution:

Sensitivity analysis
In solving the MADM problem using MCDM models, the relative importance of the criteria (weights) is determined by decision-makers. They can employ experts to rank the criteria or to establish the importance of the criteria using other means. As a consequence, the relative importance of the criteria can be to some degree subjective, carry some level of uncertainty and, therefore, may shape the analysis of the results.
In order to investigate the sensitivity of the output to the uncertainty in the criteria weight, we used an example with parameters (data) acquired for the assessment of the affordability of sustainable housing [47]. Eighteen criteria and eleven alternatives (residential housing areas) were used to build a First, selected MCDM methods were investigated if they outputs are different with selected criteria and alternative inputs by comparing aggregated values and ranking of alternatives. Therefore, the aggregated values of alternatives (Ai) and their resulting ranking (rank) were determined ( Table 2).
All aggregated values and, with exception for A1 and A2, a majority of alternatives were ranked differently by five selected MCDM methods. Table 2. Aggregated values (Ai) and ranking results (rank) using five different MCDM methods.
Second, the pairwise correlation analysis between five MCDM methods was executed. Results showed that four methods, WSM, rAHP, TOPSIS and COPRAS, performed very similarly with a Pearson correlation coefficient ranging from 0.830 to 0.995 (Table 3). These findings extend and consolidate other studies, where WSM, AHP and TOPSIS [49], AHP and TOPSIS [57], and WSM and COPRAS [58] are compared. Previous research has suggested that it is important to use alternative MCDM methods in order to achieve reliable and viable ranking results. Although none of the MCDM approaches can significantly outclass other methods, the correlation analysis suggested that the COPRAS was most consistent with other models used for the assessment of sustainable housing affordability (Table 3). Whereas previous research proposed that the WSM model, which performs accurately and delivers satisfactory results for a majority of single-criteria based problems [59], can be used as a standard for evaluating MCDM methods [1], it was found to be second best according to the analysis. Table 3. Correlation between alternative rankings computed using different MCDM methods.
Note: similarity matrix is coloured as a heat-map that shows the level of correlation between ranking results. The most dissimilar rankings are coloured in light pink. MCDM method pairs with absolutely equal rankings have a Pearson correlation value equal to "1" and are in light blue.
Next, as described in Materials and methods and by following approach described in [38], we determined the most critical criterion for each MCDM model. As shown in Figure 1A, the criterion C14 had the highest sensitivity coefficient for the best alternative in all MCDM models, whereas criteria C3, C4, C6 and C18 were identified as 2 nd or 3 rd amongst critical criteria for the best alternative. C14 was the most critical criterion for any alternative in the case of rAHP and positioned as 2 nd and 3 rd in WPM, TOPSIS and COPRAS ( Figure 1B), whereas C18 and C3 were the most critical criteria for any alternative using WPM, TOPSIS and COPRAS (C18) and WSM (C3). As for the best alternative, C3, C6 and C18 were amongst the criteria positioned as 2 nd and 3 rd . In addition criterion C10 was identified as 2 nd (WPM) and 3 rd (rAHP) for any alternative.
Taken together, criteria C3, C4, C6, C10, C14, and C18 were identified as the most critical criteria for both the best and any alternative. These results revealed that the sensitivity of different criteria varies significantly across five selected MCDM methods. Moreover, the when different alternatives are taken into consideration, different criteria can be most critical, which highly depends on the MCDM method applied. Therefore, to reduce computational costs and especially when multiple MCDM approaches are used, other measures outside those that are provided by sensitivity analysis should be considered.
In order to quantify the level of crosstalk between criteria and ranking, the sensitivity analysis was performed. It was aimed at establishing how the ranking of alternatives (aggregated score Ai) change due to alteration of criteria weights (Cj). Table 4 shows the distribution of relative sensitivity coefficients. The specific value of the sensitivity coefficient denotes that a 5% (arbitrary small change) or 50% (arbitrary large change) increase or decrease of the criterion weight leads to single, double or multiple changes in the ranking of alternatives.  The robustness of the ranking output, expressed as a percentage of the tolerable change of the criterion weight, is summarised in Table 5. Results revealed that the WPM model had the lowest relative coefficients with the fewest changes in the ranking of alternatives and highest tolerance to the 5% (or 50%) alteration in criteria weights comparing to other four MCDM models. The simulated 5% change of criteria weights did not have any influence on the ranking of alternatives by using WPM and COPRAS. The MCDM output was absolutely robust to the changes of criterion C11 weight. This suggested that the exclusion of this criterion from the MCDM matrix and subsequent analysis had no effect on the ranking results and therefore can be eliminated or substituted with other criterion if required. Table 4. Relative sensitivity coefficients calculated as a number of changes in the alternative ranking due to change of criteria weights. Table 5. Tolerable change of criteria weights established through the sensitivity analysis.
The results of the sensitivity analysis including each individual criterion were summarised in Figure 2. Notably, changes of the most critical criteria for the best and any alternatives, i.e. C14, C18 and C3 (Figure 1), comprised a most important effect on the ranking of alternatives as shown in Figure   2. However, the relative sensitivity coefficients and tolerable change of criteria weights varied significantly across five MCDM methods. These results supported the notion that other than sensitivity analysis approach should be considered aiming to reduce computational costs.

Dispersion
The   Firstly, the relative importance values of alternatives for non-essential criterion C11 was consecutively substituted with randomly generated values, which were within the different dynamic ranges, computed as the ratio between the largest and smallest values, from 1 to 10 6 , including 1 to 1.25, 1 to 1.5, 1 to 2, …, and 1 to 10 6 . Using randomly generated relative importance values of alternatives, the simulations of MCDM were performed as described in Material and methods. The resulting aggregated values Ai and ranking results were subjected to a correlation analysis. Euclidian distances accounting for dissimilarity were plotted against the CV. All models demonstrated a very high level of positive correlation between the dispersion of relative importance values of alternatives for individual criterion, measured as CV, and the dissimilarity of aggregated values, calculated as Euclidian distances ( Figure 4). For WSM and COPRAS, the correlation was absolute, with coefficient of determination R 2 = 1. Similarly, a significant correlation was also observed when ranking results were used instead of aggregated values ( Figure 5).  Altogether, the results demonstrated that the dispersion of relative importance values of alternatives strongly correlates with dissimilarity of aggregated values, which is one of the key parameters for ranking uncertainty. The dispersion can be used as an alternative measure for identification of most and least sensitive criteria, potential criteria ranking and is applicable for at least four MCDM methods including WSM, rAHP, TOPSIS and COPRAS.

Conclusions
The ranking output in the MCDM heavily depends on a nature of criteria that are used in the analysis and most notably on a distribution of the weighting amongst criteria. In addition, the factor that the criteria weights are usually established on the basis of expert perception should be taken into consideration, which can, to some extent, be subjective and may vary accordingly. Therefore, the effect of a possible deviation of the criteria relative importance values of alternatives should be evaluated.
Commonly, for the purpose of MCDM, the information about alternatives is derived in qualitative and quantitative forms. Then, qualitative data have to be transformed into measurable numerical values [60]. Both types of data can have a very different level of statistical dispersion. We used real data, collected for the assessment of the affordability of sustainable housing, to investigate how the data dispersion and distribution can influence alternative ranking using the five MCDM models including WSM, WPM, rAHP, TOPSIS and COPRAS. The sensitivity analyses of these different MCDM methodologies confirmed an earlier view that alternative MCDM methods should be used for a thorough and, most significantly, a critical assessment of decision-making. The criteria sensitivity differed amongst MCDM methods and amongst criteria themselves.
Here, we proposed that those criteria, of which the relative importance values of alternatives are distributed across a broad range of values, should be treated with particular care. For this purpose, the ranking susceptibility to the change of criteria weight can be assessed analytically using the sensitivity analysis. Then, the prospective ranking results must be taken into consideration. To verify the ranking results, alternative MCDM methods, in particular those that transform relative importance values of alternatives using constrained functions, as for example preference functions in PROMETHEE [61], can also be considered.
Finally, in this study, the sensitivity of MCDM methods to the change of weight of the decision criterion was assessed and the link between the ranking uncertainty and the dispersion of relative importance values of alternatives was established. Importantly, it was demonstrated that the criteria sensitivity and resulting uncertainty in the ranking output strongly correlates with the dispersion of relative importance values of alternatives for at least four MCDM methods including WSM, rAHP, TOPSIS and COPRAS. Therefore, the dispersion of relative importance values of alternatives can be used as a measure for identification of critical criteria.