A Novel Non-Iterative Parameter Estimation Method for Interval Type-2 Fuzzy Neural Networks Based on a Dynamic Cost Function

Non-iterative methods for parameter estimation for interval type-2 neuro-fuzzy structure are fast to implement, when compared to online methods, and need no –or a few– parameters to be tuned. In this paper, a novel dynamic cost function, which defines a relationship between the current and past errors, is defined. The minimization of the aforementioned cost function results in a decreasing sequence of error which makes the proposed method numerically more stable when compared to least squares-based methods. It is a well-known phenomenon that a matrix inversion may cause problems if the matrix to be inverted is ill-defined i.e. its condition number is far bigger than one. The use of a dynamic relationship between the current and past error adds more degrees of freedom which makes it possible to improve the condition number of the matrix. Comprehensive simulation studies are presented for the prediction of financial data sets. The simulation results shows the superior numerical stability of the proposed method as the mean value of the condition number is smaller. This finding results in more accurate matrix inversion to be done in the two-step matrix inversion.


I. INTRODUCTION
Interval type-2 fuzzy logic systems (IT2FLSs) become wellknown systems for their ability to deal with uncertainty, noisy and incomplete data.IT2FLSs benefit from type-2 membership functions (MFs) with fuzzy membership grades [1].This structure demonstrated superior performance over their type-1 counterparts specially when high levels of noise and uncertainties exist in the system [2]- [5].
Among various learning algorithms, the Gradient descentbased algorithms have been dominantly utilized for training the neural networks (NNs) [6].The back-propagation (BP) algorithms and its variants suffer from various learning issues including stop criteria, number of learning epoch and the possibility of entrapment in a local minimum during training of NNs [7] and [8].Such issues have emerged novel noniterative learning schemes [9].Various least squares (LS) based methods are proposed to solve the limitations of BP algorithms and has shown its efficiency in training the NNs [10].This method is a non-iterative method and makes it possible to find the results in two iterations.The fact that there is no need to tune any parameters, makes this algorithm a very fast algorithm which does not necessitate any iterations to select its parameters.
The LS based method in [8] can be considered as a feedforward NN (FFNN) where learning of the system takes place in two stages.In the first stage, the weights connecting input layer and the hiddden layer are selected randomly [11], [12].In the second stage, the connections with output neurons (weights of output layer) are analytically determined.This makes LS training of FFNN as a linear learning problem where only the connections with output neurons need to be adjusted.Solving a linear system with the random parameters reduces computation time than that of the iterative BP algorithms without sacrificing the prediction accuracy.The efficacy of this method over the support vector machines and its variants has been empirically shown [13].Other variants of non-iterative learning algorithms are evolutionary ELM [7], optimally pruning ELM (OP-ELM) [14], and bidirectional ELM (B-ELM) [15] that tries decreases the number of hidden nodes without affecting learning effectiveness of the algorithm.In these algorithms the optimal nodes are selected in a way that the ones with lower residual error among the several randomly generated hidden nodes.
In this paper, an innovative non-iterative training algorithm to tune the parameters of IT2FLSs is introduced.Although previous non-iterative optimization methods are based on the 978-1-5386-1728-1/19/$31.00 ©2019 IEEE minimization of sum of squared error, in the proposed method, the cost function is sum of squared of a discrete time dynamic relationship in the form of s(t) = e(t) + λe(t − 1), |λ| < 1.Such cost function has been previously used in [16], but not solved using non-iterative approaches.Since absolute value of the parameter λ is less than one, minimization of the sum of squares results in decreasing error with respect to time.Although, it can be proved that existing non-iterative training methods can optimally tune the consequent part parameters of IT2FLSs, the proposed method shows superior performance with the test data.This is mainly because of the numerical stability caused by the minimization of dynamic relationship.One of the main challenges with existing non-iterative optimization is that they include matrix inversion which may cause numerical instability.However, the minimization of the proposed cost function which is based on the current and past error improves the condition number of the matrices to be inverted.The condition number in this case depends on the selection of the parameter λ which adds more degrees of freedom to the optimization method.To test the performance of the proposed training method in dealing with real time datasets, it is applied on the prediction of SGD/USD and EURO/USD exchange rates.The simulation results show superior performance for the test data when compared to the existing non-iterative algorithms for training of the IT2FLSs.
This paper is organized as follows.The structure of IT2FLSs and existing non-iterative optimization methods are introduced in Section II.In Section III, the proposed non-iterative optimization method based on proposed dynamic cost function is discussed.Simulation results are presented in Section IV.The concluding marks are presented in Section V.

II. STRUCTURE OF THE INTERVAL TYPE 2 FUZZY LOGIC SYSTEM
Type-2 fuzzy MFs benefit from a secondary MFs which make the membership grade fuzzy.The fuzzy system, which is constructed using this type of MFs, are called generalized type-2 fuzzy logic systems.However, if the secondary membership grades are considered as either equal to zero or one, the resulting fuzzy system is called IT2FLS.This simplification makes it possible for IT2FLSs to be executed much faster than the general IT2FLSs.The main difference between T2FLSs and T1FLSs is that in IT2FLS output processing block which consists of a type-reducer followed by a defuzzifier block replaces defuzziffier block of T1FLSs (see Fig. 1).

Fig. 1: Structure of the IT2FLS.
A generalized IF-THEN rule (R n ) with N number of rules for an IT2FLS is as follows [17]: which maps the fuzzy set to a varying singleton y n .Xn i , i = 1 • • • , d, are the i th interval type-2 fuzzy subset generated from the input variable x i in the n th rule domain with N being the total number of fuzzy rules.
T represents the parameters of n th fuzzy rule.
Since Gaussian MF are sufficiently smooth functions which are one of the most frequently used fuzzy MFs.However, in the case an interval value is considered for the center of the Gaussian MF, the resulting upper and lower Gaussian MFs are not differentiable.Such Gaussian type-2 MF with a fixed σ n i parameter and an interval center value that takes on values in [m n1 i , m n2 i ], [18] and [17] is as follows: 1) has upper fuzzy MF, μ Xn i (x i )), and lower fuzzy MF ,μ Xn i (x i ), as follows: The firing strength of a rule corresponding to an input set is an interval value, where and (5) The output of the IT2FLS is obtained as follows: f n where: and x 0 = 1.
The indices l and r in the IT2 fuzzy sets [y l , y r ] represent left and right limits, respectively.There exists different approaches to calculate the outputs of IT2FLSs [19], among which Enhanced Karnik-Mendel (EKM) method [20] is used because of its efficacy and most frequently usage.Assume y = [y 1 , • • • , y N ] T as the original rule-ordered consequent values, and ỹ = [ỹ 1 , • • • , ỹN ] T as the reordered consequent values, where ỹ1 ≤ ỹ2 ≤ • • • ≤ ỹN .The parameters f n and fn are also reordered corresponding to ỹs.The resulting vectors are represented by f n and fn , respectively.Finally, outputs y l and y r in (6) are obtained as follows. and where L and R are computed using several iterations of EKM.Detailed step by step procedure of EKM model is presented in in [21] and [22].The final crisp value of IT2FLS is obtained using the mean of y l and y r as follows.
A. Basic Non-iterative Optimization Algorithm Type-2 fuzzy extreme learning algorithm (T2FELA) is a successful non-iterated algorithm designed for SLFNs using interval type-2 fuzzy premise parts [23].This algorithm is summarized as follows.
For a training tuple in the form of (x i , y i ) , an arbitrary SLFN with Ñ number of hidden nodes is represented as follows.
where g(x i ; w j , b j ) is the activation function acting on the hidden layer and w j and b j are its parameters.The prediction error is considered as follows:

III. MODIFIED NON-ITERATIVE TRAINING BASED ON DYNAMIC
COST FUNCTION A discrete time dynamic relationship between current and past error is defined as follows: where λ is a design parameter which is chosen as |λ| < 1 to guarantee the stability of the dynamic surface.The equation ( 11) can be represented in the following matrix form: where and where H is a matrix composed of the outputs of hidden layer; generated randomly with parameters w j and b j .β is the output weight matrix and y T represent the transpose of vector y.The parameter β, the optimal values of β is obtained as the solution to (12) as follows.
where H † is the Moore-Penrose generalized inverse of matrix H [24] and [25].
The proposed parameter update rule based on ( 16) is presented in Fig. 2.
Remark 1.Since the most important novelty of the present paper lies in its novel cost function, this cost function needs to be justified in more details.In the case this cost function made equal to zero, a relationship between the past and current error is satisfied.In this relationship, the current error is equal to the past error when it is multiplied by a value λ which is less than one.This means that the error has a decreasing value which may result in more stability for the training phase.
Remark 2. It is important to note that although the parameter λ adds more degrees of freedom, its selection may be challenging and may require trial and error.Based on our experiments, an appropriate selection of this parameter would be λ = −0.5.However, the tunning of this parameter may result in superior performance.

A. Performance indexes
The proposed algorithm is tested according to the most common performance indexes namely mean absolute percentage error (MAPE), adopted mean absolute percentage error (AMAPE) [26], symmetric mean absolute percentage error (SMAPE) and root mean squared error (RMSE).
where N represent test sample number.The parameters A t and F t represent actual and foretasted values at time instance t.
Symmetric mean absolute percentage error (SMAPE) is an error indicator based on the relationship between absolute error and the average value of real and predicted values.This performance index is represented as follows: where N , A t and F t are defined in the same way as in (19).

B. Prediction of the SGD/USD exchange rate
Cross border activities and foreign exchange market require the prediction of exchange rate between major currencies across the globe.Accurate forecast of the exchange rate results in considerable decrease in risks and makes the overall financial activity profitable.Singapore Dollar as one of the major currencies is chosen here to test the prediction accuracy of the proposed approach when it comes to financial data.The data is chosen from 3−Jan−1994 to 29−Apr−2016.Totally, the number of entries in investigated data set is equal to 5735 samples.The missing data appears in the dataset as a result of holidays, in which the latest exchange rate is used to fill the missing data entry.The data samples from 3 − Jan − 1994 till Six individual fuzzy rules are used to predict the financial data with their center and sigma variables being completely random variables.The data set is normalized between zero and one and the centers are uniformly distributed in this interval.The sigma values are chosen uniformly from the interval of [0.2, 0.5].In order to have statistically meaningful results, the experiments are repeated for 30 times and the mean values are reported in the Table I.As can be seen from this table, performance of the proposed method is better than T2FELA considerably.Moreover, the prediction performance for test data is illustrated in Fig. 3a.This figure shows that it is Figure 3b depicts that the graph of identified data with respect to real data is very close to y = x line.
As mentioned earlier, the reason behind better performance of the proposed algorithm over T2FELA is that the addition of a parameter to the cost function makes it more stable.In this case, the mean value of condition number for the proposed method is obtained as 1.8619 × 10 6 while it is being equal to 1.1345 × 10 7 for T2FELA.Similar to the previous case, this fact clearly shows the reason why the results obtained by the proposed method outperform the results obtained by T2FELA.

C. Prediction of the Euro/USD exchange rate
In this section, the proposed non-iterative algorithm is used to predict the daily exchange rate of Euro versus USD for 4558 days in the interval of Four rules are considered for the fuzzy system with completely random center and sigma variables.For the prediction purpose, the exchange rates are normalized to the interval of [0, 1] and the centers are uniformly distributed in this interval.However, the indices are all evaluated based on the true data.The sigma values are chosen uniformly from the interval of [0.2, 0.5].Similar to the previous case, Table II illustrates that the proposed method outperforms T2FELA considerably.In order to have statistically meaningful results, the experiments are repeated for 30 times and the mean values are reported  .Moreover, the prediction performance for test data is presented in Fig. 4a.Moreover, Figure 4c shows how identification error is distributed around zero. Figure 4d depicts that the graph of identified data with respect to real data is very close to y = x line.
Similarly, the authors believe that the reason behind better performance of the proposed algorithm over T2FELA is the proposed method is numerically more stable.The fact that the mean value of condition number for the proposed method is obtained as 3.5774×10 8 while it is being equal to 1.9220×10 9 for T2FELA supports this idea.

V. CONCLUSION
The use of dynamic cost function in this paper results in overcoming the numerical problems caused by matrix inversion needed to estimate the parameters of the consequent part of IT2FLSs when using the T2FELA.Using the proposed dynamic cost function, a novel formula is extracted for the consequent part parameters of IT2FLS.Since the proposed cost function benefits from a parameter, it has more degrees of freedom which makes it possible to obtain a more numerically stable parameter update rule which results in better approximation of the true values.The proposed method is applied to predict financial datasets.Having a good prediction is required for decreasing the risk of decisions.This is the main reason why a foreign exchange dataset is selected in this study to test the prediction capability of the proposed approach.As can be inferred from the simulation results, the proposed method makes the condition number for the matrix inversion in the estimation of the parameters of the consequent part of IT2FNN closer to one, which is highly desirable.Furthermore, the proposed algorithm shown to be capable of outperforming T2FELA if appropriate parameter values are selected for this algorithm.

Fig. 2 :
Fig. 2: Sequential diagram of the proposed training algorithm

14 −
Jan − 2005 are used for training and the data samples from 15−Jan−2005 till 29−Apr −2016 are used for testing of the proposed method.
30 − Oct − 1998 to 15 − Aug − 2016.For missing data resulted from holidays or unavailable data samples, the latest exchange rate is used.The data samples from 30 − Oct − 1998 till 8 − Aug − 2007 is used for training and the data samples from 9−Aug−2007 till 15−Aug−2016 are used for testing the proposed method.

Fig. 3 :
Fig. 3: Estimated values versus real values for the prediction of SGD/USD exchange rate

TABLE I :
Results of comparison between the proposed SMC based non-iterative optimization method and T2FELA.
possible to use the data points between 3 − Jan − 1994 and 14−Jan−2005 and the trained system is capable of predicting the exchange rate of 15 − Jan − 2005 till 29 − Apr − 2016.

TABLE II :
Results of comparison between the proposed SMC based non-iterative optimization method and T2FELA.