Multi-Layer Hybrid Fuzzy Classification Based on SVM and Improved PSO for Speech Emotion Recognition

Abstract

Speech Emotion Recognition (SER) plays a significant role in the field of Human–Computer Interaction (HCI) with a wide range of applications. However, there are still some issues in practical application. One of the issues is the difference between emotional expression amongst various individuals, and another is that some indistinguishable emotions may reduce the stability of the SER system. In this paper, we propose a multi-layer hybrid fuzzy support vector machine (MLHF-SVM) model, which includes three layers: feature extraction layer, pre-classification layer, and classification layer. The MLHF-SVM model solves the above-mentioned issues by fuzzy c-means (FCM) based on identification information of human and multi-layer SVM classifiers, respectively. In addition, to overcome the weakness that FCM tends to fall into local minima, an improved natural exponential inertia weight particle swarm optimization (IEPSO) algorithm is proposed and integrated with fuzzy c-means for optimization. Moreover, in the feature extraction layer, non-personalized features and personalized features are combined to improve accuracy. In order to verify the effectiveness of the proposed model, all emotions in three popular datasets are used for simulation. The results show that this model can effectively improve the success rate of classification and the maximum value of a single emotion recognition rate is 97.67% on the EmoDB dataset.

1. Introduction

With the gradual enrichment of material life, people’s attention has gradually shifted from the physical world to the spiritual world [1]. Research on human–computer interaction (HCI) for emotion recognition is increasingly becoming a hot topic. In the field of emotion recognition, it is necessary to develop machines that can understand human emotions better. In the field of HCI, the computer can recognize emotions through gesture, audio signals, body poses, facial expression, physiological signals, and neuroimaging methods, etc. [2,3]. Apart from expressing emotions and communicating, speech is the most natural and fastest method for speakers to convey emotions by intonation, volume, and speed compared to others. Speech emotion recognition (SER) is a method of emotion identification by extracting the emotional state of speakers from their speech signals [4,5,6]. In recent decades, SER, as the main form of emotional display, has focused on achieving a more natural interaction between people and machines and has become deeply involved in a wide bank of real-life applications, such as public safety [7], diagnosis of psychiatric diseases [8], adjustment of driving behavior from the state of drivers [9], web games, emergency call centers [10], and so on.

The wide application of SER depends on the rapid development of its technology. With the extensive expansion of deep learning research [11,12,13,14,15,16], SER technology has gradually been applied beyond the use of basic methods such as hidden Markov model (HMM), the Gaussian mixture model (GMM), support vector machine (SVM), and k-nearest neighbor (KNN) [17,18,19,20]. Deep learning and its various variants also optimize the system from the aspects of feature extraction and classifier performance [21,22,23,24,25], which are the forms of SER model become increasingly diverse. In addition to analyzing only speech signals, the emotion recognition system based on deep learning also uses the multimodal recognition method to analyze emotions combined with attributes such as text, facial expression, gesture, and electrocardiography [26,27,28]. Meanwhile, researchers have also studied the SER system on contextual dependencies of speech signals by optimizing recurrent neural networks (RNN) and long short-term memory (LSTM) networks [29,30]. However, speech signal is still one of the most relevant attributes of emotion recognition, and SVM is still widely used in classification models because it provides quite good performance for SER system with less complexity. Therefore, the importance of the most common method cannot be ignored.

The aforementioned studies improve recognition accuracy by extracting more appropriate speech emotion features and training better models, but there are two easily overlooked reasons affecting the recognition accuracy of the SER system. One is that the low recognition accuracy of one or several emotions will result in a decline of overall accuracy when the number of emotions is large. It can be observed from existing studies that the recognition rate of neutral emotion is low, and it is difficult to distinguish fear and disgust [31,32]. The other is the impact of identification information on emotional understanding. People of different gender, nationality, and age express the same feelings in different ways.

In order to solve above problems, this paper proposes an SER model called multi-layer hybrid fuzzy support vector machine (MLHF-SVM) to improve the overall recognition rate. This model uses SVM-based integrated classifier to train specific emotions that are difficult to recognize. The pre-classification layer assisted on enhanced fuzzy c-means (FCM) is added to classify speech features according to the identification information, and the classified features will be sent to the corresponding classifier. Moreover, we propose an improved particle swarm optimization (PSO)-based FCM clustering method called improved natural exponential inertia weight PSO (IEPSO) to enhance the ability of pre-classification feature parameters and the overall accuracy of the system. Meanwhile, in the feature extraction layer, feature parameters incorporate non-personalized features to ameliorate the universality and adaptability of speech signals. We train and test the model on three commonly used speech emotion databases, and the ability of the model to distinguish emotions has been greatly promoted.

The main contributions of this paper are as follows:

• We propose an MLHF-SVM model, which adopts FCM based pre-classification layer and multi-layer SVM ensemble classifier to increase the recognition accuracy of some specific emotions and reduce the impact of identification information. This method effectively increases the overall emotion recognition accuracy of the SER system.
• We present an improved natural exponential inertia weight PSO (IEPSO) to alleviate the situation that FCM is easy to fall into local minima because of the initial value. This hybrid method (FCM-IEPSO) is used as the pre-classification layer to enhance the effect of pre-classification feature parameters.
• We select three databases for comprehensive simulations, compare the recognition ability of KNN and SVM as the base classifier, and contrast the performance of PSO aided on other inertial weights. The results show that this model efficiently promotes the overall performance of the system.

The remainder of this paper is organized as follows. Section 2 discusses the related studies and tries to reveal the differences and similarities between them. Section 3 briefly introduces the basic structure of the SER system, including feature extraction, feature pre-classification, and feature classification. Section 4 elaborates on the proposed model, other innovative parts, and their application in this system. Section 5 provides the simulation results and explanation. Section 6 makes the final conclusion.

2. Related Work

The amount of research on speech emotion recognition has grown with the extensive applications of SER and accessibility of speech datasets. In order to highlight the research focus and fully compare and analyze the existing literature, we reviewed the following related literature. We particularly focus on the research on emotion recognition based on speech signal rather than multimodality.

In [33], mel-frequency cepstral coefficients (MFCC) and liner predictive cepstral coefficients (LPCC) features were extracted from input speech signals as speech feature parameters, and HMM and SVM were used as classifiers to distinguish emotions according to characteristic parameters. In addition to extracting spectral features such as MFCC and LPCC, reference [31] also extracted other acoustic features such as zero crossing rate (ZCR), energy, and fundamental frequency. The fused features contain more information on speech emotion, which makes the trained classifier more accurate. They utilized SVM and Linear Discriminant Analysis (LDA) assisted classifiers to classify speech signal emotions and tested the model in Berlin Database of Emotional Speech (EmoDB) and RML Emotion Database (RED). Considering that the use of all the features extracted will increase the total workload of SER system and that the influence of existing features on the final recognition rate is not clear, reference [34] refined the existing features by a feature reduction method after feature extraction, and the model increased the success rate of emotion recognition. Reference [35] adopted a bagged ensemble classifier comprising SVM in order to classify reduced features. The results showed that ensemble learning performs more superior than single estimators. Gradually, the design and implementation of deep learning have been more and more feasible. The feature parameter extraction and classification of signals in speech emotion recognition system have relied on deep learning.

In [21], a convolutional neural network (CNN) filter distributed in all spectrum ranges extracted more concentrated frequency domain features to accurately identify emotions, and the average accuracy was 66.1%. In order to improve classification accuracy, some papers optimize the architecture of CNN. Reference [25] modified the pooling strategy of CNN as a filter to learn depth frequency characteristics. The new lightweight effective SER model was trained on the extracted speech frequency features, and the recognition result was 77.01%. Issa et al. [36] extracted MFCC, chromagram, Tonnetz representation, and spectral contrast features from the speech signals as the input of CNN and used an incremental method to optimize SER model. One discovery is that the information of speech emotions is embedded in the long temporal context, and contextual information can assist in judging emotions [37]. Motivated by GoogleNet, Li et al. [24] proposed an attention pooling based model for SER, which utilized two groups of filters to obtain the features with context information in time domain and frequency domain and then fed the features to CNN for classification. In [23], the combination model of CNN and LSTM was used to consider the contextual information in the data. They stacked two layers of LSTM on the top of CNN and conducted end-to-end training. Similarly, Zhao et al. [30] designed a network composed of CNN and LSTM to learn long-term dependencies from the extracted features. A 1D CNN LSTM and a 2D CNN LSTM were constructed to learn local and global features from speech and a log-mel spectrogram, respectively. However, the problem that has not been considered in the above literature is that the recognition accuracy of single emotion or some confusing emotions is lower than that of other emotions, which affects the overall recognition rate.

The recognition accuracy of neural emotion is relatively low, and it is difficult to distinguish fear and disgust when the SER system recognizes speech emotions [32]. Badshah et al. [38] presented a deep CNN composed of three convolutional layers and three fully connected layers to train voice signals. The simulation results showed that the recognition accuracy of fear was still low. The bottleneck features extracted by DNN were trained by SVM to reduce the confusion of emotion recognition in [39]. This method extracted different bottleneck features for training according to different emotions and effectively reduced the confusion between emotions and increased the accuracy of fear. Reference [40] created a hybrid feature vector composed of acoustic features and depth features extracted from depth network architecture. The hybrid feature vector described speech features more accurately and could improve the success rate of single emotion classification. In order to increase the speech emotion recognition accuracy of fear by extracting more accurate features, K. Zvarevashe and O. Olugbara [41] extracted prosodic features, spectral features, and others to form a hybrid acoustic feature vector. They verified the effectiveness of the features in two benchmark datasets and trained the features on the integrated classifier based on random forest. The review of numerous literature has revealed that there is little literature on training SER model for increasing the recognition accuracy of a single emotion, and most of them optimized the model from the perspective of diversification of characteristic parameters and development of a classification model.

Different from the perspective of the above paper, we add a pre-classification layer to classify the feature parameters according to the identification information and assist pre-classification by using the proposed FCM-IEPSO. Then, the integrated classification is used to train and classify emotions in parallel so as to improve the recognition rate of each emotion.

3. Materials and Methods

3.1. Emotional Speech Database

Three databases were adopted to improve the universality and reliability of the study, and all speech samples in the databases were used.

The Berlin Database of Emotional Speech (EmoDB) [42] contains seven emotions, which are anger, boredom, disgust, fear, happiness, sadness, and neutralness. The speech samples were recorded in a studio by ten professional German actors (5 actors and 5 actresses) at a sampling frequency of 16 kHz. The Surrey Audio-Visual Expressed Emotion (SAVEE) Database [43] records the speech samples of four native English-speaking male graduate students and researchers from Surrey University, aged from 27 to 31. The database contains seven emotions: anger, disgust, fear, happiness, neutral, sadness, and surprise. The eNTERFACE’05 [44] is an audio-visual emotion dataset, which contains 42 subjects from 14 different nationalities. The distributions and types of databases are shown in

Table 1. Distribution and types of datasets used in this paper (Ang—anger; Bor—boredom; Dis—disgust; Fea—fear; Hap—happiness; Neu—neutral; Sad—sadness; Sur—surprise).

Database	Type	Distribution
		Ang	Bor	Dis	Fea	Hap	Neu	Sad	Sur
EmoDB	Acted	127	81	46	69	71	79	62	-
SAVEE	Acted	60	-	60	60	60	120	60	60
eNTERFACE’05	Elicited	126	-	68	196	198	-	182	205

.

3.2. Basic Methods

Generally, SER is composed of three steps: preprocessing, feature extraction, and classification. Considering the identification information, this system adds a pre-classification part before the classification step. This process uses FCM to cluster the database into multiple subclasses and then feeds each of them into the feature classification module, respectively. The flow diagram is shown in Figure 1.

In the figure, it is assumed that FCM divides speech signals into two categories according to gender, male and female, and the classifier divides emotions into seven categories based on EmoDB. The specific structure of the classifier is shown in Section 4.

3.2.1. Feature Extraction

Feature extraction is one of the important aspects in the SER system, and the features extracted should accurately reflect the emotional information of the speech. Two feature groups in this paper have been investigated. One is based on fundamental frequency (F0) value, zero-crossing rate (ZCR), and root-mean-square (RMS) signal frame energy. The other originates from MFCC. Meanwhile, in order to reduce excessive computation caused by feature redundancy and to improve the universality of feature parameters, seven statistical functions of each frame speech signal are calculated in this paper. The statistical functions obtained are presented in

Table 2. Extracted emotional speech features and statistical functions.

Group	Features	Statistical Functions
Prosodic features Spectrum features	F0, ZCR, RMS 1–16 MFCC	maximum value minimum value mean value standard deviation skewness value kurtosis value median value

.

Group 1: F0, ZCR, and RMS energies are the most widely used prosodic features as features that can be perceived by human beings [45,46,47]. Compared with common features such as LPCC and format, taking features in Group 1 as the feature parameters performs better [48]. F0 reflects the rhythm and intonation of speech in which contour and mean values will change with different emotion. ZCR shows the situation that adjacent samples have different algebraic symbols. RMS energy is highly correlated with emotional state. The energy of high-level arousal emotions (anger, happiness, and surprise) is much higher than that of low-level arousal emotions (sadness and disgust). In addition to these prosodic features, this paper also extracts the spectral features based on MFCC.

Group 2: MFCC is a spectrum feature that can simulate human auditory perception mechanism, and it extracts parameters through human ear perception of frequency signals [49]. The features based on MFCC have a higher success rate in the classification than those based on other spectral feature linear prediction coefficients (LPC). MFCC provides an envelope that is the real logarithm of the short-term energy spectrum provided. In this paper, sixteen MFCC coefficients were extracted for classification and recognition.

3.2.2. FCM Based Pre-Classification

Identification information has an important influence on the understanding of emotions. People with different identification information express their emotions in different ways (i.e., gender, age, and region). Moreover, preliminary classification of speech signal based on identification information is the purpose of pre-classification.

In training the pre-classification model, input data B all proceed through the feature extraction of speech signal, where $B=\{b_1,b_2,\dots ,b_N\}$. $b_j$ represents the j-th speech feature vector, which contains all its features. After clustering, N speech signals are clustered into C subsets. Let the set of subsets be $S=\{s_1,s_2,\dots ,s_C\}$, and each subset has a cluster center $s_i$. Each subset contains several speech signals in B for which its characteristics are the most similar. This clustering process means that the voice signals with the same identification information are pre-classified into the same subset. In the process of testing, the trained pre-classification model will detect the clustering center closest to the input feature, and the corresponding subset of the center is the result of input speech signal pre-classification. Suppose $b_2$ is a speech feature vector, the pre-classification result of $b_2$ can be expressed as follows.

$$\begin{array}{c}\hfill b_2\in \underset{s_i\in S}{argmin}\parallel b_2-s_i{\parallel }^2.\end{array}$$

(1)

FCM is the most commonly used clustering method that utilizes the concept of weight to improve the defects of hard clustering [50]. The clustering process of FCM is classified according to the membership degree of each data. The flow chart of FCM pre-classification is shown in Figure 2.

The following steps briefly describe the pre-classification of FCM. Firstly, C cluster centers are randomly initialized according to the preset identification information types, and the membership degree $u_{ij}$ from data point $b_j$ to the cluster center $s_i$ is calculated in order to obtain a fuzzy partition matrix $U=\left[u_{ij}\right]$. The value function of FCM is defined as follows:

$$\begin{array}{c}J(U,S)=\sum _{i=1}^C\sum _{j=1}^N{u}_{ij}^m{d}_{ij}^2=\sum _{i=1}^C\sum _{j=1}^N{u}_{ij}^m{\parallel b_j-s_i\parallel }^2\\ s.t.{\displaystyle \sum _{i=1}^C}{u}_{ij}=1\forall j\in 1,2,\dots ,N;\forall i\in 1,2,\dots ,C,\\ 0<{\displaystyle \sum _{j=1}^N}{u}_{ij}<N\forall j\in 1,2,\dots ,N;\forall i\in 1,2,\dots ,C,\end{array}$$

(2)

where $d_{ij}$ is the Euclidean distance between $b_j$ and $s_i$, and $m\in [1,+\infty )$ is the fuzzification parameter that affects the membership degree and adjusts the clustering fuzziness degree. In the iterative process, the algorithm ends only when $|J^{\left(t\right)}-J^{(t-1)}|<\epsilon$, where $\epsilon$ is the known sensitivity threshold, and t is the number of iterations. When the termination condition of the algorithm does not satisfy, the membership matrix $u_{ij}$ and clustering center $s_i$ are recalculated according to the following formulas, and the iteration continues until the J value is small enough.

$$\begin{array}{c}\hfill u_{ij}=\left\{\begin{array}{cc}{\displaystyle \sum _{k=1}^C}\frac{d_{ij}}{d_{kj}},\hfill & t=1,\hfill \\ \hfill \\ \frac{1}{{\left({\displaystyle \sum _{k=1}^C}\frac{d_{ij}}{d_{kj}}\right)}^{\frac{2}{m-1}}},\hfill & 1<t\le MG,\hfill \end{array}\right.\end{array}$$

(3)

$$s_i=\frac{{\displaystyle \sum _{j=1}^N}{u}_{ij}^m{x}_j}{{\displaystyle \sum _{j=1}^N}{u}_{ij}^m}.$$

(4)

After FCM clustering, each subclass is placed into its subclassifier for the following emotion classification.

The ability of FCM to process high-dimensional data flexibly is very suitable for pre-classification, that is, to classify characteristic parameters according to prior knowledge (identification information). Nevertheless, in the initialization step, FCM randomly selects initial values for iteration, which will affect the final classification results once a bad initial value is selected.

3.2.3. Classifier

As shown in Figure 1, the sub-feature sets after pre-classification are used to train corresponding sub-classifiers, respectively, and each sub-classifier trains all features and emotions at the same time. SVM and KNN are the classifiers used in this paper.

SVMs are supervised classifiers based on the idea of binary classification, which find the optimal hyperplane for the linearly separable dataset. The method of SVM is to find the hyperplane that separates the two kinds of data with the largest distance between the classification edges, and this hyperplane is the support vector plane. It can also be applied to multiple classification issues. If the dataset is linearly inseparable, the kernel function is introduced to map the input data to a higher dimensional space for further processing. The definition of the polynomial kernel function is shown below.

$$K(x_i,x_j)={(a{x}_i^{\top }{x}_j+c)}^d.$$

(5)

In the kernel function $K(x_i,x_j)$, $x_i$ and $x_j$ are any two vectors in the training sample, a and c are hyperparameters, and d represents the degree of the polynomial. In speech emotion recognition, the recognition ability of this technology is relatively strong.

KNN is a basic instance-based supervised learning method, and the distance is calculated by Euclidean distance. Instead of looking at the nearest neighbor category, KNN takes into account the votes ofk nearest neighbors. Specifically, this method selects k with the smallest distance between test data and training sample, that is, the closest k sequences, and endows the classification result category to the class with the highest frequency among the k sequences.

4. Proposed Multi-Layer Hybrid Fuzzy Support Vector Machine for SER

A multi-layer hybrid fuzzy support vector machine is presented for emotions that are difficult to classify, and it is optimized in the feature extraction layer and pre-classification layer to improve its emotion recognition performance. Under the preset conditions in Figure 1, the rough flow of this model is shown in Figure 3. In this section, we will introduce in detail the improved feature extraction layer, pre-classification layer based on proposed FCM-IEPSO, and the classification layer based on multi-layer SVM.

4.1. Improvement of Feature Parameters

In speech emotion recognition systems, the processing of characteristic parameters usually involves reducing unnecessary parameters or training better parameters However, the parameters obtained by using these methods are specific to the speech sample and are only generated according to the manner of individual emotion expression, which is not representative and universal. In this paper, derivative-based non-individualized features are added to increase the common ground and regularity of the same category of feature parameter vectors.

Assume that m-frame data are obtained after a certain speech sample is preprocessed, and the m-frame data will be extracted through the following steps:

(1)

Extract 19 feature parameters (16 MFCC, F0, ZCR, and RMS) for each frame of input data to obtain $19\times m$ parameters. Then, calculate the statistical data of each feature according to Table 2 to obtain $19\times 7$ personalized features;

(2)

Calculate the derivative of the input data, and its size is $m-1$. Operate Step 1 to extract the characteristic parameters and corresponding statistical data and obtain $19\times 7$ non-personalized features;

(3)

A combination of non-personalized and personalized statistical features lead to $19\times 14$ features per sample.

The operation of this process is presented in Figure 4. It is obvious that the extracted feature is high dimensional, and it can describe the speech signal more accurately. The following hybrid method based on FCM and IEPSO has a good effect on processing high-dimensional features.

4.2. Hybrid Clustering Method Based on FCM and IEPSO

In order to solve the defect that FCM is easy to be affected by initial parameters, which reduces the accuracy of pre-classification, this subsection proposes a hybrid method called FCM-IEPSO. An improved natural exponential inertia weight PSO (IEPSO) is presented to ameliorate the above problem. This method is described in detail below.

4.2.1. Improved Natural Exponential Inertia Weight PSO (IEPSO)

PSO is a population-based optimization algorithm that simulates bird foraging behavior [51]. It seeks the optimal solution through cooperation and information sharing among individuals in the group based on iteration. In each iteration of the algorithm, the optimal particle is searched by updating the particle velocity and position, and the fitness of the particle is determined by the fitness function. Updates of particle velocities require two best positions: personal best position $pbest$ and global best position $gbest$, where $pbest$ is the best position visited by the current particle and $gbest$ is the best position visited by all particles. In addition, the inertia weight $\omega$ and the acceleration constant c also affect particle velocity. Based on statistical theory analysis, inertia weight has a greater effect [52].

In general, larger inertial weights have better global search capabilities, while smaller inertial weights are more focused on local exploitation. In order to balance exploration capability and development capability, an improved natural exponential inertia weight adjustment strategy based on fitness difference before and after iteration is introduced into PSO, which is IEPSO, and its function is as follows:

$$\begin{array}{c}\hfill \omega \left(t\right)=\left\{\begin{array}{cc}{\omega }_{\min }+({\omega }_{\max }-{\omega }_{\min })·e^{-z·\frac{t}{MG}},\hfill & t=1,\hfill \\ \hfill \\ {\omega }_{\min }+({\omega }_{\max }-{\omega }_{\min })·e^{-z·\frac{t}{MG}·e^{\theta \left(t\right)}},\hfill & 1<t\le MG,\hfill \end{array}\right.\end{array}$$

(6)

$$\theta \left(t\right)=f\left(t\right)-f(t-1),$$

(7)

where ${\omega }_{\min }$ and ${\omega }_{\max }$ are the minimum and maximum of the inertial weight, and $\omega \left(t\right)$ is expressed in the inertia weight value of the tth iteration. z is a parameter representation for finetuning the inertia weight, taking 20 from experience.

By using this improved inertial weight, the velocity and position of particle p are updated as follows.

$$\begin{array}{cc}{V}_p(t+1)=\hfill & \omega V_p\left(t\right)+c_1{R}_1(pbes{t}_p\left(t\right)-X_p\left(t\right))+c_2{R}_2(gbest\left(t\right)-X_p\left(t\right)),\hfill \end{array}$$

(8)

$$X_p(t+1)=X_p\left(t\right)+V_p(t+1).$$

(9)

In the above formula, $c_1$ and $c_2$ are acceleration factors; $R_1$ and $R_2$ are the random numbers in the range $[0,1]$ and are independent of each other.

4.2.2. FCM-IEPSO Strategy

FCM always has the disadvantage of being seriously affected by the initial value, which will cause the result to fall into the local minimum value. In this paper, a hybrid fuzzy algorithm combining FCM and IEPSO is proposed, which uses the searching ability of particle swarm to find the appropriate initial value for FCM and avoids falling into the local optimum. The results indicate that this method optimizes the fitness of FCM and reduces convergence time. The pseudo code of this process is described in Algorithm 1.

As shown, the algorithm consists of two stages, including particle search and clustering. In the first stage, the objective function of FCM updates the position of the particle as the fitness function: $f_p=J_{\min }$. Then, the appropriate particles are fed to the next step as initial values, and the optimal clustering centers are obtained by iteration. According to the membership degree, the initial dataset is grouped into class C.

Algorithm 1:FCM-IEPSO Algorithm

Input:  The extracted feature parameter set B and the number of clusters C Output:  clustering results {$S_1,\dots ,S_C$} and minimum fitness value $J_{\min }$ 1: Initialize parameters: $MG$, $c_1$, $c_2$, ${\omega }_{\min }$, ${\omega }_{\max }$ and number of particles P for IEPSO; m, $\epsilon$ and maximum number of iterations $T_{\min }$ for FCM; 2: Calculate the particle position range $[X_{\min },X_{\max }]$; 3: Initialize each particle ($p=1,2,3,\dots ,P$) position $X_p$, velocity $V_p$, $pbest$ and $pbest$; 4: while$(t\le MG)$do 5:    Calculate fitness function f, which is FCM objective function J by (2); 6:    Update $\omega$ , $V_p$ and $X_p$ by (6)–(9); 7:    Update particle position boundary; 8:    Update $pbes{t}_p$ using fitness function; 9:    Store $gbest$ after traversing all p; 10:    $t=t+1$; 11: end while 12: Obtain particle $p^{*}$ as the initialization one for the next iteration; 13: Calculate the membership degree matrix u and initial cluster centers $s\left(p^{*}\right)$ by (3) and (4); 14: while$(t\le T_{\max })$do 15:    Calculate the objective function $J\left(t\right)$ by (2); 16:    if $\left[\right(t>1)\cap (J\left(t\right)-J(t-1)<\epsilon \left)\right]$ then 17:      break 18:    else 19:      $t=t+1$; 20:    end if 21:    Calculate u and s, respectively; 22: end while 23: The dataset B is divided into C parts by the final cluster centers and membership matrix; 24: Output clustering results {$S_1,\dots ,S_C$} and $J_{min}$.

4.3. Multi-Layer Hybrid Fuzzy SVM

For the situation where certain emotions are difficult to classify, this paper presents a multi-layer SVM algorithm to increase the recognition accuracy of poorly behaved emotions. The algorithm uses prior information to divide the classification difficulty of each emotion $E_l(l=1,2,\dots ,L)$ and trains classifiers containing different categories. Each classifier $cl{f}_m(m=1,2,\dots ,M)$ distinguishes at least two classes. It should be noted that the class that is the most difficult to separate needs to be trained first. The priority of classifier training is determined by the difficulty of emotion classification, and its priority also determines the order of test data entering the sub-classifiers. In order to reflect the actual structure of multi-layer classifiers, EmoDB database containing seven emotions is taken as an example. Neutralness and fear, as emotions that are difficult to distinguish, are trained with a separate classifier, while other emotions are trained with another classifier. Since anger and happiness are easiest to distinguish, the classifier trained on them is the lowest level. In order to display the emotion classification of all subclassifiers, the specific structure of multi-layer classifiers based on EmoDB is shown in Figure 5.

In this system, SVM is selected as the base classifier of multi-layer classifier model. Combined with above improved methods, the entire process of multi-layer hybrid fuzzy SVM (MLHF-SVM) is shown as follows:

(1)

The training data from the EmoDB database are preprocessed, and then $19\times 14$ characteristic parameters from each sample are extracted by using the combined method of non-personalization and personalization;

(2)

Characteristic parameters are clustered into C subclasses using FCM-IEPSO, and the number of particles is defined as particle number × the dimension of feature parameter;

(3)

Train each subdata using multi-layer SVM, and each sub-classifier trains different data;

(4)

Verify the trained model with test data, and integrate the results of the sub-classifiers. The final classification results are obtained by a 5-fold cross-validation method.

5. Simulations and Results

5.1. Data and Environment Setting

In order to validate the proposed MLHF-SVM model, we have carried out extensive simulations and specific analyses. All the speech samples in these three datasets have been used for simulations. For the purpose of realizing the model independent of speaker and environment, personalized features and non-personalized features of speech were extracted respectively. Sixteen MFCC, F0, ZCR, and RMS features were extracted as feature parameters of each speech, and seven statistical features were calculated with a total quantity of $19\ast 14$. Since the gender labels of the three databases were known, we chose gender as a prior information for pre-classification. According to prior knowledge, people of different genders have different methods of expressing emotions. Thus, in the pre-classification stage, the data generated by the feature extraction layer were clustered into two subsets by FCM-IEPSO according to gender, that is, $C=2$. Each subclass was divided into data for training and testing. According to the benchmark database, neutral and fear emotions are often confusing and have low recognition rates. As a result, in the training stage, the classifier of neutral and fear emotions sets and the other five emotions sets were trained first. Then, neutral and fear emotions were trained separately, and the remaining emotions were trained simultaneously. SVM and KNN were used as base classifiers to observe the influence of base classifiers on the results.

In order to verify the validity of the proposed model, k-fold cross-validation was used for training and testing all data. Considering the amount of data, the cyclic variable k is set to 5, and the final result is the average value of five cycles. All simulations are implemented in MATLAB R2018a and run on a Windows 10 operative system with 64-bit support using an Intel Core i5 CPU at 2.30 GHz and 7.85 GB of RAM.

5.2. Impact of FCM-IEPSO Based Pre-Classification Layer

In order to analyze the effect of the proposed pre-classification layer, in this subsection, we evaluate the impact of the proposed IEPSO on FCM and the impact of the new pre-classification layer on the overall system recognition accuracy. We test the combination of four types of PSO with FCM: the original PSO, the proposed IEPSO, and two inertia weight variants of PSO. The first variation of the inertia weight, linearly decreasing inertia weight PSO (LPSO) [53], adopts the strategy of monotonically decreasing inertia weight with the number of iterations. The second is natural exponential inertia weight PSO (EPSO) [54], which combines the idea of diminishing inertia weight with a natural exponent to improve convergence speed. The inertia weight is the only variable. When setting parameters, all other parameters except inertia weight are set to the same. Parameter settings are shown in

Table 3. Parameter settings of PSO and its variants.

Parameter	Value
Popsize	100
Acceleration factors ($c_1,c_2$)	1.49445
The maximum of inertial weight ($w_{max}$)	0.9
The minimum of inertial weight ($w_{min}$)	0.4
Sensitivity threshold ($\epsilon$)	${10}^{-5}$
Fuzzification parameter (m)	2

.

Table 4. J value of pre-classification algorithm on three databases. (The best results are highlighted in bold).

Algorithm	EmoDB			SAVEE			eNTERFACE’05
	Ave	Best	Worst	Ave	Best	Worst	Ave	Best	Worst
FCM-PSO	4656.6	4160.3	4789.9	3546.9	3278.3	3741.6	6231.2	5956.6	6435.2
FCM-LPSO	4236.8	3979.5	4336.9	3215.4	3035.7	3541.5	5796.6	5564.1	6156.3
FCM-EPSO	4196.7	4319.4	4264.5	3054.6	2978.5	3255.6	5642.6	5314.0	5956.3
FCM-IEPSO	4127.1	3959.1	4319.4	2960.9	2890.6	3221.5	5456.8	5134.2	6054.1

and

Table 5. Running time of pre-classification algorithm on three databases.

Algorithm	EmoDB			SAVEE			eNTERFACE’05
	Ave(s)	Best(s)	Worst(s)	Ave(s)	Best(s)	Worst(s)	Ave(s)	Best(s)	Worst(s)
FCM-PSO	46.68	43.63	47.65	35.64	33.45	37.75	88.61	82.34	90.53
FCM-LPSO	57.17	53.70	58.20	50.03	43.12	53.68	92.68	87.35	94.02
FCM-EPSO	49.88	46.94	52.67	37.98	36.19	42.89	90.32	86.94	92.48
FCM-IEPSO	48.99	47.33	51.12	36.35	33.93	40.68	90.15	85.16	91.44

compare the J value [55] calculated from formula (2) and operation time of different clustering algorithms, respectively, on three databases. We compare the system using FCM-PSO, FCM-LPSO, FCM-EPSO, and the proposed FCM-IEPSO method as pre-classification. The smaller the J value, the better the clustering effect, and the higher the fitness of the algorithm. In order to intuitively observe the results and ensure the randomness of calculation, the average value of the results of 50 runs is calculated, and the best and worst values are selected as auxiliary values. The best values calculated by the different algorithms are highlighted in bold.

It can be observed from Table 4 that the J value and running time are affected by different databases and pre-classification algorithms. However, on average, the FCM-IEPSO method in this paper obtains the smallest J value, and its running time is relatively short in three different datasets. In terms of operation time, the FCM-IEPSO algorithm performs better on SAVEE, which takes only 0.71 seconds more than the original PSO.

In order to verify the impact of pre-classification layer based on different algorithms on SER system, we evaluate accuracy, F-score, average recall, average precision, and average specificity based on three datasets, as shown in

Table 6. Classification test for different pre-classification algorithm on EmoDB.

Algorithm	Index
	Accuracy (%)	F-Score	Ave Recall	Ave Precision	Ave Specificity
NONE	77.60	0.77	0.77	0.77	0.95
FCM [56]	84.30	0.84	0.84	0.85	0.87
FCM-LPSO	83.27	0.81	0.81	0.84	0.87
FCM-EPSO	86.36	0.85	0.85	0.87	0.89
FCM-IEPSO	90.00	0.90	0.90	0.90	0.91

,

Table 7. Classification test for different pre-classification algorithm on SAVEE.

Algorithm	Index
	Accuracy (%)	F-Score	Ave Recall	Ave Precision	Ave Specificity
NONE	65.40	0.66	0.66	0.65	0.67
FCM [56]	73.85	0.74	0.73	0.74	0.75
FCM-LPSO	74.54	0.74	0.74	0.75	0.77
FCM-EPSO	78.39	0.78	0.79	0.78	0.80
FCM-IEPSO	80.66	0.81	0.82	0.81	0.84

and

Table 8. Classification test for different pre-classification algorithm on eNTERFACE’05.

Algorithm	Index
	Accuracy (%)	F-Score	Ave Recall	Ave Precision	Ave Specificity
NONE	58.65	0.59	0.59	0.60	0.62
FCM [56]	64.30	0.64	0.65	0.65	0.67
FCM-LPSO	63.65	0.64	0.64	0.65	0.67
FCM-EPSO	70.96	0.70	0.70	0.70	0.75
FCM-IEPSO	72.47	0.72	0.73	0.72	0.78

. In addition to the algorithms covered in Table 4 and Table 5, we also compared the system without a pre-classification layer to analyze the impact of this layer. For more comparable results, Table 6, Table 7 and Table 8 also demonstrate the performance of the pre-classification model from [56].

Obviously, the accuracy of the system without pre-classification is much lower than that of the system with pre-classification. Compared with the model from [56], our model increases 5.70%, 6.81%, and 8.17%, respectively, on EmoDB, SAVEE, and eNTERFACE’05. The results of FCM derived from LPSO are slightly worse on EmoDB and eNTERFACE’05 and slightly better on SAVEE than that of [56]. Although FCM-LPSO has little impact on system performance, proposed FCM-IEPSO still performs well.

5.3. Impact of the Proposed Multi-Layer SVM

The purpose of this subsection is to observe the impact of the proposed multi-layer SVM on the recognition accuracy of individual emotions and the overall recognition rate of the system.

The pre-classification method in this subsection uses the proposed FCM-IEPSO. Figure 6, Figure 7 and Figure 8 show the confusion matrix of each emotion using multi-layer SVM and SVM [57] as classifier, respectively. Among the seven emotions on EmoDB, six emotion accuracies increase, and one emotion accuracy (sadness) decreases in Figure 6. The success rates of anger, boredom, disgust, fear, happiness, and neutralness increased by 0.07, 0.01, 0.06, 0.09, 0.01, and 0.11. The recognition accuracies of all seven emotions on SAVEE increased in Figure 7. The values 0.03, 0.04, 0.06, 0.01, 0.04, 0.03, and 0.05 are increased emotion recognition accuracies of anger, disgust, fear, happiness, neutral, sadness, and surprise. Among the six emotions on eNTERFACE’05, the recognition accuracies of five emotions increased, and happiness decreases by 1% in Figure 8. The success rates of anger, disgust, fear, sadness, and surprise increased by 0.07, 0.14, 0.07, 0.05, and 0.04. Moreover, the use of multi-layer SVM greatly reduces the recognition confusion of fear and neutral and improve their success rate, respectively. In EmoDB, the success rate of fear increased from 0.77 to 0.86 and that of disgust with multi-layer SVM reached 0.98.

Table 9. Classification success rate based on different SVM classifiers.

Classifier	Accuracy (%)	F-Score	Ave Recall	Ave Precision	Ave Specificity
EmoDB
SVM [57]	85.33	0.85	0.86	0.85	0.87
multi-layer SVM	90.00	0.90	0.90	0.90	0.91
SAVEE
SVM [57]	76.93	0.77	0.78	0.77	0.81
multi-layer SVM	80.66	0.81	0.82	0.81	0.84
eNTERFACE’05
SVM [57]	66.43	0.66	0.67	0.66	0.74
multi-layer SVM	72.47	0.72	0.73	0.72	0.78

compares the impact of using SVM classifier [57] and our proposed multi-layer SVM classifier on overall performance. It can be observed that the accuracies based on EmoDB, SAVEE, and eNTERFACE’05 improved by 4.67%, 3.73%, and 6.04%. Moreover, the classification model has obtained good average specificity on three databases, which are 0.91, 0.84, and 0.78, respectively. Therefore, multi-layer SVM as a classifier improves the recognition accuracy of some indistinguishable emotions so as to make the overall system perform better.

5.4. Impact of Base Classifier

In the classification layer, we chose SVM as a base classifier. Considering the influence of classifier on SER system, we switch the base classifier of classification layer to KNN for comparison; that is, the classifier becomes multi-layer KNN. Meanwhile, we also compared the impact of using KNN [58] as a classifier and multi-layer KNN on system performance.

The pre-classification method in this subsection uses the proposed FCM-IEPSO. The confusion matrix results of the multi-layer KNN are shown in Figure 9, Figure 10 and Figure 11, and the overall success rates are shown in

Table 10. Classification success rate based on different KNN classifiers.

Classifier	Accuracy (%)	F-Score	Ave Recall	Ave Precision	Ave Specificity
EmoDB
KNN [58]	76.33	0.76	0.77	0.76	0.81
multi-layer KNN	79.27	0.77	0.78	0.77	0.83
SAVEE
KNN [58]	65.11	0.65	0.66	0.65	0.73
multi-layer KNN	71.20	0.71	0.72	0.71	0.77
eNTERFACE’05
KNN [58]	59.40	0.61	0.64	0.59	0.69
multi-layer KNN	65.42	0.61	0.64	0.60	0.70

on three data sets. It can be observed from the confusion matrix that on EmoDB, the success rate of five emotions increased and two emotions decreased; the recognition accuracy of all emotions on SAVEE and eNTERFACE’05 both showed an upward trend.

In Table 10, the success rate using the multi-layer KNN classifier increases by 2.94%, 6.09%, and 6.02% on three databases, respectively, compared with a single KNN classifier. Although the accuracy of multi-layer KNN has greatly improved compared that of KNN [58], the overall accuracy of multi-layer KNN is still lower than multi-layer SVM. Therefore, using a multi-layer can enhance the system’s performance, and SVM performs better than KNN as the base classifier.

5.5. Impact of Model Performance

In order to evaluate the performance of the proposed MLHF-SVM model, we compare the accuracy of our model with the accuracies of others from [56,57].

Table 11. Performance and method comparison with other literature.

Approach	Feature Extraction	Pre-Classification	Classifier	Database	Accuracy (%)
[57]	Spectral features with FCM	-	SVM	EmoDB	80.62
				SAVEE	74.20
				eNTERFACE’05	63.38
[56]	Prosodic features Spectral features	FCM	Random forest	EmoDB	77.97
				SAVEE	72.85
				eNTERFACE’05	64.39
Proposed method	Prosodic features Spectral features	FCM-IEPSO	SVM	EmoDB	85.33
				SAVEE	76.93
				eNTERFACE’05	66.43
Proposed method	Prosodic features Spectral features	FCM-IEPSO	multi-layer SVM	EmoDB	90.00
				SAVEE	80.66
				eNTERFACE’05	72.47

shows the processes, methods, and recognition accuracies of these models based on EmoDB, SAVEE, and eNTERFACE’05.

It can be observed from Table 11 that the accuracies of model from [57] reach 80.62%, 74.20%, and 63.38%, and the accuracies of model from [56] reach 77.97%, 72.85%, and 64.39% on three databases. Ref. [57] selected MFCC features using FCM clustering and used an original SVM classifier to recognize speech emotion. They did not consider the impact of certain emotions and identification characteristics on overall performance. Compared with [57], the results of our model with original SVM improved by 4.71%, 2.73%, and 3.05% on EmoDB, SAVEE, and eNTERFACE’05 due to the pre-classification of our FCM-IEPSO algorithm. After adopting multi-layer SVM as the classifier, our model performs better.

Ref. [56] extracted spectral features and prosodic features, preprocessed features with original FCM, and identified emotions with random forest. They did not consider that FCM can easily fall into local minima. The results of our MLHF-SVM model are 12.03%, 7.81%, and 8.08% higher than the model of [56]. Therefore, the FCM-IEPSO algorithm we presented performs better in pre-classification. Compared with these state-of-art technologies, our model has more comprehensive consideration, higher recognition accuracy, and better performance.

6. Conclusions

An MLHF-SVM model based on clustering and classification was proposed for speech emotion classification. In MLHF-SVM, in order to alleviate the classification error caused by identified information, the proposed FCM-IEPSO divides the feature datasets into corresponding subclasses according to prior information, and multi-SVM is presented to distinguish the emotional information of each subset. Meanwhile, in the process of feature extraction, the personalized features and non-personalized features of the speech sample are extracted simultaneously to render the speech signal more representative. The validity of the model is simulated by comparing different clustering algorithms and two classifier models on the EmoDB database. It is observed that the proposed IEPSO performs better than PSO, LPSO, and EPSO in alleviating the initial value and local minimum of fuzzy clustering. When IEPSO and FCM jointly serve for the pre-classification step, FCM-IEPSO performs better than other methods in the speed term, indicating that MLHF-SVM will not induce additional efficiency burdens. When the multi-classifier method is used for classification, the success rate is improved regardless of classifier used. By using the method we suggested, the maximum accuracy rate was achieved at 90.00% with SVM classifiers.

Our future studies will focus on more intelligent algorithms and methods of training classifiers in order to improve the performance of emotion recognition. In recent years, adversarial training, as one of the fields of machine learning, is a new training method for emotion recognition. Furthermore, we will examine techniques for labeling phonetic emotions more accurately in a more natural context.