Research On High-performance Training Algorithms Of TSK Fuzzy Systems And Their Applications

Nowadays,deep learning has been widely applied in many fields owing to its outstand-ing performance,such as health care,transportation,national defense,industry,agriculture,etc.However,deep learning algorithms are complex,costly in training,difficult to inter-prete,and vulnerable to adversarial attacks.These shortcomings are seriously threatening the security of intelligent systems and limiting further applications of deep learning.There-fore,the development of interpretable and lightweight machine learning algorithms is now a top priority in the development of artificial intelligence.Fuzzy systems,also known as fuzzy neural networks,which can explain their prediction using fuzzy sets and rules,may be one promising solution.Traditional fuzzy system optimization algorithms may have challenges when the data size and/or dimensionality increase,making fuzzy systems uncompetitive with other ma-chine learning algorithms.To tackle this problem,this Dissertation aims to explore how to improve the performance of Takagi-Sugeno-Kang(TSK)fuzzy system based on mini-batch gradient descent algorithms by adapting the cutting-edge theories and technologies of machine learning,especially deep learning,while maintaining the interpretability of fuzzy systems.The main contributions of this Dissertation are as follows.To improve the performance of TSK fuzzy systems,this Dissertation proposed TSK-Cons BN-UR,which integrates two novel techniques: First,uniform regularization(UR),which forces the rules to have similar average firing levels,and hence to avoid the unde-sirable case that only a few rules dominate the prediction? and,second,batch normalization(BN)in TSK fuzzy classifier,namely Cons BN,which extends BN from deep neural net-works to TSK fuzzy classifiers to expedite the convergence.At testing phase,parameters of the Cons BN can be merged into the consequent layer,keeping the original architecture of the TSK classifier unchanged.Experimental results demonstrated that UR can significantly im-prove the performance of TSK fuzzy systems,and Cons BN can further significantly improve the performance of UR.The reason why TSK fuzzy systems with Gaussian membership functions may fail on high-dimensional inputs was revealed: After transforming TSK defuzzification to an equiva-lent form of the softmax function,this Dissertation found that as the dimensionality increases,the input of softmax will monotonically decrease,causing the saturation of the softmax.The saturation will lead to problems such as significant decrease of the rule utilization rate,and more rugged loss landscape.To solve this problem,a high-dimensional defuzzification algo-rithm named HTSK was proposed,mitigating the saturation problem at the defuzzification level.Experimental results demonstrated that the proposed HTSK can significantly improve the performance of fuzzy systems in high-dimensional problems.The reason why TSK fuzzy systems are sensitive to the choice of the optimizer was revealed: The small magnitude of the normalized rule firing levels causes the gradient van-ishing problem on the rule consequent parameters,especially when the number of rules is large.Thus,the rule consequents are easily trapped into a bad local minimum with poor generalization performance.To tackle this problem,this Dissertation proposed an algorithm named HTSK-LN-Re LU,which uses layer normalization(LN)to amplify the small firing levels,and then the rectified linear unit(Re LU)to filter the negative firing levels generated by LN,maintaining the interpretability and improving the robustness.Experimental results demonstrated that the proposed HTSK-LN-Re LU can effectively alleviate the gradient van-ishing problem,and significantly improve the generalization performance regardless of the choice of optimizer and the rulebase size.Furthermore,the proposed HTSK-LN-Re LU can also reduce the performance differences caused by different optimizers.A deep fuzzy neural network named Deep HTSKD and a multi-level prediction smooth-ing(MPS)algorithm were proposed for driver drowsiness estimation.Specifically,to im-prove model’s generalization perforamance,Deep HTSKD uses a newly proposed multi-layer neural network with the Drop Out technique as feature extractor and the HTSK al-gorithm with the Drop Rule technique as regressor.The MPS approach combines temporal smoothing at feature level and label level,which significantly reduces the affect of noise on model predictions.Experimental results demonstrated that the performance of Deep HTSKD is significantly better than neural networks or fuzzy systems alone,and the MPS approach can further improve the performance of Deep HTSKD.Overall,this Dissertation solves the problems when fuzzy systems are optimized by gra-dient descent algorithms.The proposed algorithms are simple,generic and effective,which can be easily implemented by commonly used machine learning frameworks.Furthermore,the proposed algorithms are also adapted to a brain-computer interface scenario,and experi-mental results also verified the effectiveness of the proposed algorithms on this application.