On Correntropy-Based Machine Learning Models for Nonlinear Signal Processing: Addressing Sparsity, Recursive Estimation and Outlier-Robustness

Abstract

This thesis investigates machine learning methods based on information theory, reproduction kernel Hilbert spaces and recurrent neural networks to address nonlinear dynamical system modeling problems, such as system identification and time series prediction. It is argued that such methods are capable of providing the correct treatment to data generated by non-linear and non-stationary systems, often immersed in scenarios contaminated with non-Gaussian noise. These characteristics make the problem of model building considerably more complex, especially when the parameter estimation process must be performed online. That said, this thesis proposes to use the concept of correntropy, which is fundamental within the informationtheoretic learning framework, in conjunction with kernel adaptive filtering methods and recurrent neural network architectures to develop new outlier-robust algorithms endowed with online learning capability to be applied to nonlinear dynamical system modeling tasks. Among the contributions in this thesis, the following ones are highlighted. (i) Introduction of a sparse variant of the kernel correntropy learning (CKL) model that improves on the original CKL model by using the approximate linear dependence sparsity criterion and recursive computation of kernel matrices. (ii) Introduction of a second variant of the CKL model, also sparse and online, but now equipped with a fully adaptive dictionary, that is, it can grow or shrink in size as time passes. (iii) A new solution of the primal CKL model that is based on random Fourier features (RFF) which are used as a positive definite kernel function to induce a reproducing kernel Hilbert space with predefined dimensionality. (iv) Proposition of an alternative method of constructing dictionaries through the Kullback–Leibler divergence, a method that is applied to regularized networks in the reproducing kernel Hilbert space. (v) Introduction of a new approach to online learning called echo states with a recursive kernel of maximum correntropy, whose distinction is to put forward a spatiotemporal mapping using reservoir computing. (vi) Finally, a recurrent neural network with a training algorithm based on correntropy is proposed to model the dynamics of chaotic time series. All predictive models resulting from the six proposals are evaluated in challenging scenarios using a variety of small and large-scale data sets, for different levels of outlier contamination. The results achieved reveal that the proposed models are in fact robust to outliers, being able to maintain high predictive power even under online and non-stationary learning.