Adaptive Kernel-based regression for robust system Identification

Abstract

The Least Squares Support Vector Regression (LSSVR) and Fixed Size LSSVR (FS-LSSVR) models are interesting alternatives to the Support Vector Regression (SVR). Those are derived from cost functions based on the sum-of-squared-errors (SSE) and equality constraints, unlike the SVR model, whose associated quadratic programming problem does not scale up well, and besides consumes considerable processing time. The optimization problems of the LSSVR and FS-LSSVR models become simpler because they rely on the ordinary least squares method to find a solution. For the LSSVR model, nevertheless, the solution thus found is non-sparse, implying the use of all training data as support vectors. In turn, the formulation of the FS-LSSVR model is based on the primal optimization problem, which leads to a sparse solution (i.e. a portion of the training data is used by the predictor). However, there are applications in system identification and signal processing in which online parameter estimation is required for each new sample. In this sense, the application of kernelization to linear filters has helped to establish an emerging field, that of kernel adaptive filtering for nonlinear processing of signals. A pioneering algorithm in this field is the Kernel Recursive Least Squares (KRLS) estimator. One of the contributions of this thesis consists in using the KRLS algorithm to transform the LSSVR model into an adaptive and sparse model. Beyond the question of the sparsity of the solution, the additional contributions of this work are motivated by the appropriate treatment of non-Gaussian noise and outliers. As the LSSVR and FS-LSSVR models, the KRLS model is also built upon an SSE cost function, which guarantees optimal performance only for Gaussian white noise. In other words, the performance of those models tend to considerably degrade when that condition is not observed. That said, four robust approaches for the LSSVR, FS-LSSVR and KRLS models are developed in this thesis. The framework of the robust parameter estimation known as the M-estimation is used for this purpose. For the LSSVR model, a more heuristic approach is followed, in which a robust, but non-sparse, is simply obtained by replacing the least squares estimation method by the RLM algorithm (a robust version of the RLS). For the FS-LSSVR and KRLS models, more theoretical approaches are developed, in which their original cost functions are changed in order to obtaining the proposed robust models. The performances of the proposed models are comprehensively discussed in robust system identification tasks with synthetic and real-world datasets in scenarios with k-steps ahead prediction and free simulation