On the efficient design of extreme learning machines using intrinsic plasticity and evolutionary computation approaches

Abstract

The Extreme Learning Machine (ELM) has become a very popular neural network ar- chitecture due to its universal function approximation property and fast training, which is accomplished by setting randomly the hidden neurons’ weights and biases. Although it offers a good generalization performance with little time consumption, it also offers considerable challenges. One of them is related to the classical problem of defining the network size, which influences the ability to learn the model and will overfit if it is too large or underfit if it is too small. Another is related to the random selection of input-to-hidden- layer weights that may produce an ill-conditioned hidden layer output matrix, which derails the solution for the linear system used to train the output weights. This leads to a solution with a high norm that becomes very sensitive to any contamination present in the data. Based on these challenges, this work provides two contributions to the ELM network design principles. The first one, named R-ELM/BIP, combines the maximization of the hidden layer’s information transmission, through Batch Intrinsic Plasticity, with outlier-robust estimation of the output weights. This method generates a reliable solution in the presence of corrupted data with a good generalization capability and small output weight norms. The second method, named Adaptive Number of Hidden Neurons Approach (ANHNA), is defined as a general solution encoding that allows populational metaheuristics to evolve a close to optimal architecture for ELM networks combined with activation function’s parameter optimization, without losing the ELM’s main feature: the random mapping from input to hidden space. Comprehensive evaluations of the proposed approaches are performed using regression datasets available in public repositories, as well as using a new set of data generated for learning visuomotor coordination of humanoid robots.