Some new results on the architecture, training process, and estimation error bounds for learning machines

The importance of the problem of designing learning machines rests on the promise of one day delivering systems able to learn complex behavior and assisting us in a myriad of situations. Despite its long standing in the scientific arena, progress towards producing useful machines is hampered by many unanswered questions. This dissertation makes some important contributions towards this overall goal. In particular it focuses on providing a practical solution that allows to build systems that can learn and modulate dynamic behavior, on presenting an incremental learning scheme that permits to check if a learning machine has attained generalization capability just from studying its adaptation behavior, and on studying a bound that limits the learning capacity of any machine. The first contribution develops a Dynamic Neural Network (DNN), a hybrid architecture that employs a Recurrent Neural Network (RNN) in cascade with a Non-Recurrent Neural Network (NRNN). The RNN is in charge of generating a simple limit cycle while the NRNN is devoted to reshaping the limit cycle into the desired spatio-temporal behavior. The main advantage of this architecture is the simplicity of training which results from the simplification of the overall training task due to its decomposition into independent spatial and temporal learning subtasks, which in turn permits to reduce the overall training complexity to that of training a feedforward neural network alone. The second contribution of this dissertation presents an incremental learning procedure that permits to determine whether a learning system has generalized or not. The procedure employs some concepts from statistical learning theory to prove that when a system generalizes the probability that it will encounter unexpected situations decreases exponentially to zero. The third contribution uses the well known fact that the problem underlying the design of a learning machine corresponds to an estimation problem and is thus bounded by the Fisher information quantity. Given how important it is to know more about this bound, a series of properties of the Fisher information are presented.