Exploring the computational capabilities of recurrent neural networks

This dissertation addresses the issues surrounding the computational capabilities of recurrent neural networks. My results apply not only to simple recurrent networks, Jordan networks, and higher order recurrent networks, but many other networks implemented as input-parameterized iterated functions.;The following reasons have driven my efforts to understand the computational capabilities of recurrent networks. First, the question of knowledge content arises whenever we attempt to understand how a given network produces its behavior. Second, knowing the range of what is computable by a recurrent network can guide us in their intelligent application. Finally, this knowledge may also help us to develop new training strategies that bias the network toward desirable solutions.;Recurrent networks can clearly perform complex computation by simulating machine tapes and stacks. Such simulations are always the product of design. We know the functional decomposition of the network with respect to the computation it implements because the designer can identify the intended roles of each part. Unfortunately, weak learning methods, like back-propagation, which discover operable network weights cannot explain the internal functionality of the final product. Thus, missing functional specifications force us to externally determine the recurrent network's computation process by observing its structure and behavior.;To this end, I identify three facets of recurrent networks that directly affect their emergent computational descriptions: system dynamics, input modulation of state dynamics, and output generation. System dynamics, the mapping of current state to next state, have been traditionally considered the source of complex behavior. Input modulation occurs as a finite set of input vectors induces behavior in the networks like that of iterated function systems. This selection creates state space representations for information processing states that display recursive structure. I show that the mechanism producing discrete outputs dramatically affects the apparent system complexity by imposing information processing regularities in the output stream strong enough to manipulate both complexion (number of states) and generative class of the observed computation.;As for new training methods, I outline a method of network training called entrainment learning that offers a novel explanation of the transmission of grammatical behavior structures between agents.