| In this thesis, we use constructive homotopy methods both to geometrically explore the mapping capabilities of finite neural networks, and to rigorously develop a robust method for computing optimal solutions to systems of nonlinear equations which, like neural network equations, have an unknown number of solutions and may have solutions at infinity.; First, a geometric perspective is presented which separates a general input-output optimization problem into a finite-dimensional projection problem, and a manifold parameterization problem. Using this separation the topology of the solutions can be characterized using natural homotopy and differential geometry, while simultaneously yielding quantitative and intuitive insight into the mapping abilities of the nonlinear system. This approach is applied to a natural homotopy between linear and nonlinear feedforward neural networks, thereby extending the valuable geometric representations of the linear network properties to nonlinear networks. Consequently, we identify conditions under which an unique solution is guaranteed, and when infinite or multiple, either isolate or functionally related, weight solutions arise, thereby facilitating future constructive design of optimal and parasimonious nonlinear networks.; A new algorithm is developed which connects disjointed homotopy path segments by solving a secondary homotopy equation on a manifold intersecting these path segments. This approach finds multiple solutions is unknown. We develop an efficient numerical implementation for continuation on general embedded manifolds using only local coordinate maps, and demonstrate performance advantages on standard optimization benchmark problems. Using this algorithm is found that neural network equations have an extraordinary number of saddle points relative to the number of minima, making inefficient any training method which computes the absolute minimum by exhaustively finding all stationary points.; Many of the geometric results developed in the thesis are directly applicable to, or can be extended to, alternative nonlinear structures and performance measures. The two-stage homotopy approach is expected to be independently valuable for other applications, including nonlinear circuit simulation, and the numerically-robust continuation algorithms developed allow for rapid prototypes of new homotopies defined on arbitrary manifolds. |
