Robust hierarchical image-augmented navigation in urban terrain with 3D landmarks

Most current navigation systems rely on GPS for accurate position. However, in urban terrain, GPS is notoriously inaccurate due to the presence of multipath and obscuration of signals from different satellites. There is a burgeoning supply of aerial-collected data from aircraft flyovers, which is commonly fused into accurate 3D geometric urban models, but is not being leveraged into current urban navigation systems. This thesis explores a new systems approach to solving the urban canyon GPS-denied navigation problem by augmenting GPS with other sensors such as video sensors and inertial navigation systems (INS). The main idea is to use 3D landmarks derived from geometric urban models and correlate them to observed landmarks in video imagery, as the navigation system moves about the urban canyon. We devise herein, a robust Landmark Correlation System (LCS) capable of providing an absolute position measurement comparable to GPS in open terrain, and additionally, provide absolute north-relative azimuth and system tilt to within about a degree of accuracy. The LCS is fully automated and designed to supplement or replace GPS when unavailable by providing updates at 1Hz, similar to most GPS, to a traditional GPS/INS navigation filter.;The LCS has three key interacting hierarchical operating modes: the first mode, CL, is a coarse location function employing grid-based methods to estimate an initial position. This mode relies on matching without 1-1 correspondences of 2D extracted image features and 2D projected landmark features from the 3D database. To remain tractable, coarse location searches over quantized positions and azimuth angles that best align predicted and observed roof edge contours. In cases where multiple ambiguous locations are returned, we use a multi-hypothesis estimation framework fused with the inertial measurement unit (IMU) sensor, to disambiguate pose by using a bank of parallel extended Kalman filters (EKF) tracking each hypothesis followed by temporal integration of the innovations, comparing IMU predicted motions with each propagated hypotheses pose until an answer is identified. The second mode, PS, is the pose solver, which serves to refine the coarse pose by establishing 1-1 correspondences between projected viewable landmarks and landmark features extracted from the imagery. We use a small set of 100 pose particles conditioned upon which we perform nearest neighbor optimal assignment, enforcing mutual exclusivity, to produce alternative correspondences. We impose geometric constraints between triplet combinations of associated landmarks/features by solving the absolute orientation 6-degree of freedom pose problem, all wrapped within a RANdom SAmple Consensus (RANSAC) framework to find a maximal consistent set of correspondences. The final pose is further refined using non-linear least squares with the full inlier set. The third mode, PT, is the pose tracker, which operates similar to the pose solver except only a single pose particle is considered, based on the propagated pose solver solution. We re-evaluate the inlier set of correspondences, again using RANSAC. The LCS operates in a sequential manner through the hierarchy; as one mode fails or succeeds, the level of operation changes adaptively.;We conducted outdoor experiments in a real 520x350m GPS-denied urban canyon environment adjacent to Draper Laboratory. We constructed a database of 3D landmarks from a 3D geometric wire-frame model generated from MassGIS aerial lidar data and orthonormal imagery. A data collection system was assembled consisting of a synchronized camera (with fisheye lens), an IMU, and a data-logging computer. A 1.1km path was traversed to include challenging sections with both large numbers of clutter features, as well as tall narrow canyon sections with sparse landmarks/features. We demonstrate the system capability to coarsely locate itself reliably within a 10m/5deg grid cell with no initial prior, and handle an ambiguous situation. We show the system is subsequently able to refine its 6-dof pose and maintain lock throughout the 7240 frames over the 12 minute traverse. Accuracy while tracking was maintained to approximately 1m in position and 1deg in azimuth and tilt. In challenging tall canyon sections, where clutter-feature to landmark ratios were high, accuracies degraded to approximately 2m in position and 2deg in azimuth/tilt.