| Language, music, spatial configurations, event chronologies, action sequences, and other types of data exhibit hierarchical compositional structure, in which high-level entities represent combinations of lower-level entities. Compositional (part-whole) relationships serve as critical components of representation for artificial intelligence, just as taxonomic (is-a ) relationships do. Existing work with hand-built compositional hierarchies demonstrates their ability to make inferences that integrate bottom-up and top-down influences, spanning multiple levels of spatial/temporal resolution. Despite the existence of basic learning algorithms for taxonomies, there has not been analogous foundational work on learning predictive compositional hierarchies. This dissertation demonstrates that predictive compositional hierarchies can also be learned purely from primitive data in a way that is general, unsupervised, and on-line.; learning of compositional hierarchies from unsegmented sequences. Systems capable of performing the task will have many possible uses resulting from their ability to fill in missing data, resolve ambiguities, etc. I introduce two learning systems that address this problem, both employing compositional hierarchies but based on different classes of traditional learning models. The first, based on symmetric-recurrent neural networks with probabilistic semantics, provides a novel on-line structure modification rule for such networks. The second, based on n-grams, incorporates several contributions, including an on-line method for storing only high-frequency patterns, a method for weighting the statistical reliability of frequency estimates based on different lengths of observation, and two methods for combining n-grams.; The essence of on-line compositional hierarchy learning is the bottom-up identification of frequently occurring patterns. This enables the future discovery of larger patterns. Both systems introduced can compose larger patterns as they see more data, with no prespecified bound, nonetheless using smaller storage space than that taken by the data. This cumulative process has the potential to scale from fine-grained data to coarser, high-level representations tuned to the statistical characteristics of the environment, bridging a gap that has long been a stumbling block on the way to creating highly-intelligent autonomous agents. |
