Real-time tracking and analysis of Drosophila behavior and gene expression

This thesis describes new methods that make it possible to screen efficiently complex behavioral phenotypes in the fruit fly, Drosophila melanogaster. The behaviors of interest include those related to higher-order processing or 'choice' (e.g. social interactions), which are difficult to screen and quantify by conventional methods. We developed an electronic 'fly world'---a modular system of hardware and software capable of reliably tracking the motion of flies for different behavioral assays within controlled sensory environments. We developed an approach for real time tracking of multiple flies using an array of calibrated and synchronized cameras. The system constructs 3D visual hull models of multiple flies simultaneously from detected image silhouettes. The reconstructed fly visual hulls are then used by an extended Kalman filter to generate tracks of all the flies at real time rates of 60 frames/sec. We also developed pattern classification and recognition methods to describe and classify 3D fly visual hull shapes into a set of predefined states, and recognize behaviors made up by these states. Additionally, in order to correlate specific temporal patterns of gene expression with temporal patterns of animal activity and behavior, we developed algorithms to allow tissue-specific Green Fluorescent Protein and Discosoma sp. Red Fluorescent Protein reporter expression to be quantified and correlated with 3D fly movement and behavior in real time. In addition to time series data modeling approaches, we developed methods for the analysis and comparison of movement trajectories, and for the correlation of trajectories with specific complex behaviors like courtship.