Making long-run predictions from short-run data: Using transition trees to improve forecasting accuracy

Most companies confront the need to use very limited marketing data to both forecast aggregate demand for different products or product families, and also to identify which specific customers are most likely to buy which products and when. Previous management science and econometric models have been proposed to forecast aggregate and individual customer demands, but they have been limited by (a) Inability to model heterogeneity in individual customer behaviors with high accuracy and confidence; (b) Requirements for adequate lengths of time series data (typically, 40 months or more) to support product-specific forecasting; and (c) Inability to forecast changes in individual purchase probabilities over time without making radically simplifying assumptions. This dissertation proposes a novel approach to forecasting that overcomes all three limitations. It applies techniques developed in data mining and machine learning to create robust, nonparametric (or “model-free”) estimates of probable customer behaviors directly from available marketing data. Behaviors are modeled as transitions among states. A key feature is that both the most predictively useful definitions of the states as well as the transition rates among them can be estimated from very short observation periods and at a very fine geographical level. This gives the resulting “Transition Tree-Based Forecasting” algorithm a decisive pragmatic advantage over current time series methods in situations where long series of training data are unavailable. Moreover, the technique automatically detects and exploits heterogeneity in individual behaviors to reduce the error in its forecasts, rather than requiring the introduction of new and speculative modeling assumptions to represent heterogeneity. The main contributions of the dissertation are to introduce a new conceptual framework and computational techniques for more useful practical forecasting in many applied settings; to set this new approach within the context of related techniques in machine learning and data mining that have not previously been applied to such forecasting problems; and to demonstrate in detail the practical advantages of the new approach when applied to real marketing and behavioral data from several thousand telecommunications customers.