An evaluation of judgmental and statistical model forecasts

There is considerable controversy over the relative predictive abilities of judgmental and statistical forecasting methods. Previous empirical comparisons of these methods have generally used small samples or experimental data. This dissertation evaluates real-time forecasts of National Football League game outcomes made by 74 experts, 31 statistical models, and the betting line for the 2000 and 2001 seasons.; We found that there was no statistically significant difference between the accuracy rates of experts and statistical systems in predicting game winners or predicting winners against the line. The betting line outperformed experts and statistical systems in predicting game winners and gave unbiased predictions of margins of victory. There was no evidence that experts' forecasts improved as a season progressed, but there was some evidence that predictions of statistical systems improved over the course of a season. There were statistically significant differences among individual forecasters in their success rates of picking game winners, but all had similar (and poor) success rates in predicting against the line.; An examination of agreement among forecasters using the kappa coefficient found: (1) substantial levels of agreement in predicting games winners, (2) higher levels of agreement among forecasts made by statistical systems than among those made by experts, and (3) greater agreement in predicting game winners than in picking against the line.; Local experts (i.e., those who issued forecasts in newspapers in cities having NFL franchises) exhibited bias in favor of their home teams in predicting game winners. They were more likely to pick their home teams to win than national experts and also had lower success rates in predicting game winners. The results for local experts who predicted against the line were less clear. While these local experts chose their home teams to win more frequently against the line than did national experts, they also had a statistically significantly higher success rates in their forecasts against the line than did national experts.