| Earthquake prediction is one of the most important unsolved problems in the geosciences. Over the past decade, earthquake prediction research has been revitalized, and predictability experiments are currently active worldwide. In considering these experiments, a number of issues related to prediction evaluation are vital: a detailed experiment specification, the measure of success to be used, and a choice of appropriate reference model(s). Here, we address each of these, with an emphasis on testing prospective earthquake predictions.;We consider a general class of earthquake forecasts for which the forecast format allows a binary interpretation; that is, for any given interval of space and time, we can infer whether or not an earthquake of a given size is expected. This generalization allows us to test deterministic and probabilistic forecasts and compare the results; furthermore, the tests are easily understood because they are essentially the sum of many yes/no questions. As an introduction to binary performance measures and their wide applicability, we considered Reverse Tracing of Precursors (RTP), a recent earthquake prediction algorithm intended to forecast damaging earthquakes. We introduce and analyze several methods for measuring predictive performance but concede that the RTP experiment results are likely unstable due to the small number of earthquakes occurring to date.;In the context of an experiment with three 10 year seismicity forecasts---Relative Intensity, Pattern Informatics, and National Seismic Hazard Map---we introduce the area skill score, a measure of success derived from the Molchan diagram. Using this experiment and applying approaches from statistical hypothesis testing, we illustrate the importance of choosing an appropriate reference model, and show that added model complexity does not necessarily yield a significant improvement in predictive skill.;Having demonstrated the use of the area skill score as a performance metric, we explore its statistical properties and the related computational procedures in some detail. Based on this work and the previous experiment results, we used the area skill score to explore the evolution of regional seismicity and optimize simple forecast models. |
