Uncertainty analysis in large models

Uncertainties in large models are due to complexities such as numerous inputs and model predictions are multivariate and/or time-dependent functions of the input variables. Their assessment will help effectively interpret model results and provide guidelines to decision makers and forecasters. We classify sources of uncertainty in large models into those arising from (1) inputs, (2) model, and (3) outputs and provide a systematic methodology of current techniques corresponding to each source of uncertainty. We illustrate applications of some uncertainty analysis techniques on the Short-Term Integrated Forecasting System (STIFS)--an example of a large model--which makes forecasts of the domestic energy market of the United States.; For assessing input uncertainty in large models, we recommend using the method of prediction variance (MPV) which employs a resampling method called Latin hypercube sampling for efficient sampling of inputs from their probability distributions. MPV constructs importance measures for inputs which are used to identify the dominant input variables affecting model predictions.; Model uncertainty can be assessed by applying either model expansion or model mixing methods depending upon which of the following three categories be-fits the model: (1) models with inaccessible structure, (2) models with accessible structure, and (3) multiple models with known model structures. These methods examine model uncertainty by attaching weights to the model(s) and then propagate this uncertainty in the form of predictive distributions of the model outputs.; We propose a new method of constructing 100{dollar}beta{dollar}% prediction intervals for forecasts produced by large models which quantifies uncertainty in model outputs. Prediction limits are obtained by using a robust quantile estimation method to estimate quantiles of the distribution of conditional forecast errors. The proposed method gives improved coverage of prediction intervals for both symmetric and skewed distributions (of conditional forecast errors), it is fairly robust to violations of assumptions about the forecast errors being iid, and it is applicable to any forecasting model regardless of it's functional form. We illustrate an application of the proposed method on the STIFS model and construct 95% prediction intervals for two quarters ahead forecasts of Coal Production.