Forecast evaluation and model selection


The course is designed to acquire tools and techniques to evaluate the accuracy model based forecasts and operate model selection in a variety of settings. The tools and techniques can be applied to a variety of settings ranging from macroeconomic forecast evaluation to finance, although a specific focus is devoted to financial applications. The course has several specific objectives:

  1. Discuss mathematical/statistical tools for forecast evaluation and model selection for scalar, vector and matrix valued outcomes and lay their theoretical foundation
  2. Provide clear definition and interpretation of the metrics used to assess forecast accuracy
  3. Focus on problems arising in forecast evaluation of latent target variables with emphasis on the case of volatility forecasts evaluation
  4. Analyze recent work in forecast evaluation and model selection based on forecast accuracy and discuss their empirical implementation


  1. Measures of forecast accuracy: Notion of empirical ranking. Definition of statistical loss functions (in scalar/vector/matrix spaces), construction and interpretation. Definition of economic loss functions (Minimum variance portfolio, Option pricing accuracy). Regression-based evaluation of predictive accuracy. Observable vs. Latent target variables. (3h)
  2. Special issues 1: Evaluation of nested models. Estimation sampling schemes (Expanding vs. rolling window). Structural breaks/time varying parameters. Absolute vs. conditional forecasting performance (1.5h)
  3. Tests of Forecast Comparison – Theory and Hypotheses: Single hypothesis (Diebold-Mariano, West, Clark-McCracken, Giacomini-White, Mincer-Zarnowitz), Multiple hypotheses (Superior Predictive Ability, Model Confidence Set). Simple vs. Composite hypotheses (testing equalities vs. weak inequalities) (5h)
  4. Special issues 2: Evaluation latent target variables forecasts (Volatility forecast evaluation):
    1. Introduction to ex-post measures of volatility (semi and non.parametric univariate and multivariate) (realized variance, multi-power variations, realized kernels), related issues : microstructure noise, observation frequency, sampling schemes and prefiltering, seasonality, robustification, jump detection and synchronicity (3h)
    2. Consistency of forecasts ranking under statistical and economic loss functions. Sufficient and necessary conditions for ranking consistency (4.5h)
    3. Additional topic (depending on time)
  5. Predictive density evaluation (Diebold-Gunther-Tay, Corradi-Swanson, Amisano-Giacomini) (1h)


Andersen T, Bollerslev T, Diebold F, Labys P. (2003) Modeling and forecasting realized volatility. Econometrica 71, 579-625
Barndorff-Nielsen O, Hansen P, Lunde A, Shephard N. (2008) Designing realized kernels to measure the ex post variation of equity prices in the presence of noise. Econometrica, 76 :1481-1536
Clark, T. and M. McCracken (2015), Nested Forecast Model Comparisons : A New Approach to Testing
Equal Accuracy, J. of Econometrics, Vol. 186, 160-177
Clark, T. and M. McCracken (2005), The Power of Tests of Predictive Ability in the Presence of Structural Breaks, J. of Econometrics 124(1), 1-31
Diebold. F, Mariano R. (1995), Comparing Predictive Accuracy, J. of Business and Economic Statistics 13
Elliott G., Timmermann A, (2008). Economic forecasting. J. of Economic Literature 46, 3-56.
Giacomini, R. and H. White (2006), Tests of Conditional Predictive Ability, Econometrica 74(6).
Hansen P. (2006) A test for superior predictive ability, J. Business and Economic Statistics, 23 :365-380.
Hansen P, Lunde A, Nason J. (2011) The model confidence set, Econometrica, 79 :453-497.
Hansen P, Lunde A. (2006), Consistent ranking of volatility models. J. of Econometrics, 131 :97-121
Laurent S, Rombouts JVK, Violante F. (2013), On loss functions and ranking forecasting performances of multivariate volatility models, J. of Econometrics 173 1-10
Mincer, J. and V. Zarnowitz (1969), The Evaluation of Economic Forecasts, in J. Mincer, ed., Economic
Forecasts and Expectations, 3-46
Patton A. (2011), Volatility forecast comparison using imperfect volatility proxies. J. of Econometrics,  160
West K, McCracken M. (1998), Regression-Based Tests of Predictive Ability, Int. Economic Review, 39
White H. (2000) Reality check for data snooping. Econometrica, 68 :1097-1126
Corradi, V. and N. Swanson (2001) Predictive Density Evaluation, in : Granger, C., G. Elliott and A.
Timmermann (eds.), Handbook of Economic Forecasting Vol. 1
Amisano, G. and R. Giacomini (2007), Comparing Density Forecasts via Weighted Likelihood Ratio Tests, J. of Business and Economic Statistics 25, 177-190