Confidence envelopes for parametric model selection criteria and post-model selection inference

In choosing a candidate model in likelihood-based modeling via an information criterion, the practitioner is often faced with the difficult task of deciding just how far up the ranked list to look. Motivated by this pragmatic necessity, we construct an uncertainty band for a generalized (model selection) information criterion (GIC), defined as a criterion for which the limit in probability is identical to that of the normalized log-likelihood. This includes common special cases such as AIC & BIC. The method starts from the asymptotic normality of the GIC for the joint distribution of the candidate models in an independent and identically distributed (IID) data framework, and proceeds by deriving the (asymptotically) exact distribution of the minimum. This is a non-standard result from the theory of order statistics since although the original data are IID, the sample of GIC values are in fact dependent. The calculation of an upper quantile for its distribution then involves the computation of multivariate Gaussian integrals, which is amenable to efficient implementation via the R package "mvtnorm.'' The performance of the methodology is tested on simulated data by checking the coverage probability of nominal upper quantiles and compared to the three versions of bootstrap, parametric, non-parametric, and semi-parametric. All methods give coverages close to nominal for large samples except the non-parametric, but the bootstrap is two orders of magnitude slower. The methodology is subsequently extended to two other commonly used model structures: regression and time series. In the regression case, we derive the corresponding asymptotically exact distribution of the minimum GIC invoking Lindeberg-Feller type conditions for triangular arrays and are thus able to similarly calculate upper quantiles for its distribution via multivariate Gaussian integration. The bootstrap once again provides a default competing procedure, and we find that similar comparison performance metrics hold as for the IID case. The time series case is complicated by a far more intricate asymptotic regime for the joint distribution of the model GIC statistics. Under a Gaussian likelihood, the default in most packages, one needs to derive the limiting distribution of a normalized quadratic form for a realization from a stationary series. Under conditions on the process satisfied by ARMA models, a multivariate normal limit is once again achieved using the joint distribution of the normalized quadratic forms based on Toeplitz matrices as covariance matrices. The bootstrap can however be employed for its computation, whence we are once again in the multivariate Gaussian integration paradigm for upper quantile evaluation. Comparisons of this bootstrap-aided semi-exact method with the full-blown bootstrap once again reveal a similar performance, but faster computation speeds. Also, the derived asymptotic normal distribution for GIC values can be applied for real-world applications for all data structures. Using the uncertainty band for GIC values, we constructed a confidence envelope for the minimum GIC, we could see how far we needed to look at the ranked models when it comes to selecting the best model for the data. This method allowed us to spend less time in this process rather than considering all candidate models and see how confident we are in our AIC or BIC. One of the most difficult problems in contemporary statistical methodological research is to be able to account for the extra variability introduced by model selection uncertainty; the so-called post-model selection inference (PMSI). We explore ways in which the GIC uncertainty band can be inverted to make inferences on the parameters which can be used in a parametric setting. This is being attempted in the IID case by pivoting the CDF of the asymptotically exact distribution of the minimum GIC. For inference one parameter at a time, and a small number of candidate models, this works well, whence the attained PMSI confidence intervals are wider than the MLE-based Wald, as expected. For a large number of models, this approach is predictably quagmired by the lack of monotonically of the CDF as a function of the parameter and other numerical issues.

Embargo status: Restricted until 09/2027. To request the author grant access, click on the PDF link to the left.