Some diseases have genetic basis.

Figuring out which genetic variants are most responsible for a disease is difficult.

Genetic fine-mapping is a variable selection task.

However, there exists extremely high correlation among the features.

With these structures consistent model selection impossible.

Consider a much simpler dataset.

Suppose we have data with 4 features in a 2-block design
and the following relationship

\[ \mathbb E(Y_i|X_i) = X_{i,1} + X_{i,3} \] but \(\textrm{Cor} \left( X_{i,1}, X_{i,2} \right) = \textrm{Cor} \left( X_{i,3}, X_{i,4} \right) = \rho\) is very high.

We would like to make statements like…

choose an element from each of \(\{X_{\bullet,1}, X_{\bullet,2} \}\) and \(\{X_{\bullet,3}, X_{\bullet,4} \}\).¹

Consider a slightly more complicated dataset.

Suppose we have data with 9 features in a 3-block design and
the following relationship

\[ \mathbb E(Y_i|X_i) = 3 X_{i,1} + 2 X_{i,4} + X_{i,7} \] but again there is high within-block correlation.

We want to uncover ordered sets of variables.

1. First, select any variable from \(\{X_{\bullet, 1}, X_{\bullet, 2}, X_{\bullet, 3} \}\).
2. Then, select any variable from \(\{X_{\bullet, 4}, X_{\bullet, 5}, X_{\bullet, 6} \}\).
3. Then, select any variable from \(\{X_{\bullet, 7}, X_{\bullet, 8}, X_{\bullet, 9} \}\).

Let’s consider generating the first variable set
(or univariate model set).

To construct model sets, we first acknowledge that each model has a probability of minimizing the in-sample error.

Let’s approximate the model selection probability with resampling.

Using methods from the multinomial ranking and selection literature, we can find a threshold \(\theta\) for which \(\mathbb P( \argmax_q \gamma_q \in \{r : \hat \gamma_r \geq \max_s \hat\gamma_s - \theta \} ) \geq 1 - \alpha\) (Gupta and Nagel 1967; Panchapakesan 1971).

We can now select univariate model sets.

Recall the previous example \[ \mathbb E(Y_i|X_i) = 3 X_{i,1} + 2 X_{i,4} + X_{i,7} \] where each of the above features has 2 highly correlated noise features.

How can we construct model sets for \(k\)-variable models?

Using these tools, we can construct a single a set of univariate models. In order to obtain larger models, we need only embed this within forward selection.

d <- nstep
current_models <- '1'
candidate_models <- paste0(current_models, ' + X', 1:p)
for(i in 1:d) {
  nselect <- length(current_models)
  new_models <- c()
  for(j in 1:nselect) {
    variable_set <- find_variable_set(candidate_models[j]) # returns vector of variable names
    new_models <- c(new_models, paste0(candidate_models[j], ' + ', variable_set))
  }
  current_models <- new_models
}

Example graphical output

🚨There is no stopping rule!🚨

We can generate a stopping rule with cross-validation.

Let \(\mathcal D^i\) equal \(\mathcal D\) but with element \(i\) replaced with an iid copy of \((X_i, Y_i)\). For all \(\hat f\), the assumptions require that the first and second order stability of the loss follows

\[\begin{align*} \nabla_i \ell & := \ell(X_0; \hat f (\mathcal D)) - \ell(X_0; \hat f (\mathcal D^i)) & << n^{-1/2} \\ \nabla_j \nabla_i \ell & & << n^{-3/2}. \end{align*}\]

We then can generate model confidence sets.

Consider the ordered variable set example with a low signal-to-noise ratio.

We apply this theory to risk differences for better power.

Using this theory, we can generate model confidence sets to compare models of different complexity.

Suppose we have data of the form \[ \mathbb{E}(Y_i|X_i) = X_{i,1} + X_{i,2} \] and \(\mathcal{M} = \{ f_1, f_{1,2} \}\) are linear models that use the first feature and both features, respectively.

• If the model confidence set \(\mathcal M^*\) only contains \(f_{1,2}\), it is indeed better than \(f_1\).
• If the model confidence set \(\mathcal M^*\) both models, \(f_{1,2}\) is not meaningfully better than \(f_{1}\).

There are 2 ways to apply the cross-validation model confidence set
to MPS.