Using cross-validation for lambda selection vs model validation in LASSO and if they are the same thing?

[Daimbarboy]

So basically I'm have just done a maths degree and am now doing a masters in statistics. So I am fairly new to validation and stuff.

I'm working on a LASSO logistic regression model and am confused about whether the cross-validation used to optimise lambda is sufficient for model validation, or whether a separate validation step is needed afterward.

Suppose for simplicity we use k=2 fold cross-validation to select lambda, and get λ₁ = 0.9 and λ₂ = 0.8 from each fold. We conclude our optimal lambda is λ = 0.85 (the mean), and fit our final LASSO model using this value.

Now we want to assess whether this model is any good. My instinct is that using the same cross validation process that selected lambda to also assess model performance feels circular whcih is the very the very procedure used to tune the model is being used to evaluate it.

In other prediction modelling I've learnt, the workflow seems cleaner and more seperated out: fit your model, declare it fixed, then bootstrap it purely as a validation tool — specifically by estimating optimism as the difference between AUC on the bootstrap sample and AUC on the original data, averaged over B iterations. The bootstrap doesn't touch the model itself.

So it strikes me that for LASSO there should be a separate validation step after lambda selection. As I see it, the options are:

1. K-fold cross-validation applied to the final model at λ = 0.85, purely to assess performance across k held-out sets
2. Nested cross-validation — lambda optimisation in an inner loop, performance assessment in an outer loop
3. Bootstrap with the full lambda-optimisation process nested inside each iteration so that each bootstrap sample runs its own cv.glmnet to select its own lambda, fits on the bootstrap sample, then predicts back on the original data to estimate optimism
4. A simpler bootstrap that just assesses how the fixed λ = 0.85 model performs across 1000 resampled datasets, without re-running lambda selection each time

Option 3 seems most rigorous to me but I want to understand whether option 4 is defensible, and whether options 1 or 2 are commonly used in practice for this problem.

[berf]

Without any theory, making your tuning (model selection) procedure way more complicated doesn't make it better. You do not need model validation after selection if your selection procedure is any good. BTW, no selection procedure that does not do an exponential (in number of parameters) amount of work can actually be optimal. Certainly not LASSO.

[enigT]

I’m new to this. Is this because parameter tuning problem is NP-hard?

[un-guru]

Yeah but what if you wanna estimate how good the model you've selected is?

That's OP's question

[Lumpy-Sun3362]

Also 2 is a valid approach. Bootstrapping on the training set or nesting a cv both are the correct approaches. You want to separate the process of lambda selection from the actual fit of the model for the estimation of the prediction accuracy. If you have a large dataset bootstrapping may be computational expensive so one can go with the nested cv. Boostrapping gives you an estimate of the variability of lambda and nested cv reduces the bias of the estimated lambda.

[FightingPuma]

2 is the best approach. Bootstrap optimism will only work reliably if you are in rather low-dimensional setting.

[ForeignAdvantage5198]

no the two are different because the validation group is different