-
-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
subsampling LOO estimates with diff-est-srs-wor start #496
base: master
Are you sure you want to change the base?
Conversation
This reverts commit 781e331.
I wanted to use R2, and as I had rewrote summary stats anyway, added R2 and made all mse, rmse, and R2 to use only normal approximation with as much shared computation as possible With the added R2 support, this PR will close also #483 |
I can take a look |
I have added some comments, but I'm not done with the review yet. Besides, I think documentation needs to be updated (at least re-roxygenized, but also progressr should be mentioned), the vignettes perhaps as well, and I haven't run |
# store for later calculations | ||
mse_e <- value | ||
if (!is.null(summaries_baseline)) { | ||
# delta=TRUE, variance of difference of two normally distributed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is something missing at the end; perhaps "random variables"?
((mu_baseline - y)^2 - mse_b))[loo_inds], | ||
y_idx = loo_inds, | ||
w = wobs) | ||
cov_mse_e_b <- srs_diffe$y_hat / n_full^2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just for my understanding: This procedure for estimating the covariance between mse_e
and mse_b
assumes that the summands within mse_e
and mse_b
coming from different observations are uncorrelated, right? At first, I thought this was violated here because mu
, mu_baseline
, and summaries_fast$mu
are model-based and hence there is potential for cross-observations dependencies, but then I realized that mu
, mu_baseline
, and summaries_fast$mu
all are based on the leave-one-out principle, so is this the reason why we can assume a cross-observation correlation of zero here?
When I changed several bootstraps to analytic approximations and improved other approximations, I thought the math I was writing in the code was so trivial that I didn't write all the derivations and assumptions separately. Now I see I should have done that, as it takes also me a bit of time to re-check any of these when you ask a question, so they are not as obvious as I thought them to be. If you like, I can some day write the equations and assumptions for easier checking. Before that, at least every approximation I wrote is based on the tests at least as accurate as the earlier bootstrap, but much faster. |
R/summary_funs.R
Outdated
mse_y <- mean(wobs * (mean(y) - y)^2) | ||
value <- 1 - mse_e / mse_y - ifelse(is.null(summaries_baseline), 0, 1 - mse_b / mse_y) | ||
# the first-order Taylor approximation of the variance | ||
var_mse_y <- .weighted_sd((mean(y) - y)^2, wobs)^2 / n_full |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For mean(y)
, don't we need to take wobs
into account? I think this is similar to line var_mse_b <- .weighted_sd((mu_baseline - y)^2, wobs)^2 / n_full
where the parameter estimates from which mu_baseline
is computed also take wobs
into account.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This also concerns several other occurrences of mean(y)
here in get_stat()
.
R/summary_funs.R
Outdated
if (!is.null(summaries_baseline)) { | ||
# delta=TRUE | ||
mse_e <- mse_e - mse_b | ||
} | ||
value_se <- sqrt((value_se^2 - | ||
2 * mse_e / mse_y * cov_mse_e_y + | ||
(mse_e / mse_y)^2 * var_mse_y) / mse_y^2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If my understanding from stan-dev/loo#205 (comment) is correct, then I think we would need a trivariate delta method in the !is.null(summaries_baseline)
case (because mse_b
comes in, too). I haven't checked whether such a trivariate delta method would give the same formula as used here. Have you checked this?
For me,
does not work (runs into some error, similarly as
works for me. Does it work for you as well (on |
With |
That sounds good. The warnings probably arise from the first creation of the snapshots. If you are running the tests via Since this solution seems to be working for you, I will push a commit to |
…of `cv_varsel()`
This fixes commit 0d73c8e. However, before commit 0d73c8e, `is.null(mu_baseline)` should have never occurred because if `summaries_baseline` was `NULL`, then `mu_baseline` was set to `0` (and if `summaries_baseline` was not `NULL`, then `mu_baseline` was set to `summaries_baseline$mu` which should not be `NULL` either). Hence, this fixup here does not only fix commit 0d73c8e, but also the incorrect behavior which existed before it.
# log-normal approximation | ||
# https://en.wikipedia.org/wiki/Log-normal_distribution#Arithmetic_moments | ||
mul <- log(value^2 / sqrt(value_se^2 + value^2)) | ||
varl <- log(1 + value_se^2 / value^2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make sense to use log1p()
here (for numerical stability)?
fix `validate_baseline()`'s check
always needs to match `summaries_fast`)
(`mu_fast`, `loo_inds`, and `n_loo` were not properly updated when de-aggregating the aggregated dataset)
applied to the subsampled LOO results)
also need to be applied to the subsampled LOO results)
While working on the tests, I discovered a few bugs and fixed them. However (apart from what is still open in the comments above), we still need to think about the following:
For the tests, I think it would make sense to add subsampled-LOO Finally, while experimenting with the tests, I discovered some strange projpred/tests/testthat/test_varsel.R Lines 1452 to 1456 in 7469aea
excl_nonargs(args_cvvs_i, nms_excl_add = "validate_search") ) on tstsetup <- "rstanarm.glm.cumul.stdformul.without_wobs.with_offs.latent.default_meth.default_cvmeth.default_search_trms" . This needs to be investigated.
|
`validate_search = FALSE`, the search is not run again when creating `summaries_fast`. Only the performance evaluation (including the re-projections required for it) is run again. Hence, it would be inconsistent to treat `summaries_fast` like the search-related arguments of `cv_varsel.refmodel()` when calling `cv_varsel.refmodel()` from within `cv_varsel.vsel()`. Thus, a lot of code related to `summaries_fast` can be removed, which is done here.
I think I have forgotten the context to understand this question
Fine for me. NA handling was there before I started this PR, and I left most of that there. |
No problem, I think I figured this out and pushed commit 0dcfcf2 which resolves this.
Ok, I will check this in detail and will commit changes (if possible). |
why some test snapshots changed unexpectedly).
I just pushed commit 6151b0c which reverts changes that are unrelated to subsampled LOO-CV because I observed some unexpected changes in the test snapshots. The state before reverting these subsampling-unrelated changes is now in branch Lines 797 to 891 in 6151b0c
Lines 819 to 902 in 0dcfcf2
misc_from_fix-subsampling ) after this PR has been merged. And, of course, there is still branch mixed_deltas_plot which needs to be worked on and eventually merged as well.
|
I just pushed commit 360354a which ensures that the tests run through with this PR. The changes in the test snapshots are now as expected. However, the parts
and
from #496 (comment) are still to-do. |
Work in progress.
Tested with
Notes
cv_varsel
if nloo<n and fast PSIS-LOO result is not yet available, fast PSIS-LOO result is computedcv_varsel
if nloo<n, fast PSIS-LOO result is stored in slot $summaries_fastNext
tagging @n-kall