Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subsampling LOO estimates with diff-est-srs-wor start #496

Open
wants to merge 106 commits into
base: master
Choose a base branch
from

Commits on Apr 10, 2024

  1. Configuration menu
    Copy the full SHA
    bba6adf View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    781e331 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d50f4bd View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4de50b7 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c50bdbf View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. fix tests

    avehtari committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    649f0ad View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. Configuration menu
    Copy the full SHA
    aba8670 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    81b5dc3 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Configuration menu
    Copy the full SHA
    0ed8391 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Configuration menu
    Copy the full SHA
    a36a3d8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2c846a3 View commit details
    Browse the repository at this point in the history
  3. don't warn about subsampling

    avehtari committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    fbda70e View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2024

  1. improve messages

    avehtari committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    1fa7fcd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7e7fc7a View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. verbosity improvements

    avehtari committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    09223c6 View commit details
    Browse the repository at this point in the history
  2. fix

    avehtari committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    45e22a6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1cb78fc View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    98757a9 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. Configuration menu
    Copy the full SHA
    bf5af29 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. remove unneeded code

    avehtari committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    01f3bed View commit details
    Browse the repository at this point in the history
  2. remove unnecessary sum

    avehtari committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    a5c5103 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. Configuration menu
    Copy the full SHA
    a223725 View commit details
    Browse the repository at this point in the history
  2. remove unneeded code

    avehtari committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    24fc370 View commit details
    Browse the repository at this point in the history
  3. document deltas=TRUE change

    avehtari committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    36f3543 View commit details
    Browse the repository at this point in the history
  4. wcv -> wobs in summary_funs

    avehtari committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    eeef49a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    1fc4669 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2024

  1. Configuration menu
    Copy the full SHA
    9139ce0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    10ab731 View commit details
    Browse the repository at this point in the history
  3. re-add a comment

    fweber144 committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    c23fa78 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. Configuration menu
    Copy the full SHA
    b81e401 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Configuration menu
    Copy the full SHA
    71a2198 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b41ad1e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    81b4bf0 View commit details
    Browse the repository at this point in the history
  4. add function get_use_progressr() to avoid redundancies;

    also introduce global option `projpred.use_progressr` so that all users (even
    those with `progressr` installed) may refrain from using `progressr`;
    also handle the export of `use_progressr`
    fweber144 committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    497a245 View commit details
    Browse the repository at this point in the history
  5. rename p to progressor_obj to identify it more clearly and

    to avoid potential problems when exporting to parallel workers (the export is
    handled here explicitly as well)
    fweber144 committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    5ac4041 View commit details
    Browse the repository at this point in the history
  6. use argument steps of progressr::progressor() explicitly and

    fix a bug in `kfold_varsel()` where object `inds` does not exist
    fweber144 committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    825fc3d View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    1655c84 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    1633137 View commit details
    Browse the repository at this point in the history
  9. add the possibility to use progressr at the remaining occurrences o…

    …f `foreach::foreach()`
    fweber144 committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    0996e7c View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. fix a bug (could not find function "do_call") when using the `doFut…

    …ure`
    
    backend for parallelization (this issue does not get resolved when installing
    projpred (instead of using `devtools::load_all()`))
    fweber144 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8488e3e View commit details
    Browse the repository at this point in the history
  2. remove .select <- .select (the issue does not occur when installing

    projpred (instead of using `devtools::load_all()`))
    fweber144 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    118838c View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Configuration menu
    Copy the full SHA
    d663df8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b6fe949 View commit details
    Browse the repository at this point in the history
  3. don't include the argument content in the message as the argument con…

    …tent may be a list, for example
    fweber144 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    d9fdbaf View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2024

  1. Configuration menu
    Copy the full SHA
    5b2f283 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    17be148 View commit details
    Browse the repository at this point in the history
  3. attempt to fix vsel.summary

    avehtari committed Aug 11, 2024
    Configuration menu
    Copy the full SHA
    6b6520a View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. Configuration menu
    Copy the full SHA
    fc8c665 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. fixes

    avehtari committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    99e9c79 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. docs: fix minor typos

    fweber144 committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    cd4b248 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2024

  1. avoid object within cv_varsel.refmodel() (for consistency; I don'…

    …t know why
    
    both `object` and `refmodel` exist)
    fweber144 committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    7176dfe View commit details
    Browse the repository at this point in the history
  2. fix a verbose message (at ?projpred::cv_varsel, the documentation for

    argument `ndraws` explains that it is `nclusters` which decides the
    clustering/thinning mechanism by being `NULL` or not)
    fweber144 committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    4109aae View commit details
    Browse the repository at this point in the history
  3. mention thinning in the verbose message which gives information about

    the clustering/thinning used
    fweber144 committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    7616128 View commit details
    Browse the repository at this point in the history
  4. minor cleaning

    fweber144 committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    48a6af1 View commit details
    Browse the repository at this point in the history
  5. fix usage of argument summaries_fast (at that place, `sel_cv$summar…

    …ies_fast` is always `NULL`)
    fweber144 committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    c0593c6 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    0c85eb6 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    4f893f9 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    d40acd8 View commit details
    Browse the repository at this point in the history
  9. fix input for argument search_path_fulldata when running fast LOO-C…

    …V for subsampled PSIS-LOO-CV
    fweber144 committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    d1d37bd View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. for argument verbose, default to a new global option:

    `getOption("projpred.verbose", interactive())`
    fweber144 committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    b9f8368 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Configuration menu
    Copy the full SHA
    a9ce55f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    23b4a2a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    cc26cf7 View commit details
    Browse the repository at this point in the history
  4. minor enhancements

    fweber144 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    66f2e34 View commit details
    Browse the repository at this point in the history
  5. fix a verbose message (at ?projpred::cv_varsel, the documentation for

    argument `ndraws` explains that it is `nclusters` which decides the
    clustering/thinning mechanism by being `NULL` or not; the same holds for
    `ndraws_pred` / `nclusters_pred`)
    fweber144 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    e61f916 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    750c81a View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Configuration menu
    Copy the full SHA
    f84fe55 View commit details
    Browse the repository at this point in the history
  2. fix verbose message

    fweber144 committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    8dde0bc View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Configuration menu
    Copy the full SHA
    9c4d1a4 View commit details
    Browse the repository at this point in the history
  2. UNFINISHED: move out the new "mixed deltas" variant of plot.vsel(),…

    … the
    
    new column behavior of `summary.vsel()`, and the omittance of option `"best"` of
    argument `baseline`
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    ea18e9a View commit details
    Browse the repository at this point in the history
  3. Revert "UNFINISHED: move out the new "mixed deltas" variant of `plot.…

    …vsel()`, the"
    
    This reverts commit ea18e9a.
    
    Reason for the revert: It's easier if we first add everything that should be in
    the final (separate) pull request that incorporates all changes from this pull
    request (including the new "mixed deltas" plot, the new column behavior of
    `summary.vsel()` etc.). Afterwards, we can simplify the current pull request and
    leave things like the new "mixed deltas" plot and the new column behavior of
    `summary.vsel()` for the separate pull request.
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    aea7f08 View commit details
    Browse the repository at this point in the history
  4. in .onAttach(), keep the temporary "NOTE" in separate lines (to

    avoid that `\n` accidentally remains after removing the "NOTE" in the future)
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    867f29f View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2d9a652 View commit details
    Browse the repository at this point in the history
  6. simplify summaries_fast_sub <- varsel$summaries_fast$sub and `summa…

    …ries_fast_sub <- NULL` to one line
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    35cc542 View commit details
    Browse the repository at this point in the history
  7. loo_inds as stored in vsel objects was unused so far

    the intentioned usage was probably in `get_stat()` (because `loo_inds <-
    which(!is.na(lppd))` was error-prone; e.g., in the augmented-data or latent
    projection, `lppd` might consist of only `NA`s)
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    19948e3 View commit details
    Browse the repository at this point in the history
  8. in get_stat(), the is.null(summaries_fast) checks are not necessary

    also perform some cleaning in terms of whitespace and object names (in R,
    calling objects `n` is not convenient because it complicates debugging [there,
    `n` means "next"]; also, such a generic name makes it hard to search for these
    objects in the code; for `w`, the problem is that there are several different
    kinds of weights within projpred, so we should point out what they refer to)
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    f26e4fb View commit details
    Browse the repository at this point in the history
  9. avoid object name n at more places

    (in R, calling objects `n` is not convenient because it complicates debugging
    [there, `n` means "next"]; also, such a generic name makes it hard to search for
    these objects in the code; for `w`, the problem is that there are several
    different kinds of weights within projpred, so we should point out what they
    refer to)
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    319c2b7 View commit details
    Browse the repository at this point in the history
  10. the definition of loo_ref_oscale does not make sense to be placed a…

    …mong the empty initialized objects, so move it back
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    dfcc58c View commit details
    Browse the repository at this point in the history
  11. simplify an SRS-WOR value computation (if mu_baseline is NULL, …

    …then
    
    `correct_baseline` is `0` anyway)
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    f53bc48 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    59afc97 View commit details
    Browse the repository at this point in the history
  13. avoid redundant computations by moving `sqrt(srs_diffe$v_y_hat + srs_…

    …diffe$hat_v_y)` into `.srs_diff_est_w()`
    fweber144 committed Aug 23, 2024
    2 Configuration menu
    Copy the full SHA
    2727cff View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    52fd9c3 View commit details
    Browse the repository at this point in the history
  15. add a comment and a check in loo_varsel() for `!validate_search && …

    …nloo < refmodel[["nobs"]]`
    fweber144 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    81ec495 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Revert "avoid redundant computations by moving `sqrt(srs_diffe$v_y_ha…

    …t + srs_diffe$hat_v_y)` into `.srs_diff_est_w()`"
    
    This reverts commit 2727cff.
    
    Reason for the revert: See <2727cff#commitcomment-145785150>.
    fweber144 committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    348827c View commit details
    Browse the repository at this point in the history
  2. simplify definitions of mu_baseline (possible because

    `NULL$mu` is `NULL` and `rep(NULL$mu, c(4, 5))` (for example) as well)
    fweber144 committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f4f9760 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    519dac2 View commit details
    Browse the repository at this point in the history
  4. move out the new "mixed deltas" variant of plot.vsel(), the

    new column behavior of `summary.vsel()`, and the omittance of option `"best"` of
    argument `baseline`
    fweber144 committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    a7458b9 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. use a consistent order of the if cases differentiating between

    subsampled LOO (`n_loo < n_full`) and everything else (`n_loo == n_full`)
    fweber144 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    43878e1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    925f2cd View commit details
    Browse the repository at this point in the history
  3. there was only one use of var_mse_e and since value_se

    does not change from definition of `var_mse_e` to use of `var_mse_e`, the extra
    definition of `var_mse_e` can be avoided
    fweber144 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    040d05e View commit details
    Browse the repository at this point in the history
  4. remove unused mu_baseline (only unused in case of

    `is.null(summaries_baseline)`); one reason is that I'm not sure whether it was
    supposed to read `mu_baseline <- y` in that case (instead of `mu_baseline <- 0`)
    fweber144 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    0d73c8e View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2024

  1. Configuration menu
    Copy the full SHA
    288d948 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. re-document

    fweber144 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6a85b41 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ef33da6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    cdaf384 View commit details
    Browse the repository at this point in the history
  4. fixup! remove unused mu_baseline (only unused in case of

    This fixes commit 0d73c8e. However, before
    commit 0d73c8e, `is.null(mu_baseline)` should
    have never occurred because if `summaries_baseline` was `NULL`, then
    `mu_baseline` was set to `0` (and if `summaries_baseline` was not `NULL`, then
    `mu_baseline` was set to `summaries_baseline$mu` which should not be `NULL`
    either). Hence, this fixup here does not only fix commit
    0d73c8e, but also the incorrect behavior which
    existed before it.
    fweber144 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    8aacd5d View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Configuration menu
    Copy the full SHA
    9e79883 View commit details
    Browse the repository at this point in the history
  2. vsel_obj$nloo can be NULL (for vsel_obj created by varsel()), so

    fix `validate_baseline()`'s check
    fweber144 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    1127412 View commit details
    Browse the repository at this point in the history
  3. fix the get_stat() call for the reference model statistics (loo_inds

    always needs to match `summaries_fast`)
    fweber144 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    c331cb4 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8389d39 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. fix the stat %in% c("acc", "pctcorr", "auc") case in get_stat()

    (`mu_fast`, `loo_inds`, and `n_loo` were not properly updated when
    de-aggregating the aggregated dataset)
    fweber144 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    a246dd3 View commit details
    Browse the repository at this point in the history
  2. fix .tabulate_stats() (catmaxprb() also needs to be

    applied to the subsampled LOO results)
    fweber144 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    2360630 View commit details
    Browse the repository at this point in the history
  3. fix .tabulate_stats() (several steps in case of the latent projection

    also need to be applied to the subsampled LOO results)
    fweber144 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    7469aea View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. Since summaries_fast is created by a call to loo_varsel() with

    `validate_search = FALSE`, the search is not run again when creating
    `summaries_fast`. Only the performance evaluation (including the re-projections
    required for it) is run again. Hence, it would be inconsistent to treat
    `summaries_fast` like the search-related arguments of `cv_varsel.refmodel()`
    when calling `cv_varsel.refmodel()` from within `cv_varsel.vsel()`. Thus, a lot
    of code related to `summaries_fast` can be removed, which is done here.
    fweber144 committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    0dcfcf2 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Revert changes that are unrelated to subsampled LOO-CV (to find out

    why some test snapshots changed unexpectedly).
    fweber144 committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    6151b0c View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Configuration menu
    Copy the full SHA
    360354a View commit details
    Browse the repository at this point in the history