Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hierarchical stacking #174

Open
yao-yl opened this issue May 17, 2021 · 3 comments
Open

Add hierarchical stacking #174

yao-yl opened this issue May 17, 2021 · 3 comments
Labels

Comments

@yao-yl
Copy link
Collaborator

yao-yl commented May 17, 2021

Given the new research progress on "hierarchical stacking", it would be useful to implement this new tool into the loo package.

This new tool needs more user customization: after a few models are fit, the user needs to

  1. Run pointwise model evaluation. For IID data, this is highly automated.
  2. Specify what variable that stacking weight should depend on. This cannot be automated.
  3. Come up with a stacking model. A template may look like https://github.com/yao-yl/hierarchical-stacking-code/blob/main/well_switch/optim.stan. But this model could be more complicated too.
  4. Specify hyper prior.

The challenge for putting this package into the present package lies in that

  1. Extra dependence on stan.
  2. May even need stanglm/brms syntax to write stacking models.
@yao-yl yao-yl added the feature label May 17, 2021
@jgabry
Copy link
Member

jgabry commented May 18, 2021

Yeah it would be nice to support this. Regarding the extra dependence on Stan, would the loo package have to run the Stan code from step 3 for the user?

@yao-yl
Copy link
Collaborator Author

yao-yl commented May 18, 2021

Perhaps in this case the loo package can serves as a stan-model-generator: compute pointwise loo, suggest a default stacking model, and generate a stan file. Then user can run/modify this generated stan file on their own.

Is it possible that there could be dependence on rstan, but such dependence is only checked/brought when certain loo::function is called?

@jgabry
Copy link
Member

jgabry commented May 18, 2021

Perhaps in this case the loo package can serves as a stan-model-generator: compute pointwise loo, suggest a default stacking model, and generate a stan file. Then user can run/modify this generated stan file on their own.

Interesting idea. Would be curious to know what other people think of this option, but yeah that's a possibility.

Is it possible that there could be dependence on rstan, but such dependence is only checked/brought when certain loo::function is called?

In theory yes, but RStan isn't a typical package so we'd need to check if that works (but I think it should). Do you need to call rstan::log_prob or rstan::grad_log_prob? If not then I think it would actually be easier to do this with CmdStanR than RStan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants