Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crps for non Bayesian models #221

Closed
bnicenboim opened this issue Mar 31, 2023 · 6 comments
Closed

crps for non Bayesian models #221

bnicenboim opened this issue Mar 31, 2023 · 6 comments

Comments

@bnicenboim
Copy link
Contributor

Related to #220
Since crps should become MAE for deterministic models, it could be nice if this works out of the box for the predictions of non-Bayesian models. Right now there is the need of converting a vector of prediction into a matrix, I guess crps could do that automatically.

fit <- lm(kid_score ~ mom_hs + mom_iq, data = kidiq)
ypred <- predict(fit)
# I think that this step shouldn't be needed:
ypred <- matrix(ypred,nrow = 1)
crps(ypred, ypred,  y= kidiq$kid_score)

I can implement this step, if that's fine.

I can also show how to compare Bayesian and non-Bayesian models in a vignette, I'll be preparing this for a course anyway.

@jgabry
Copy link
Member

jgabry commented Apr 1, 2023

I can implement this step, if that's fine.

Sounds ok to me, thanks.

I can also show how to compare Bayesian and non-Bayesian models in a vignette, I'll be preparing this for a course anyway.

That would be great. @avehtari Any reason not to have something like this?

@jgabry
Copy link
Member

jgabry commented Apr 1, 2023

fit <- lm(kid_score ~ mom_hs + mom_iq, data = kidiq)
ypred <- predict(fit)
# I think that this step shouldn't be needed:
ypred <- matrix(ypred,nrow = 1)
crps(ypred, ypred,  y= kidiq$kid_score)

One potential issue that occurs to me is that in the non-Bayesian case the predictions ypred are deterministic (if you're using predict after lm you'll get the same predictions every time), whereas in the Bayesian case we can get conditionally independent predictions (e.g. calling posterior_predict twice gives different predictions). Tagging @yao-yl @LeeviLindgren @avehtari, who have thought more about CRPS than I have.

@bnicenboim
Copy link
Contributor Author

bnicenboim commented Apr 2, 2023 via email

@avehtari
Copy link
Collaborator

avehtari commented Apr 3, 2023

I think a better place for functions supporting lm would be either performance (part of easystats) or yardstick (part of
tidymodels). performance package already supports many metrics for non-Bayesian and Bayesian models (stanreg and brms) and uses loo package for the Bayesian LOO, so I guess that would be the better option to start with.

@bnicenboim
Copy link
Contributor Author

ok, understood, feel free to close then.

@avehtari
Copy link
Collaborator

avehtari commented Apr 4, 2023

Please tell if this idea works with performance package, and I'm happy to support improving performance package to use the new loo package features

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants