crps for non Bayesian models

[bnicenboim]

Related to #220
Since crps should become MAE for deterministic models, it could be nice if this works out of the box for the predictions of non-Bayesian models. Right now there is the need of converting a vector of prediction into a matrix, I guess crps could do that automatically.

fit <- lm(kid_score ~ mom_hs + mom_iq, data = kidiq)
ypred <- predict(fit)
# I think that this step shouldn't be needed:
ypred <- matrix(ypred,nrow = 1)
crps(ypred, ypred,  y= kidiq$kid_score)

I can implement this step, if that's fine.

I can also show how to compare Bayesian and non-Bayesian models in a vignette, I'll be preparing this for a course anyway.

[jgabry]

I can implement this step, if that's fine.

Sounds ok to me, thanks.

I can also show how to compare Bayesian and non-Bayesian models in a vignette, I'll be preparing this for a course anyway.

That would be great. @avehtari Any reason not to have something like this?

[jgabry]

fit <- lm(kid_score ~ mom_hs + mom_iq, data = kidiq)
ypred <- predict(fit)
# I think that this step shouldn't be needed:
ypred <- matrix(ypred,nrow = 1)
crps(ypred, ypred,  y= kidiq$kid_score)

One potential issue that occurs to me is that in the non-Bayesian case the predictions ypred are deterministic (if you're using predict after lm you'll get the same predictions every time), whereas in the Bayesian case we can get conditionally independent predictions (e.g. calling posterior_predict twice gives different predictions). Tagging @yao-yl @LeeviLindgren @avehtari, who have thought more about CRPS than I have.

[bnicenboim]

On Sat, Apr 1, 2023, 11:00 PM Jonah Gabry ***@***.***> wrote: fit <- lm(kid_score ~ mom_hs + mom_iq, data = kidiq)ypred <- predict(fit)# I think that this step shouldn't be needed:ypred <- matrix(ypred,nrow = 1) crps(ypred, ypred, y= kidiq$kid_score) One potential issue that occurs to me is that in the non-Bayesian case the predictions ypred are deterministic (at least if you're using predict after lm you'll get the same predictions every time), whereas in the Bayesian case we get independent predictions (conditional on the model parameters). Tagging @yao-yl <https://github.com/yao-yl> @LeeviLindgren <https://github.com/LeeviLindgren> @avehtari <https://github.com/avehtari>, who have thought more about CRPS than I have.

Yes, but that shouldn't be a problem because in that case the CRPS reduces to -MAE (max abs error), which is actually the case when the predictions are deterministic. But one thing about that I don't really get about the current implementation is why the user is asked to input the predictions of a Bayesian models twice given that for each data point you have 4000 (Or as many as draws) predictions. One could ask for a single matrix of predictions with at least two predictions for data point and split it in two, right? —

…

Reply to this email directly, view it on GitHub <#221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNUQ6QJYWTFT67D5ABZ633W7CJOFANCNFSM6AAAAAAWOP2U4M> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[avehtari]

I think a better place for functions supporting lm would be either performance (part of easystats) or yardstick (part of
tidymodels). performance package already supports many metrics for non-Bayesian and Bayesian models (stanreg and brms) and uses loo package for the Bayesian LOO, so I guess that would be the better option to start with.

[bnicenboim]

ok, understood, feel free to close then.

[avehtari]

Please tell if this idea works with performance package, and I'm happy to support improving performance package to use the new loo package features