-
-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Pareto-khat diagnostic #237
Comments
Draft of functions
|
After some discussion with @avehtari I've been working on cleaning up the functions added in @ozanstats work. @paul-buerkner one thing to get your opinion on: |
Both works. Could there perhaps be two functions? One that just returns khat and an extended version with diagnostics? I prefer that |
Maybe the alternative is to have a |
Ok, I'll try a few options and add some example outputs here |
I've now implemented a preliminary version of the tail selection and was thinking about the use cases for "right" vs "left" vs "both" when assessing Monte Carlo draws, with h(theta) = theta, and r(theta) = 1. Specifically I was wondering whether the tails should be fit on the constrained or unconstrained draws, and if the resulting convergence rates should be consistent. (Note that currently For example, for the
|
Should be fit on the scale that is reported
No. E.g. the tails of log transformed variable are much shorter.
It is expected and the one that is reported should be used.
It does make sense, and bounded tails have k<0. But usually not that interesting to report, and that's why it would be fine to report the bigger of left and right (although that will cause some overestimation) |
Thanks for the answers, the use is clearer now. Then I suggest that the default be |
I agree. |
Closed with #283 |
Pareto-khat diagnostic can be used in general for any Monte Carlo expectation estimate (not just importance sampling)
I'm drafting here different functions and their arguments. Some of these are based on the forthcoming PSIS paper revision
gpdfit()
fromloo
package (just point estimates)gpdpost()
with different marginal and joint posteriors, see extend gpdfit to return marginal posterior of k #186 loo#188pareto_khat()
function with support for draws objects, with argumentsreturns
ps_min_ss(k)
returns minimum sample size for reliable Pareto smoothed estimateps_khat_threshold(S)
function for computing khat-threshold for reliable Pareto smoothed estimate given sample sizeps_convergence_rate(k, S)
Function for computing the Pareto smoothed estimate RMSE convergence rate from k and S@paul-buerkner can you help to name the functions?
pinging also @Ozan147
EDIT: edited option names
EDIT: edited names
EDIT: edited typo log -> sqrt
The text was updated successfully, but these errors were encountered: