Queries about the SBC procedure

[paarth-dudani]

Context: I am trying to establish an SBC pipeline for assessing miscalibration on more complex problems by testing it out on simple, toy problems.

Here I've shown loss curve, ground truth estimates compared with an estimate from SNPE with the resultant SBC rank plot:

Following is the code:

num_sbc_samples = 150  # choose a number of sbc runs, should be ~100s

# generate ground truth parameters and corresponding simulated observations for SBC.
thetas = uniform_prior.sample((num_sbc_samples,))
xs = simulator_custom(thetas, num_sbc_samples, input_array=t, initial_condition=initial_condition)
from sbi.inference.posteriors.direct_posterior import DirectPosterior

# for i in range(len(results)):
Wrapped_posterior = DirectPosterior(
    posterior_estimator = SNPE_estimate  #NFlowsFlow object
    prior=uniform_prior,
)

# run SBC: for each inference we draw 3000 posterior samples.
num_posterior_samples = 3000
num_workers = 2
ranks, dap_samples = run_sbc(
    thetas, xs, Wrapped_posterior, num_posterior_samples=num_posterior_samples, num_workers=num_workers
)
check_stats = check_sbc(
    ranks, thetas, dap_samples, num_posterior_samples=num_posterior_samples
)
print(
    f"""kolmogorov-smirnov p-values \n
    check_stats['ks_pvals'] = {check_stats['ks_pvals'].numpy()}"""
)
print(
    f"c2st accuracies \ncheck_stats['c2st_ranks'] = {check_stats['c2st_ranks'].numpy()}"
)
f, ax = sbc_rank_plot(
    ranks=ranks,
    num_posterior_samples=num_posterior_samples,
    plot_type="hist",
    num_bins=None,  # by passing None we use a heuristic for the number of bins.
)
 f, ax = sbc_rank_plot(ranks, 3000, plot_type="cdf")

Question: Is it normal to get a histogram in which the bars go beyond the 'expected variation'. Also, is it normal to get ks value closer to 0.05 such as 0.1 or 0.07 etc?

For the above I got the following values:

kolmogorov-smirnov p-values 

        check_stats['ks_pvals'] = [0.55567336]
c2st accuracies 
check_stats['c2st_ranks'] = [0.44666667]

[janfb]

Hi @paarth-dudani and thanks for your question!

Answer: yes, it is normal to see such a histogram. The grey area shows the 95% confidence region for a uniform distribution of ranks, so depending on the number of histogram bins one would expect (accept) a couple of them to lie outside. Similarly, the KS test is a hypothesis test with the null-hypothesis being that the ranks are uniform. Thus, if we get a p-value <0.05, then we might tend to reject this hypothesis in the light of the data (ranks) we have seen.

In general, I would judge the results you are showing here is normal, i.e., the estimator seems to be well-calibrated and the deviations are just random. In a real-world example you would likely see more extreme deviations from uniformity.

If you have not already, please take a look at our diagnostics tutorial here: https://sbi-dev.github.io/sbi/dev/tutorials/11_diagnostics_simulation_based_calibration/

Does this help?

Cheers,
Jan

[paarth-dudani]

Thanks for your reply!

Yes, this helps. I am assuming that the same answer holds for the empirical cdf going beyond the grey zones?