New ISC Project Proposal for R package SimTools

[dvats]

My co-developers (James Flegal, UC Riverside and Galin Jones, U Minnesota) and I (Dootika Vats, IIT Kanpur, India) are looking to submit an ISC grant proposal for an R package: SimTools. The primary goal of this package will be to quantify and visualize variability from simulations based methods. This includes:

• Monte Carlo simulations
• Bootstrap
• Importance Sampling
• Markov chain Monte Carlo
• Other stochastic simulation

Often when comparing methodology in statistical methodology development, researchers produce a graphical summary of their Monte Carlo simulation demonstrating "superiority" of their method. Such plots never have a variability summary in them. For example, consider comparing OLS, Ridge, and Lasso for a particular regression model over repeated simulation of the dataset. Repeating the simulation 100 times (or 2000 times), we may compare boxplot of the mean squared error. The top side-by-side boxplot is what is often shared, and the bottom side-by-side boxplot is what we hope to build using our SimTools package. The blue boxes around the estimate of the quantiles are non-conservative simultaneous confidence intervals.
simstudy100.pdf
simstudy2000.pdf

The principles of statistical simulation that allow us to make the plots above, also allow us to

• Visualize simultaneous confidence intervals for Bayesian credible intervals estimated via iid or correlated sampling
• Show variability in bootstrap estimators
• Make confidence bands around density estimates from iid or correlated sampling

Specifically for MCMC, we will equip SimTools with MCMC output analysis tools that will have the functionality of coda but uses more statistically robust tools, which we as a group have studied over the past years. This will involve streamlining our current mcmcse package (over 30k downloads and 71 citations on GoogleScholar) and producing output that summarizes MCMC objects and returns posterior visualization with Monte Carlo uncertainty.

There are a few packages that produce Bayesian posterior visualizations, like RStan, tidybayes, MCMCvis, basicMCMCplots, but none of them equip their plots with Monte Carlo errors, which we find to be crucial to reproducibility of simulation based research.

We know that this package will serve a wide range of R users and thus hope that is suitable to be considered for funding by the ISC. I welcome feedback on the appropriateness of the package, and would be happy to answer any questions on this.