Welcome Stan 2.27 and beyond!

[hsbadr]

Summary:

This adds support to the develop branch of stan. One of the major disadvantages of RStan is the delayed support to the latest version of Stan and being behind Stan development. This tries to make RStan follow Stan development.

• Using Stan transpiler (stanc3) to generate C++ code from Stan code.
• Reorganized directory structure and removed redundant math submodule.
• Updated the symlinks in StanHeaders/inst/include for the reorganized directory structure.
• Updated the stan and math submodules, recursively, and fixed related issues.
• Updated the stan and math headers.
• Updated GHA workflow to install the development version.
• Updated GHA workflow to build and upload the source packages (Artifacts).
• Updated GHA workflow to build and upload binary packages (Artifacts).
• Updated GHA workflow to upload separate source packages for StanHeaders and rstan.
• Updated package dependencies.
• Updated StanHeaders build script.
• Using StanHeaders build script in GHA workflows.
• Installing StanHeaders in a separate step in GHA workflows.
• Robust installation of the development version of StanHeaders on Windows, macOS, and Linux.
• Fixed rstan compilation errors on Windows:
  • Addressed symlinks-related issues.
  • Defined STRICT_R_HEADERS (see define STRICT_R_HEADERS to avoid definition of Free, ... RcppCore/Rcpp#898).
• Supported the new TBB interface (see Update TBB interface and allow using external lib RcppCore/RcppParallel#141, Update TBB interface and allow using external lib math#2257, and Fix the new TBB interface and use tbb::task_arena math#2261).
• Respecting RcppParallel flags for rstan plugin.
  • RcppParallel now supports the new interface of Intel TBB and allows using external library, including oneTBB which uses a new interface. This requires specific compiler flags to link to the correct version of TBB library.
• Added support for stan::io::validate_dims.
• Introducing a new stanc_process() helper function to process Stan code.
• Updated stanc to print errors and return code for debugging.
• Linking the newest version of Sundials form math (currently, sundials 5.7.0).
• Fixed rstan dependency on the development version of StanHeaders.
• Fixed automatic installation of StanHeaders in the GitHub workflows.
• Fixed no valid constructor errors and rebuilt the Rcpp module to expose C++ classes in R.
• Fixed Stan and C++ code preprocessing and parsing issues.
• Fixed V8 (Embedded JavaScript and WebAssembly Engine for R) issues.
• Fixed (possibly) invalid URLs and used HTTPS.
• Formatted Rcpp module C++ code using clang-format.
• Updated DESCRIPTION, bumped version, and added contributor.
• Tested code generation, sampling, and reverse dependencies, including rstanarm and brms.
• Added support for within-chain threading (tested with Welcome Stan 2.27 and beyond! #887, backends: rstan: Support within-chain threading with Stan >= 2.25 paul-buerkner/brms#1074, and Fix the new TBB interface and use tbb::task_arena math#2261).
  • A new rsran option threads_per_chain has been added to control the within-chain number of threads (n):

  rstan::rstan_options(threads_per_chain = n)

  • If the model is compiled with threading support, the number of threads to use in parallelized sections within an MCMC chain (e.g., when using the Stan functions reduce_sum() or map_rect()). The actual number of CPU cores used is chains * threads_per_chain where chains is the number of parallel chains. For an example of using threading, see Reduce Sum: A Minimal Example.
• Initializing TBB arena and global control only if threads_per_chain > 1 (See Fix the new TBB interface and use tbb::task_arena math#2261).
• Linking to Boost C++ headers (BH >= 1.75); see Roll back integrate 1d patch math#2342.
• Added OpenCL headers to StanHeaders; see Add OpenCL header to StanHeaders #715.
• Exposed stanc3 flags in stanc and stan_model:
  • allow_optimizations: Allow the compiler to apply all optimizations to the Stan code. Currently not implemented.
  • standalone_functions: If set, the generated C++ will be the standalone functions C++ code.
  • use_opencl: If set, try to use matrix_cl signatures.
  • warn_pedantic: Emit warnings about common mistakes in Stan programs.
  • warn_uninitialized: Emit warnings about uninitialized variables. Currently an experimental feature.
• Fixed expose_stan_functions with stanc3.
• Updated and cleaned up rstan startup messages.
• Updated parse_data() to remove trailing double underscores and invalid regex matches for working with Eigen::Map types in the Stan model (thanks to @SteveBronder).

This has been tested on R Under development (unstable) (2021-06-14 r80502) -- "Unsuffered Consequences".

Intended Effect:

Full support to Stan development.

How to Verify:

• Remove old versions of StanHeaders and rstan:

  remove.packages(c("StanHeaders", "rstan"))

• Install remotes package

  install.packages("remotes")

• Install the development version ofStanHeaders:

  remotes::install_git("https://github.com/hsbadr/rstan", subdir = "StanHeaders", ref= "develop")

  # OR

  remotes::install_github("hsbadr/rstan/StanHeaders@develop")

• Install the development version of rstan:

  remotes::install_git("https://github.com/hsbadr/rstan", subdir = "rstan/rstan", ref = "develop")

  # OR

  remotes::install_github("hsbadr/rstan/rstan/rstan@develop")

• Run RStan examples, such as those in this vignette,

• Reinstall rstanarm and brms and run their tests.

• Test your own Stan models.

Side Effects:

Need to double check for backward compatibility.

Documentation:

N/A

Reviewer Suggestions:

@bgoodri & @rok-cesnovar

Copyright and Licensing

Hamada S. Badr [email protected]

[SteveBronder]

Why do you want to remove allow_optimizations, if it isn't implemented by default?

We can leave it, but I'm worried people are going to see it and go, "well yes of course I want all the optimizations" whenever we haven't tested when and where some of the optimizations are good. Like in the stan-dev/stanc3#549 I'm proposing we actually change turning all optimizations to -Od where the d is for dangerous

[hsbadr]

I think we can replace allow_optimizations argument in stanc with optimization_level when that PR is merged. We'll handle the correct stanc3 flag internally and describe the different levels (and our recommendations) in the documentation.

[wds15]

Any news from openMX as to when they push things to CRAN? I recall that openMX is the limiting factor keeping us from uploading rstan 2.26 everyone is waiting for, right?

[hsbadr]

Any news from openMX as to when they push things to CRAN? I recall that openMX is the limiting factor keeping us from uploading rstan 2.26 everyone is waiting for, right?

I'm not sure about the current stage of the preparation for CRAN submission. @bgoodri hasn't been responding and I don't know if he's waiting for something else besides OpenMx release.

[jpritikin]

@wds15 We submitted OpenMx w/ the needed changes to CRAN back in 26 March. I'm not sure what's going on with CRAN, but they seem incredibly slow this year. I gave them a nudge this morning.

[jpritikin]

I'm glad to report that CRAN approved OpenMx today.

[wds15]

R 4.1.0 is out for some days now making RTools 3.x history on Windows. Does that help us to get rstan to CRAN any time soon?

[hsbadr]

Does that help us to get rstan to CRAN any time soon?

I defer to @bgoodri.

[hsbadr]

@bgoodri @rok-cesnovar Is there any progress on the CRAN release? If you're busy, would it make sense for me to take care of it? I'll need @bgoodri's help, as the maintainer of rstan. I think we should plan to release rstan v2.27.0 on time and follow Stan release cycle (with likely different minor version number for minor updates and bug fixes, etc.).

[bgoodri]

I've been falling behind. Once I get into my office Friday I'll restart the computers so I can run all the reverse dependency tests, and we can see where everything is at.

…

On Thu, Jun 3, 2021 at 12:04 AM Hamada S. Badr ***@***.***> wrote: @bgoodri <https://github.com/bgoodri> @rok-cesnovar <https://github.com/rok-cesnovar> Is there any progress on the CRAN release? If you're busy, would it make sense for me to take care of it? I'll need @bgoodri <https://github.com/bgoodri>'s help, as the maintainer of rstan. I think we should plan to release rstan v2.27.0 on time and follow Stan release cycle (with likely different minor version number for minor updates and bug fixes, etc.). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#887 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZ2XKWLBQMNFA3YMNS3KLLTQ35LRANCNFSM4VC4HPFA> .

[hsbadr]

run all the reverse dependency tests

@bgoodri Should we update #912 to 2.27.0? If so, we need to create the corresponding branches in Math & Stan. I'd like to coordinate with you on this.

[wds15]

@bgoodri are there major hickups with the reverse dependency thing? In case my packages - RBesT & OncoBayes2 - need attention, then let me know; maybe this will benefit others as well.

[hsbadr]

I'm glad to report that CRAN approved OpenMx today.

@jpritikin Unfortunately, Intel TBB is required for Stan/Math now. Could you let OpenMx link to RcppParallel and add its flags (RcppParallel::CxxFlags() and RcppParallel::LdFlags())? This will fix the following error:

StanHeaders/include/stan/math/prim/core/init_threadpool_tbb.hpp:12:10:
    fatal error: tbb/task_scheduler_init.h: No such file or directory
   12 | #include <tbb/task_scheduler_init.h>

Check the complete OpenMx installation log here, using the new Stan/Math.

[jpritikin]

@hsbadr Thanks for the heads up. We'll get that changed and submitted to CRAN, ASAP.

[hsbadr]

We'll get that changed and submitted to CRAN, ASAP.

Great! Thanks @jpritikin!