Odd interaction between ode_adams and threading

[WardBrian]

Description

I have noticed some odd behavior using ode_adams and STAN_THREADS. This does not recur when using ode_bdf or ode_rk45.

Example

Using a simple SIR model, courtesy of @charlesm93:

functions {
  vector sir(real t, vector y, real beta, real gamma, int N) {
    real S = y[1];
    real I = y[2];
    // real R = y[3];

    real dS_dt = - beta * I * S / N;
    real dI_dt = beta * I * S / N - gamma * I;
    real dR_dt = gamma * I;

    return [dS_dt, dI_dt, dR_dt]';
  }

}

data {
  int<lower=1> n_days;
  vector[3] y0;
  real t0;
  array[n_days] real ts;
  int N;
  array[n_days] int cases;
}

parameters {
  real<lower=0> gamma;
  real<lower=0> beta;
  real<lower=0> phi_inv;
}

transformed parameters {
  real phi = 1. / phi_inv;
  array[n_days] vector[3] y = ode_adams(sir, y0, t0, ts, beta, gamma, N);
}

model {
  //priors
  beta ~ normal(2, 1);
  gamma ~ normal(0.4, 0.5);
  phi_inv ~ exponential(5);

  cases ~ neg_binomial_2(y[,2], phi);
  // cases ~ poisson(y[,2]);
}

generated quantities {
  real R0 = beta / gamma;
  real recovery_time = 1 / gamma;
  array[n_days] real pred_cases = neg_binomial_2_rng(y[,2], phi);
  // array[n_days] real pred_cases = poisson_rng(y[,2]);
}

data:

{
    "N": 763,
    "cases": [3, 8, 26, 76, 225, 298, 258, 233, 189, 128, 68, 29, 14, 4],
    "n_days": 14,
    "ts": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
    "t0": 0,
    "y0": [762, 1, 0]
}

inits

{
    "beta": 1.00247,
    "gamma": 1.15014,
    "phi_inv": 0.008622444
}

When compiled via cmdstan with STAN_THREADS=1, I then run with

./sir sample num_chains=4 data file=sir.data.json init=sir.init.json num_threads=4

It seems to vary a bit between runs, but when run repeatedly the most common outcome is that two chains run/print their progress, until completion, and then the process hangs. I have recreated this on two separate linux systems.

strace reports a lot of using of sched_yield from tbb, but otherwise there is nothing happening. gdb seems to implicate set_zero_all_adjoints in a backtrace, but not in any way I can track down.

Expected Output

Expect all four chains to run simultaneously and the program to not hang.

Current Version:

v4.7.0

[WardBrian]

Here's what gdb reports when chain 1 and 4 are stuck:

Details

(gdb) info threads
  Id   Target Id                               Frame 
* 1    Thread 0x7f54f8ea9740 (LWP 13426) "sir" 0x000055c3a6af4708 in cvStep ()
  2    Thread 0x7f54f81fb640 (LWP 13427) "sir" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7f54f7dfa640 (LWP 13428) "sir" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x7f54f79f9640 (LWP 13429) "sir" 0x000055c3a694c420 in operator new(unsigned long)@plt ()
(gdb) bt
#0  0x000055c3a6af4708 in cvStep ()
#1  0x000055c3a6afca60 in CVode ()
#2  0x000055c3a6984d9e in stan::math::cvodes_integrator<1, sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, int const&>::operator() (this=this@entry=0x7ffd46151200) at stan/lib/stan_math/stan/math/rev/functor/cvodes_integrator.hpp:309
#3  0x000055c3a696a52b in _ZZN4stan4math18ode_adams_tol_implIN19sir_model_namespace23sir_variadic2_functor__EN5Eigen3MapINS4_6MatrixIdLin1ELi1ELi0ELin1ELi1EEELi0ENS4_6StrideILi0ELi0EEEEEddJNS0_9var_valueIdvEESC_iELPv0EEESt6vectorINS6_INS_11return_typeIJT0_T1_T2_DpT3_EE4typeELin1ELi1ELi0ELin1ELi1EEESaISN_EEPKcRKT_RKSG_RKSH_RKSE_ISI_SaISI_EEddlPSoDpRKSJ_ENKUlDpRKT_E_clIJSC_SC_iEEEDaS1A_ (__closure=<optimized out>)
    at stan/lib/stan_math/stan/math/rev/functor/ode_adams.hpp:64
#4  _ZN4stan4math8internal10apply_implIZNS0_18ode_adams_tol_implIN19sir_model_namespace23sir_variadic2_functor__EN5Eigen3MapINS6_6MatrixIdLin1ELi1ELi0ELin1ELi1EEELi0ENS6_6StrideILi0ELi0EEEEEddJNS0_9var_valueIdvEESE_iELPv0EEESt6vectorINS8_INS_11return_typeIJT0_T1_T2_DpT3_EE4typeELin1ELi1ELi0ELin1ELi1EEESaISP_EEPKcRKT_RKSI_RKSJ_RKSG_ISK_SaISK_EEddlPSoDpRKSL_EUlDpRKT_E_RKSt5tupleIJSE_SE_iEEJLm0ELm1ELm2EEEEDcOSU_OSI_St16integer_sequenceImJXspT1_EEE (i=..., t=..., f=...) at stan/lib/stan_math/stan/math/prim/functor/apply.hpp:26
#5  _ZN4stan4math5applyIZNS0_18ode_adams_tol_implIN19sir_model_namespace23sir_variadic2_functor__EN5Eigen3MapINS5_6MatrixIdLin1ELi1ELi0ELin1ELi1EEELi0ENS5_6StrideILi0ELi0EEEEEddJNS0_9var_valueIdvEESD_iELPv0EEESt6vectorINS7_INS_11return_typeIJT0_T1_T2_DpT3_EE4typeELin1ELi1ELi0ELin1ELi1EEESaISO_EEPKcRKT_RKSH_RKSI_RKSF_ISJ_SaISJ_EEddlPSoDpRKSK_EUlDpRKT_E_RKSt5tupleIJSD_SD_iEEEEDcOST_OSH_ (t=..., f=...)
    at stan/lib/stan_math/stan/math/prim/functor/apply.hpp:47
#6  stan::math::ode_adams_tol_impl<sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, int, (void*)0> (msgs=<optimized out>, max_num_steps=<optimized out>, absolute_tolerance=<optimized out>, relative_tolerance=<optimized out>, ts=..., t0=<optimized out>, 
    y0=..., f=..., function_name=<optimized out>) at stan/lib/stan_math/stan/math/rev/functor/ode_adams.hpp:66
#7  stan::math::ode_adams<sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, int, (void*)0> (f=..., y0=..., t0=<optimized out>, ts=..., msgs=<optimized out>) at stan/lib/stan_math/stan/math/rev/functor/ode_adams.hpp:157
 

#8  0x000055c3a698abc2 in sir_model_namespace::sir_model::log_prob_impl<true, true, Eigen::Matrix<stan::math::var_value<double, void>, -1, 1, 0, -1, 1>, Eigen::Matrix<int, -1, 1, 0, -1, 1>, (void*)0, (void*)0, (void*)0> (this=0x55c3a85446f0, params_r__=..., params_i__=..., pstream__=0x7ffd46151600) at sir.hpp:327
#9  0x000055c3a698b034 in sir_model_namespace::sir_model::log_prob<true, true, stan::math::var_value<double, void> > (pstream=<optimized out>, params_r=..., this=<optimized out>)
    at stan/lib/stan_math/lib/eigen_3.4.0/Eigen/src/Core/PlainObjectBase.h:968
#10 stan::model::model_base_crtp<sir_model_namespace::sir_model>::log_prob_propto_jacobian (this=<optimized out>, theta=..., msgs=<optimized out>) at stan/src/stan/model/model_base_crtp.hpp:132
#11 0x000055c3a6a661b8 in stan::model::model_base::log_prob<true, true, stan::math::var_value<double, void> > (msgs=<optimized out>, params_r=..., this=<optimized out>) at stan/src/stan/model/model_base.hpp:326
#12 stan::model::model_functional<stan::model::model_base>::operator()<stan::math::var_value<double, void> > (x=..., this=0x7ffd461515a0) at stan/src/stan/model/model_functional.hpp:21
#13 stan::math::gradient<stan::model::model_functional<stan::model::model_base> > (f=..., x=..., fx=@0x55c3a854fd00: 7409.7008271914856, grad_fx=...) at stan/lib/stan_math/stan/math/rev/functor/gradient.hpp:51
#14 0x000055c3a6a66703 in stan::model::gradient<stan::model::model_base> (model=..., x=..., f=@0x55c3a854fd00: 7409.7008271914856, grad_f=..., logger=...) at stan/src/stan/model/gradient.hpp:27
#15 0x000055c3a6a6d977 in stan::mcmc::base_hamiltonian<stan::model::model_base, stan::mcmc::diag_e_point, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::update_potential_gradient (this=this@entry=0x55c3a854fd20, z=..., logger=...)
    at stan/src/stan/mcmc/hmc/hamiltonians/base_hamiltonian.hpp:63
#16 0x000055c3a6a6e031 in stan::mcmc::expl_leapfrog<stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > >::update_q (this=<optimized out>, z=..., hamiltonian=warning: RTTI symbol not found for class 'stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >'
..., epsilon=<optimized out>, logger=...)
    at stan/src/stan/mcmc/hmc/integrators/expl_leapfrog.hpp:25
#17 0x000055c3a69bb16f in stan::mcmc::base_leapfrog<stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > >::evolve (logger=..., epsilon=0.125, hamiltonian=warning: RTTI symbol not found for class 'stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >'
..., z=..., this=0x55c3a854fd18)
    at stan/src/stan/mcmc/hmc/integrators/base_leapfrog.hpp:20
#18 stan::mcmc::base_hmc<stan::model::model_base, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::init_stepsize (logger=..., this=0x55c3a854fcc0) at stan/src/stan/mcmc/hmc/base_hmc.hpp:125
#19 stan::services::util::run_adaptive_sampler<stan::mcmc::adapt_diag_e_nuts<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >, stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > (sampler=warning: RTTI symbol not found for class 'stan::mcmc::adapt_diag_e_nuts<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >'
..., model=..., num_warmup=1000, num_samples=1000, num_thin=1, refresh=100, 
    save_warmup=false, rng=..., interrupt=..., logger=..., sample_writer=..., diagnostic_writer=..., metric_writer=..., chain_id=1, num_chains=4, cont_vector=..., cont_vector=...)
    at stan/src/stan/services/util/run_adaptive_sampler.hpp:63
#20 0x000055c3a6a6f6f0 in stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}::operator()(tbb::blocked_range<unsigned long> const&) const (r=..., __closure=<optimized out>) at stan/src/stan/services/sample/hmc_nuts_diag_e_adapt.hpp:391
#21 tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io--Type <RET> for more, q to quit, c to continue without paging--
::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>::run_body(tbb::blocked_range<unsigned long>&) (r=..., this=<optimized out>) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/parallel_for.h:115
#22 tbb::interface9::internal::simple_partition_type::execute<tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>, tbb::blocked_range<unsigned long> >(stan::model::model_base&, tbb::blocked_range<unsigned long>&) (range=..., start=warning: RTTI symbol not found for class 'tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>'

..., this=0x7f54f87e3cd0) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/partitioner.h:510
#23 tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>::execute() (this=0x7f54f87e3c40) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/parallel_for.h:142
#24 0x00007f54f8f13755 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f54f87ee600, context_guard=..., t=0x7f54f87e3c40, 
    isolation=isolation@entry=0) at ../tbb_2020.3/src/tbb/custom_scheduler.h:474
#25 0x00007f54f8f13aa2 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f54f87ee600, parent=..., child=<optimized out>)
    at ../tbb_2020.3/src/tbb/custom_scheduler.h:636
#26 0x00007f54f8f113a7 in tbb::internal::generic_scheduler::local_spawn_root_and_wait (this=0x7f54f87ee600, first=0x7f54f87e3c40, next=@0x7f54f87e3c38: 0x0) at ../tbb_2020.3/src/tbb/scheduler.cpp:738
#27 0x000055c3a6a72ebe in tbb::task::spawn_root_and_wait (root=...) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/task.h:809
#28 tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}--Type <RET> for more, q to quit, c to continue without paging--
, tbb::simple_partitioner const>::run(tbb::blocked_range<unsigned long> const&, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1} const&, tbb::simple_partitioner const&) (partitioner=..., body=..., range=...) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/parallel_for.h:95
#29 tbb::parallel_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}>(tbb::blocked_range<unsigned long> const&, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1} const&, tbb::simple_partitioner const&) (
    partitioner=..., body=..., range=...) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/parallel_for.h:208
#30 stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > (model=..., num_chains=num_chains@entry=4, init=std::vector of length 4, capacity 4 = {...}, 
    init_inv_metric=std::vector of length 4, capacity 4 = {...}, random_seed=random_seed@entry=4184322767, init_chain_id=init_chain_id@entry=1, init_radius=init_radius@entry=2, num_warmup=1000, 
    num_samples=1000, num_thin=1, save_warmup=false, refresh=100, stepsize=stepsize@entry=1, stepsize_jitter=stepsize_jitter@entry=0, max_depth=10, delta=delta@entry=0.80000000000000004, 
    gamma=gamma@entry=0.050000000000000003, kappa=kappa@entry=0.75, t0=t0@entry=10, init_buffer=75, term_buffer=50, window=25, interrupt=..., logger=..., 
    init_writer=std::vector of length 4, capacity 4 = {...}, sample_writer=std::vector of length 4, capacity 4 = {...}, diagnostic_writer=std::vector of length 4, capacity 4 = {...}, 
    metric_writer=std::vector of length 4, capacity 4 = {...}) at stan/src/stan/services/sample/hmc_nuts_diag_e_adapt.hpp:384
#31 0x000055c3a6a73c04 in stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > (model=..., num_chains=num_chains@entry=4, init=std::vector of length 4, capacity 4 = {...}, random_seed=random_seed@entry=4184322767, init_chain_id=init_chain_id@entry=1, 
    init_radius=init_radius@entry=2, num_warmup=num_warmup@entry=1000, num_samples=1000, num_thin=1, save_warmup=false, refresh=100, stepsize=stepsize@entry=1, stepsize_jitter=stepsize_jitter@entry=0, 
    max_depth=10, delta=delta@entry=0.80000000000000004, gamma=gamma@entry=0.050000000000000003, kappa=kappa@entry=0.75, t0=t0@entry=10, init_buffer=75, term_buffer=50, window=25, interrupt=..., logger=..., 
    init_writer=std::vector of length 4, capacity 4 = {...}, sample_writer=std::vector of length 4, capacity 4 = {...}, diagnostic_writer=std::vector of length 4, capacity 4 = {...}, 
    metric_writer=std::vector of length 4, capacity 4 = {...}) at stan/src/stan/services/sample/hmc_nuts_diag_e_adapt.hpp:560
#32 0x000055c3a69d3dd0 in cmdstan::command (argc=<optimized out>, argv=<optimized out>) at src/cmdstan/command.hpp:654
#33 0x000055c3a696727a in main (argc=<optimized out>, argv=<optimized out>) at src/cmdstan/main.cpp:6
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f54f79f9640 (LWP 13429))]
#0  0x000055c3a694c420 in operator new(unsigned long)@plt ()
(gdb) bt
#0  0x000055c3a694c420 in operator new(unsigned long)@plt ()
#1  0x000055c3a69959ab in __gnu_cxx::new_allocator<double>::allocate (this=0x7f54f79f7c90, __n=9) at /usr/include/c++/11/ext/new_allocator.h:103
#2  std::allocator_traits<std::allocator<double> >::allocate (__n=9, __a=...) at /usr/include/c++/11/bits/alloc_traits.h:464
#3  std::_Vector_base<double, std::allocator<double> >::_M_allocate (__n=9, this=0x7f54f79f7c90) at /usr/include/c++/11/bits/stl_vector.h:346
#4  std::_Vector_base<double, std::allocator<double> >::_M_create_storage (__n=9, this=0x7f54f79f7c90) at /usr/include/c++/11/bits/stl_vector.h:361
#5  std::_Vector_base<double, std::allocator<double> >::_Vector_base (__a=..., __n=9, this=0x7f54f79f7c90) at /usr/include/c++/11/bits/stl_vector.h:305
#6  std::vector<double, std::allocator<double> >::vector (__a=..., __n=9, this=0x7f54f79f7c90) at /usr/include/c++/11/bits/stl_vector.h:511
#7  stan::math::cvodes_integrator<1, sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, int const&>::rhs_sens (ySdot=0x7f54ec0167b0, yS=0x7f54ec014e60, y=0x7f54ec0124e0, t=5.7446523995551716e-24, this=0x7f54f79f8180)
    at stan/lib/stan_math/stan/math/rev/functor/cvodes_integrator.hpp:149
#8  stan::math::cvodes_integrator<1, sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, int const&>::cv_rhs_sens (Ns=<optimized out>, t=5.7446523995551716e-24, y=<optimized out>, ydot=<optimized out>, yS=0x7f54ec014e60, ySdot=0x7f54ec0167b0, 
    user_data=0x7f54f79f8180, tmp1=0x7f54ec01c4c0, tmp2=0x7f54ec012cb0) at stan/lib/stan_math/stan/math/rev/functor/cvodes_integrator.hpp:83
#9  0x000055c3a6af35e3 in cvSensRhsWrapper ()
#10 0x000055c3a6b0afa6 in cvNlsResidualSensStg ()
#11 0x000055c3a6b101ff in SUNNonlinSolSolve_Newton ()
#12 0x000055c3a6af461c in cvStep ()
#13 0x000055c3a6afca60 in CVode ()
#14 0x000055c3a6984d9e in stan::math::cvodes_integrator<1, sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, int const&>::operator() (this=this@entry=0x7f54f79f8180) at stan/lib/stan_math/stan/math/rev/functor/cvodes_integrator.hpp:309
#15 0x000055c3a696a52b in _ZZN4stan4math18ode_adams_tol_implIN19sir_model_namespace23sir_variadic2_functor__EN5Eigen3MapINS4_6MatrixIdLin1ELi1ELi0ELin1ELi1EEELi0ENS4_6StrideILi0ELi0EEEEEddJNS0_9var_valueIdvEESC_iELPv0EEESt6vectorINS6_INS_11return_typeIJT0_T1_T2_DpT3_EE4typeELin1ELi1ELi0ELin1ELi1EEESaISN_EEPKcRKT_RKSG_RKSH_RKSE_ISI_SaISI_EEddlPSoDpRKSJ_ENKUlDpRKT_E_clIJSC_SC_iEEEDaS1A_ (__closure=<optimized out>)
    at stan/lib/stan_math/stan/math/rev/functor/ode_adams.hpp:64
#16 _ZN4stan4math8internal10apply_implIZNS0_18ode_adams_tol_implIN19sir_model_namespace23sir_variadic2_functor__EN5Eigen3MapINS6_6MatrixIdLin1ELi1ELi0ELin1ELi1EEELi0ENS6_6StrideILi0ELi0EEEEEddJNS0_9var_valueIdvEESE_iELPv0EEESt6vectorINS8_INS_11return_typeIJT0_T1_T2_DpT3_EE4typeELin1ELi1ELi0ELin1ELi1EEESaISP_EEPKcRKT_RKSI_RKSJ_RKSG_ISK_SaISK_EEddlPSoDpRKSL_EUlDpRKT_E_RKSt5tupleIJSE_SE_iEEJLm0ELm1ELm2EEEEDcOSU_OSI_St16integer_sequenceImJXspT1_EEE (i=..., t=..., f=...) at stan/lib/stan_math/stan/math/prim/functor/apply.hpp:26
#17 _ZN4stan4math5applyIZNS0_18ode_adams_tol_implIN19sir_model_namespace23sir_variadic2_functor__EN5Eigen3MapINS5_6MatrixIdLin1ELi1ELi0ELin1ELi1EEELi0ENS5_6StrideILi0ELi0EEEEEddJNS0_9var_valueIdvEESD_iELPv0EEESt6vectorINS7_INS_11return_typeIJT0_T1_T2_DpT3_EE4typeELin1ELi1ELi0ELin1ELi1EEESaISO_EEPKcRKT_RKSH_RKSI_RKSF_ISJ_SaISJ_EEddlPSoDpRKSK_EUlDpRKT_E_RKSt5tupleIJSD_SD_iEEEEDcOST_OSH_ (t=..., f=...)
    at stan/lib/stan_math/stan/math/prim/functor/apply.hpp:47
#18 stan::math::ode_adams_tol_impl<sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, int, (void*)0> (msgs=<optimized out>, max_num_steps=<optimized out>, absolute_tolerance=<optimized out>, relative_tolerance=<optimized out>, ts=..., t0=<optimized out>, 
    y0=..., f=..., function_name=<optimized out>) at stan/lib/stan_math/stan/math/rev/functor/ode_adams.hpp:66
#19 stan::math::ode_adams<sir_model_namespace::sir_variadic2_functor__, Eigen::Map<Eigen::Matrix<double, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, double, double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, int, (void*)0> (f=..., y0=..., t0=<optimized out>, ts=..., msgs=<optimized out>) at stan/lib/stan_math/stan/math/rev/functor/ode_adams.hpp:157
#20 0x000055c3a698abc2 in sir_model_namespace::sir_model::log_prob_impl<true, true, Eigen::Matrix<stan::math::var_value<double, void>, -1, 1, 0, -1, 1>, Eigen::Matrix<int, -1, 1, 0, -1, 1>, (void*)0, (void*)0, (void*)0> (this=0x55c3a85446f0, params_r__=..., params_i__=..., pstream__=0x7f54f79f8580) at sir.hpp:327
#21 0x000055c3a698b034 in sir_model_namespace::sir_model::log_prob<true, true, stan::math::var_value<double, void> > (pstream=<optimized out>, params_r=..., this=<optimized out>)
    at stan/lib/stan_math/lib/eigen_3.4.0/Eigen/src/Core/PlainObjectBase.h:968
#22 stan::model::model_base_crtp<sir_model_namespace::sir_model>::log_prob_propto_jacobian (this=<optimized out>, theta=..., msgs=<optimized out>) at stan/src/stan/model/model_base_crtp.hpp:132
#23 0x000055c3a6a661b8 in stan::model::model_base::log_prob<true, true, stan::math::var_value<double, void> > (msgs=<optimized out>, params_r=..., this=<optimized out>) at stan/src/stan/model/model_base.hpp:326
#24 stan::model::model_functional<stan::model::model_base>::operator()<stan::math::var_value<double, void> > (x=..., this=0x7f54f79f8520) at stan/src/stan/model/model_functional.hpp:21
#25 stan::math::gradient<stan::model::model_functional<stan::model::model_base> > (f=..., x=..., fx=@0x55c3a8550198: 7409.7008271914856, grad_fx=...) at stan/lib/stan_math/stan/math/rev/functor/gradient.hpp:51
#26 0x000055c3a6a66703 in stan::model::gradient<stan::model::model_base> (model=..., x=..., f=@0x55c3a8550198: 7409.7008271914856, grad_f=..., logger=...) at stan/src/stan/model/gradient.hpp:27
#27 0x000055c3a6a6d977 in stan::mcmc::base_hamiltonian<stan::model::model_base, stan::mcmc::diag_e_point, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::update_potential_gradient (this=this@entry=0x55c3a85501b8, z=..., logger=...)
    at stan/src/stan/mcmc/hmc/hamiltonians/base_hamiltonian.hpp:63
#28 0x000055c3a6a6e031 in stan::mcmc::expl_leapfrog<stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > >::update_q (this=<optimized out>, z=..., hamiltonian=warning: RTTI symbol not found for class 'stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >'
..., epsilon=<optimized out>, logger=...)
    at stan/src/stan/mcmc/hmc/integrators/expl_leapfrog.hpp:25
#29 0x000055c3a69bb16f in stan::mcmc::base_leapfrog<stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > >::evolve (logger=..., epsilon=0.125, hamiltonian=warning: RTTI symbol not found for class 'stan::mcmc::diag_e_metric<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >'
..., z=..., this=0x55c3a85501b0)
    at stan/src/stan/mcmc/hmc/integrators/base_leapfrog.hpp:20
#30 stan::mcmc::base_hmc<stan::model::model_base, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::init_stepsize (logger=..., this=0x55c3a8550158) at stan/src/stan/mcmc/hmc/base_hmc.hpp:125
#31 stan::services::util::run_adaptive_sampler<stan::mcmc::adapt_diag_e_nuts<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >, stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > (sampler=warning: RTTI symbol not found for class 'stan::mcmc::adapt_diag_e_nuts<stan::model::model_base, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >'
..., model=..., num_warmup=1000, num_samples=1000, num_thin=1, refresh=100, 
--Type <RET> for more, q to quit, c to continue without paging--
    save_warmup=false, rng=..., interrupt=..., logger=..., sample_writer=..., diagnostic_writer=..., metric_writer=..., chain_id=4, num_chains=4, cont_vector=..., cont_vector=...)
    at stan/src/stan/services/util/run_adaptive_sampler.hpp:63
#32 0x000055c3a6a6f6f0 in stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}::operator()(tbb::blocked_range<unsigned long> const&) const (r=..., __closure=<optimized out>) at stan/src/stan/services/sample/hmc_nuts_diag_e_adapt.hpp:391
#33 tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>::run_body(tbb::blocked_range<unsigned long>&) (r=..., this=<optimized out>) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/parallel_for.h:115
#34 tbb::interface9::internal::simple_partition_type::execute<tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>, tbb::blocked_range<unsigned long> >(stan::model::model_base&, tbb::blocked_range<unsigned long>&) (range=..., start=warning: RTTI symbol not found for class 'tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>'

..., this=0x7f54f87d7dd0) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/partitioner.h:510
#35 tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, stan::services::sample::hmc_nuts_diag_e_adapt<stan::model::model_base, std::shared_ptr<stan::io::var_context>, std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, stan::callbacks::writer, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > >(stan::model::model_base&, unsigned long, std::vector<std::shared_ptr<stan::io::var_context>, std::allocator<std::shared_ptr<stan::io::var_context> > > const&, std::vector<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> >, std::allocator<std::unique_ptr<stan::io::dump, std::default_delete<stan::io::dump> > > > const&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, std::vector<stan::callbacks::writer, std::allocator<stan::callbacks::writer> >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::unique_stream_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&, std::vector<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > >, std::allocator<stan::callbacks::json_writer<std::basic_ofstream<char, std::char_traits<char> >, std::default_delete<std::basic_ofstream<char, std::char_traits<char> > > > > >&)::{lambda(tbb::blocked_range<unsigned long> const&)#1}, tbb::simple_partitioner const>::execute() (this=0x7f54f87d7d40) at stan/lib/stan_math/lib/tbb_2020.3/include/tbb/parallel_for.h:142
#36 0x00007f54f8f13755 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f54f87c3e00, context_guard=..., t=0x7f54f87d7d40, 
    isolation=isolation@entry=0) at ../tbb_2020.3/src/tbb/custom_scheduler.h:474
--Type <RET> for more, q to quit, c to continue without paging--
#37 0x00007f54f8f13aa2 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f54f87c3e00, parent=..., child=<optimized out>)
    at ../tbb_2020.3/src/tbb/custom_scheduler.h:636
#38 0x00007f54f8f0d20c in tbb::internal::arena::process (this=0x7f54f87e7480, s=...) at ../tbb_2020.3/src/tbb/arena.cpp:196
#39 0x00007f54f8f0b8d8 in tbb::internal::market::process (this=0x7f54f87f3580, j=...) at ../tbb_2020.3/src/tbb/market.cpp:667
#40 0x00007f54f8f07c40 in tbb::internal::rml::private_worker::run (this=0x7f54f8507000) at ../tbb_2020.3/src/tbb/private_server.cpp:266
#41 0x00007f54f8f07ead in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../tbb_2020.3/src/tbb/private_server.cpp:219
#42 0x00007f54f8894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#43 0x00007f54f8926a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[syclik]

I'll try to reproduce when I have some free time.

[syclik]

I'm on develop for cmdstan and I haven't seen it yet (maybe a dozen different seeds). Can you throw out a seed that causes it to fail if you have one?

[WardBrian]

This was develop on both machines I was using, independent of seed. Both were running Linux however