"Cannot run an executor inside another executor" - in tests

[humb1t]

1.6 version of async-std crashed our tests that works fine on prev version.

According to the stack trace asycn-std fails on stream.flush().await.map_err(|e| e.to_string())?; usage with a strange error message about executor inside another executor while we use #[async-std] attribute on tests and do not create executors here.

Maybe recent changes in sync module (for example we found a need to use Result instead of Optionin channel::Receiver) and our usage of it were the cause.

thread 'tests::single_threaded_async_stateful' panicked at 'cannot run an executor inside another executor', <::std::macros::panic macros>:2:4
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/6github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/6github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1063
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:204
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:224
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:470
  11: std::panicking::begin_panic
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/panicking.rs:397
  12: smol::work_stealing::Worker::enter
             at /home/humb1t/workspace/soramitsu/iroha/<::std::macros::panic macros>:2
  13: smol::run::run::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:113
  14: smol::run::run
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:115
  15: async_std::task::builder::Builder::blocking::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/builder.rs:154
  16: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/task_locals_wrapper.rs:60
  17: std::thread::local::LocalKey<T>::try_with
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/thread/local.rs:262
  18: std::thread::local::LocalKey<T>::with
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/thread/local.rs:239
  19: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/task_locals_wrapper.rs:55
  20: async_std::task::builder::Builder::blocking
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/builder.rs:154
  21: async_std::task::block_on::block_on
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/block_on.rs:33
  22: <iroha_network::mock::RequestStream as futures_io::if_std::AsyncWrite>::poll_flush
             at iroha_network/src/mock.rs:78
  23: <alloc::boxed::Box<T> as futures_io::if_std::AsyncWrite>::poll_flush
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/futures-io-0.3.5/src/lib.rs:413
  24: <async_std::io::write::flush::FlushFuture<T> as core::future::future::Future>::poll
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/io/write/flush.rs:17
  25: std::future::poll_with_tls_context
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:102
  26: iroha_network::mock::Network::handle_message_async::{{closure}}
             at iroha_network/src/mock.rs:175
  27: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:44
  28: std::future::poll_with_tls_context
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:102
  29: iroha_network::tests::single_threaded_async_stateful::{{closure}}::handle_connection::{{closure}}
             at iroha_network/src/lib.rs:292
  30: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:44
  31: std::future::poll_with_tls_context
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:102
  32: iroha_network::mock::Network::listen::{{closure}}
             at iroha_network/src/mock.rs:144
  33: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:44
  34: std::future::poll_with_tls_context
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:102
  35: iroha_network::tests::single_threaded_async_stateful::{{closure}}::{{closure}}
             at iroha_network/src/lib.rs:296
  36: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/future.rs:44
  37: <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/builder.rs:174
  38: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/task_locals_wrapper.rs:60
  39: std::thread::local::LocalKey<T>::try_with
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/thread/local.rs:262
  40: std::thread::local::LocalKey<T>::with
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/thread/local.rs:239
  41: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/task_locals_wrapper.rs:55
  42: <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/builder.rs:172
  43: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libcore/future/future.rs:118
  44: async_task::raw::RawTask<F,R,S,T>::run
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-task-3.0.0/src/raw.rs:502
  45: async_task::task::Task<T>::run
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-task-3.0.0/src/task.rs:265
  46: smol::work_stealing::Worker::execute::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/work_stealing.rs:176
  47: scoped_tls_hkt::ScopedKey<T>::set
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/scoped-tls-hkt-0.1.2/src/lib.rs:488
  48: smol::throttle::setup
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/throttle.rs:31
  49: smol::work_stealing::Worker::execute
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/work_stealing.rs:176
  50: smol::run::run::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:151
  51: core::ops::function::FnOnce::call_once
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libcore/ops/function.rs:232
  52: smol::context::enter
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/context.rs:8
  53: smol::run::run::{{closure}}::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:112
  54: scoped_tls_hkt::ScopedKey<T>::set
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/scoped-tls-hkt-0.1.2/src/lib.rs:488
  55: smol::thread_local::ThreadLocalExecutor::enter
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/thread_local.rs:57
  56: smol::run::run::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:112
  57: smol::run::run::{{closure}}::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:113
  58: smol::work_stealing::WORKER::<impl smol::work_stealing::WORKER>::set
             at /home/humb1t/workspace/soramitsu/iroha/<::scoped_tls_hkt::scoped_thread_local macros>:40
  59: smol::work_stealing::Worker::enter
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/work_stealing.rs:149
  60: smol::run::run::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:113
  61: smol::run::run
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/smol-0.1.10/src/run.rs:115
  62: async_std::task::builder::Builder::blocking::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/builder.rs:154
  63: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::{{closure}}
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/task_locals_wrapper.rs:60
  64: std::thread::local::LocalKey<T>::try_with
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/thread/local.rs:262
  65: std::thread::local::LocalKey<T>::with
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/thread/local.rs:239
  66: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/task_locals_wrapper.rs:55
  67: async_std::task::builder::Builder::blocking
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/builder.rs:154
  68: async_std::task::block_on::block_on
             at /home/humb1t/.cargo/registry/src/6github.com-1ecc6299db9ec823/async-std-1.6.0/src/task/block_on.rs:33
  69: iroha_network::tests::single_threaded_async_stateful
             at iroha_network/src/lib.rs:276
  70: iroha_network::tests::single_threaded_async_stateful::{{closure}}
             at iroha_network/src/lib.rs:276
  71: core::ops::function::FnOnce::call_once
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libcore/ops/function.rs:232
  72: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/liballoc/boxed.rs:1017
  73: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:86
  74: std::panicking::try
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/panicking.rs:281
  75: std::panic::catch_unwind
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/panic.rs:394
  76: test::run_test_in_process
             at src/libtest/lib.rs:542
  77: test::run_test::run_test_inner::{{closure}}
             at src/libtest/lib.rs:451
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test tests::single_threaded_async_stateful ... FAILED

[Licenser]

I think this is the result of calling task::block_on() in another task::block_on() here is a example of this failing: https://github.com/Licenser/nested-executors

This seems to be a very serious limitation as it makes it virtually impossible to use it in any place other than the mainline of a binary. For example if a library exposes asynchronous interface and uses any asynchronous code underneath it blocks on, this library can't be used in any other asynchronous function any more.

Unless you know the source of it, the error is also not exactly pointing users to the right place as it requires the knowledge that block_on spawns an executor.

[Licenser]

This is related to #760

[gmg137]

The a-chat example program runs with the same error.

[yoshuawuyts]

The problem here is that nested calls to block_on always had the potential to deadlock and once they did it wasn't really possible to debug. With the latest release we're now asserting that won't ever happen and throwing a runtime error when we detect it might.

This is unfortunate, but also solves a pretty serious bug. For the exact details or inquiries it's probably best to raise it in smol as that's where we source the block_on impl from now.

---

The a-chat example program runs with the same error.

Oh no! -- we should patch this!

[humb1t]

@yoshuawuyts ty, will update deps and try to find nested calls.