CUDA PC2 on Tesla V100S panic #1522

kenshyx · 2021-10-12T15:06:06Z

kenshyx
Oct 12, 2021

Checklist

This is not a security-related bug/issue. If it is, please follow please follow the security policy.
This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
This is not a new feature request. If it is, please file a feature request instead.
This is not an enhancement request. If it is, please file a improvement suggestion instead.
I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
I did not make any code changes to lotus.

Lotus component

Lotus Version

lotus-worker version 1.13.0-rc2+mainnet+git.530011fd0

Describe the Bug

I've compiled lotus-worker with FIL_PROOFS_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70" to target Tesla V100. After this I'm unable to do PC2 because of panic error :)

Logging Information

Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.027 INFO filcrypto::proofs::api > seal_pre_commit_phase2: start
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:start
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:finish
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:start
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase2
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > Building trees [295536 descriptors max available]
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree c using the GPU
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > Building column hashes
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.048 INFO neptune::proteus::program > Using kernel on CUDA.
Oct 12 16:41:40  lotus-worker[73229]: thread 'worker-thread-2' panicked at 'failed to create tree gpu batcher: GpuError("Cuda(NoBinaryForGpu)")', .cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-10.0.0/src/stacked/vanilla/proof.rs:596:30
Oct 12 16:41:40  lotus-worker[73229]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Oct 12 16:41:40  lotus-worker[73229]: thread '<unnamed>' panicked at 'failed to receive base_data, tree_data for tree_c: RecvError', .cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-10.0.0/src/stacked/vanilla/proof.rs:652:26
Oct 12 16:41:40  lotus-worker[73229]: thread 'worker-thread-1' panicked at 'failed to send columns: SendError { .. }', .cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-10.0.0/src/stacked/vanilla/proof.rs:588:34
Oct 12 16:41:40  lotus-worker[73229]: thread '<unnamed>' panicked at 'Worker Pool was poisoned', .cargo/registry/src/github.com-1ecc6299db9ec823/yastl-0.1.2/src/wait.rs:50:13
Oct 12 16:41:40  lotus-worker[73229]: stack backtrace:
Oct 12 16:41:40  lotus-worker[73229]:    0:          0x20bbbf0 - std::backtrace_rs::backtrace::libunwind::trace::hdcf4f90f85129e83
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
Oct 12 16:41:40  lotus-worker[73229]:    1:          0x20bbbf0 - std::backtrace_rs::backtrace::trace_unsynchronized::h2669e30cb82f6732
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Oct 12 16:41:40  lotus-worker[73229]:    2:          0x20bbbf0 - std::sys_common::backtrace::_print_fmt::hfbda19e17f6db318
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:67:5
Oct 12 16:41:40  lotus-worker[73229]:    3:          0x20bbbf0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1a8751bf59281272
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:46:22
Oct 12 16:41:40  lotus-worker[73229]:    4:          0x212be5f - core::fmt::write::h7aa6cd0067dca82a
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/core/src/fmt/mod.rs:1094:17
Oct 12 16:41:40  lotus-worker[73229]:    5:          0x20b0385 - std::io::Write::write_fmt::hd7dd3a1df9b6befb
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/io/mod.rs:1580:15
Oct 12 16:41:40  lotus-worker[73229]:    6:          0x20bf94b - std::sys_common::backtrace::_print::h551e9ec8a9fa8106
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:49:5
Oct 12 16:41:40  lotus-worker[73229]:    7:          0x20bf94b - std::sys_common::backtrace::print::ha4b1c5e95fa040b3
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:36:9
Oct 12 16:41:40  lotus-worker[73229]:    8:          0x20bf94b - std::panicking::default_hook::{{closure}}::h0b34c9ab7fb9f857
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:208:50
Oct 12 16:41:40  lotus-worker[73229]:    9:          0x20bf42d - std::panicking::default_hook::h3067e8318decd17a
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:225:9
Oct 12 16:41:40  lotus-worker[73229]:   10:          0x20c003d - std::panicking::rust_panic_with_hook::h81b8facc50f34daa
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:591:17
Oct 12 16:41:40  lotus-worker[73229]:   11:          0x20231b6 - std::panicking::begin_panic::{{closure}}::h17aa203bdf3b53f4
Oct 12 16:41:40  lotus-worker[73229]:   12:          0x202317e - std::sys_common::backtrace::__rust_end_short_backtrace::h992fb3327e62bc2a
Oct 12 16:41:40  lotus-worker[73229]:   13:           0x5029ac - std::panicking::begin_panic::hf1f74302d08d14e1
Oct 12 16:41:40  lotus-worker[73229]:   14:          0x2025373 - yastl::wait::WaitGroup::join::h03280543330f3f44
Oct 12 16:41:40  lotus-worker[73229]:   15:          0x1b6d8fc - yastl::scope::Scope::zoom::h2dc3f87d72d5d2cf
Oct 12 16:41:40  lotus-worker[73229]:   16:          0x1a98d66 - yastl::Pool::scoped::hc33c0a9214f792c8
Oct 12 16:41:40  lotus-worker[73229]:   17:          0x19b4471 - storage_proofs_core::measurements::measure_op::h1260afd960ab575d
Oct 12 16:41:40  lotus-worker[73229]:   18:          0x2212823 - storage_proofs_porep::stacked::vanilla::proof::StackedDrg<Tree,G>::replicate_phase2::h125e70534093daef
Oct 12 16:41:40  lotus-worker[73229]:   19:          0x217218b - filecoin_proofs::api::seal::seal_pre_commit_phase2::ha3d06c2bb552dfde
Oct 12 16:41:40  lotus-worker[73229]:   20:          0x21eb349 - filecoin_proofs_api::seal::seal_pre_commit_phase2_inner::he260fc02a6d000f5
Oct 12 16:41:40  lotus-worker[73229]:   21:          0x21ae3e8 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h979bad57e9431d37
Oct 12 16:41:40  lotus-worker[73229]:   22:          0x2160d3e - ffi_toolkit::catch_panic_response::h8d0bb7f164831c2b
Oct 12 16:41:40  lotus-worker[73229]:   23:          0x1b27c8a - fil_seal_pre_commit_phase2
Oct 12 16:41:40  lotus-worker[73229]:   24:          0x18f4597 - _cgo_c8a539579dfc_Cfunc_fil_seal_pre_commit_phase2
Oct 12 16:41:40  lotus-worker[73229]:                                at /tmp/go-build/cgo-gcc-prolog:1232:11
Oct 12 16:41:40  lotus-worker[73229]:   25:           0x58bb30 - runtime.asmcgocall
Oct 12 16:41:40  lotus-worker[73229]:                                at /usr/local/go/src/runtime/asm_amd64.s:667
Oct 12 16:41:40  lotus-worker[73229]: thread panicked while panicking. aborting.
Oct 12 16:41:40  lotus-worker[73229]: SIGILL: illegal instruction
Oct 12 16:41:40  lotus-worker[73229]: PC=0x20bffbc m=27 sigcode=2
Oct 12 16:41:40  lotus-worker[73229]: instruction bytes: 0xf 0xb 0xf 0xb 0x4c 0x89 0x6c 0x24 0x10 0x4c 0x89 0x64 0x24 0x18 0x48 0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 0 [idle]:
Oct 12 16:41:40  lotus-worker[73229]: runtime: unknown pc 0x20bffbc
Oct 12 16:41:40  lotus-worker[73229]: stack: frame={sp:0x7f189d7f7c50, fp:0x0} stack=[0x7f189cffa108,0x7f189d7f9d08)
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b50:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b60:  00007f1917ad5160  00000000020b0385
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b70:  00007f189d7f7c20  00007f189d7f7bf8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b80:  0000000000000004  00007f187c017980
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b90:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ba0:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bb0:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bc0:  0000000000000000  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bd0:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7be0:  00007f1917ad5160  00000000020be0e0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bf0:  0000000004d08d48  00007f189d7f7bd0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c00:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c10:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c20:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c30:  0000000004d08b38  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c40:  0000000004d08b38  00000000020bffbc
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c50: <0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c60:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c70:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c80:  0000000003f986d6  00007f187c0177d0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c90:  0000000000000001  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ca0:  00007f187c003270  00000000020231b6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cb0:  0000000000000000  0000000003f986d6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cc0:  0000000000000018  000000000202317e
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cd0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ce0:  0000000004d08d48  00000000005029ac
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cf0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d00:  0000000004d08d48  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d10:  00007f187c003290  0000000002025373
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d20:  0000000004ccf7c8  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d30:  0000000000000201  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d40:  00007f189d7f7d01  00007f187c0044e0
Oct 12 16:41:40  lotus-worker[73229]: runtime: unknown pc 0x20bffbc
Oct 12 16:41:40  lotus-worker[73229]: stack: frame={sp:0x7f189d7f7c50, fp:0x0} stack=[0x7f189cffa108,0x7f189d7f9d08)
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b50:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b60:  00007f1917ad5160  00000000020b0385
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b70:  00007f189d7f7c20  00007f189d7f7bf8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b80:  0000000000000004  00007f187c017980
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b90:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ba0:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bb0:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bc0:  0000000000000000  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bd0:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7be0:  00007f1917ad5160  00000000020be0e0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bf0:  0000000004d08d48  00007f189d7f7bd0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c00:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c10:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c20:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c30:  0000000004d08b38  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c40:  0000000004d08b38  00000000020bffbc
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c50: <0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c60:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c70:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c80:  0000000003f986d6  00007f187c0177d0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c90:  0000000000000001  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ca0:  00007f187c003270  00000000020231b6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cb0:  0000000000000000  0000000003f986d6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cc0:  0000000000000018  000000000202317e
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cd0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ce0:  0000000004d08d48  00000000005029ac
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cf0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d00:  0000000004d08d48  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d10:  00007f187c003290  0000000002025373
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d20:  0000000004ccf7c8  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d30:  0000000000000201  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d40:  00007f189d7f7d01  00007f187c0044e0
Oct 12 16:41:40  lotus-worker[73229]: goroutine 298 [syscall]:
Oct 12 16:41:40  lotus-worker[73229]: runtime.cgocall(0x18f4570, 0xc002dd1ad0, 0x28064e0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/cgocall.go:154 +0x5b fp=0xc002dd1aa0 sp=0xc002dd1a68 pc=0x51c7fb
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/filecoin-ffi/generated._Cfunc_fil_seal_pre_commit_phase2(0x7f187c000ca0, 0x5e9, 0x7f187c0012a0, 0x7f187c0012d0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         _cgo_gotypes.go:2021 +0x49 fp=0xc002dd1ad0 sp=0xc002dd1aa0 pc=0x12d6e49
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/filecoin-ffi/generated.FilSealPreCommitPhase2(0xc0023e6600, 0x5e9, 0x5eb, 0x5e9, 0xc001c50a50, 0x22, 0xc001c50a80, 0x23, 0xc001c50960)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/filecoin-ffi/generated/generated.go:766 +0x177 fp=0xc002dd1bb0 sp=0xc002dd1ad0 pc=0x12e0f97
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/filecoin-ffi.SealPreCommitPhase2(0xc0023e6600, 0x5e9, 0x5eb, 0xc001c509f0, 0x21, 0xc001c508d0, 0x22, 0x0, 0x0, 0x0, ...)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/filecoin-ffi/proofs.go:370 +0xbe fp=0xc002dd1c50 sp=0xc002dd1bb0 pc=0x12e4efe
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).SealPreCommit2(0xc0005a1560, 0x2e65988, 0xc0011097d0, 0x5693, 0x3a1, 0x8, 0xc0023e6600, 0x5e9, 0x5eb, 0x0, ...)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/ffiwrapper/sealer_cgo.go:543 +0x2cd fp=0xc002dd1de8 sp=0xc002dd1c50 pc=0x12f0a0d
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage.(*LocalWorker).SealPreCommit2.func1(0x2e65988, 0xc0011097d0, 0x5693, 0x3a1, 0xc646e6988e91639b, 0x281ecc800f84d78c, 0xc0015680c0, 0xc001898f88, 0xc00043378c, 0xc001899038)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/worker_local.go:359 +0x91 fp=0xc002dd1ea0 sp=0xc002dd1de8 pc=0x130f591
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage.(*LocalWorker).asyncCall.func1(0xc000215050, 0x2e65528, 0xc00063b680, 0xc000e12690, 0x5693, 0x3a1, 0xc646e6988e91639b, 0x281ecc800f84d78c, 0x2a78d99, 0xe)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/worker_local.go:242 +0x102 fp=0xc002dd1f90 sp=0xc002dd1ea0 pc=0x130e9e2
Oct 12 16:41:40  lotus-worker[73229]: runtime.goexit()
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc002dd1f98 sp=0xc002dd1f90 pc=0x58bec1
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/filecoin-project/lotus/extern/sector-storage.(*LocalWorker).asyncCall
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/worker_local.go:234 +0x31d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 1 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91c70, 0x72, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc001137498, 0x72, 0x0, 0x0, 0x2a2fda3)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Accept(0xc001137480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:401 +0x212
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).accept(0xc001137480, 0x232a03edc5ed0af8, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_unix.go:172 +0x45
Oct 12 16:41:40  lotus-worker[73229]: net.(*TCPListener).accept(0xc000d6ee58, 0x61659ea4, 0xc0022075b8, 0x5b21a6)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
Oct 12 16:41:40  lotus-worker[73229]: net.(*TCPListener).Accept(0xc000d6ee58, 0xc002207608, 0x18, 0xc000000180, 0x8b7bfb)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/tcpsock.go:261 +0x65
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*Server).Serve(0xc0003502a0, 0x2e5d570, 0xc000d6ee58, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:2981 +0x285
Oct 12 16:41:40  lotus-worker[73229]: main.glob..func5(0xc000793840, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:550 +0x24ba
Oct 12 16:41:40  lotus-worker[73229]: github.com/urfave/cli/v2.(*Command).Run(0x4d55fc0, 0xc0007934c0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:164 +0x4dd
Oct 12 16:41:40  lotus-worker[73229]: github.com/urfave/cli/v2.(*App).RunContext(0xc000245200, 0x2e654b8, 0xc000040078, 0xc00003c090, 0x9, 0x9, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:306 +0x810
Oct 12 16:41:40  lotus-worker[73229]: github.com/urfave/cli/v2.(*App).Run(...)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:215
Oct 12 16:41:40  lotus-worker[73229]: main.main()
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:112 +0x6ec
Oct 12 16:41:40  lotus-worker[73229]: goroutine 6 [select, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc0001d9bf0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/[email protected]/writer/writer.go:71 +0x106
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/ipfs/go-log/writer.NewMirrorWriter
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/[email protected]/writer/writer.go:36 +0xb9
Oct 12 16:41:40  lotus-worker[73229]: goroutine 7 [select]:
Oct 12 16:41:40  lotus-worker[73229]: go.opencensus.io/stats/view.(*worker).start(0xc0001f8300)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/[email protected]/stats/view/worker.go:276 +0xcd
Oct 12 16:41:40  lotus-worker[73229]: created by go.opencensus.io/stats/view.init.0
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x68
Oct 12 16:41:40  lotus-worker[73229]: goroutine 274 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f6620, 0x0, 0xc0003248c0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 297 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca919b8, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc000f26218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc000f26200, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc000f26200, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x7f18eca919c0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc000011ff8, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*connReader).Read(0xc00063a570, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:800 +0x1b9
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc00121fb00)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc00121fb00, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).readRequest(0xc00063d540, 0x2e65480, 0xc000f14280, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:983 +0xec5
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).serve(0xc00063d540, 0x2e65528, 0xc000f14280)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:1878 +0x705
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Server).Serve
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:3013 +0x39b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 159 [chan receive, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/cli/util.ReqContext.func1(0xc000d38ea0, 0xc000319520)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cli/util/api.go:396 +0x34
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/filecoin-project/lotus/cli/util.ReqContext
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cli/util/api.go:395 +0x108
Oct 12 16:41:40  lotus-worker[73229]: goroutine 161 [syscall, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: os/signal.signal_recv(0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/sigqueue.go:168 +0xa5
Oct 12 16:41:40  lotus-worker[73229]: os/signal.loop()
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x25
Oct 12 16:41:40  lotus-worker[73229]: created by os/signal.Notify.func1.1
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/os/signal/signal.go:151 +0x45
Oct 12 16:41:40  lotus-worker[73229]: goroutine 169 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:90 +0xb1
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:142 +0x429
Oct 12 16:41:40  lotus-worker[73229]: goroutine 258 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc001716400, 0xc000648060)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1200 +0x125
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:360 +0x9a5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 275 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f6620, 0x1, 0xc0003248c0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 170 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0xd7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:143 +0x44b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 171 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:825 +0x228
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x5a5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 168 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0003500e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:206 +0x105
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:237 +0x18c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 172 [select, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:762 +0x125
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:150 +0x5c7
Oct 12 16:41:40  lotus-worker[73229]: goroutine 173 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0011b9ce0, 0xc0001b4860)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 174 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0011b9d10, 0xc0001b4860)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 175 [select, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc001556800, 0xc0001b4920)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1200 +0x125
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:360 +0x9a5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 176 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f64d0, 0x0, 0xc0001b4c60)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 177 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f64d0, 0x1, 0xc0001b4c60)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 226 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc001556800, 0xc0001b4c80, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1107 +0xb7
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.Open.func5(0xc000010db0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:377 +0x37
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:376 +0x1445
Oct 12 16:41:40  lotus-worker[73229]: goroutine 228 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0015569d0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1948 +0x15c
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*valueLog).open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1114 +0x6c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 242 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc001556800, 0xc0003b00e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:862 +0x533
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:411 +0xf34
Oct 12 16:41:40  lotus-worker[73229]: goroutine 243 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0015569d0, 0xc0003b01e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1851 +0x65
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:415 +0x1165
Oct 12 16:41:40  lotus-worker[73229]: goroutine 244 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc0011b9da0, 0xc0003b05e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/publisher.go:67 +0x145
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:419 +0x101b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 245 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0xc000f08060)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/[email protected]/datastore.go:172 +0xe5
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/ipfs/go-ds-badger2.NewDatastore
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/[email protected]/datastore.go:160 +0x65d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 246 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc000f12090, 0xc0003b0780)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 247 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc000f120c0, 0xc0003b0780)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 276 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc001716400, 0xc000324900, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1107 +0xb7
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.Open.func5(0xc000b0c008)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:377 +0x37
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:376 +0x1445
Oct 12 16:41:40  lotus-worker[73229]: goroutine 279 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc001716400, 0xc000324b00)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:862 +0x533
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:411 +0xf34
Oct 12 16:41:40  lotus-worker[73229]: goroutine 278 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0017165d0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1948 +0x15c
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*valueLog).open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1114 +0x6c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 280 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0017165d0, 0xc000324b20)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1851 +0x65
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:415 +0x1165
Oct 12 16:41:40  lotus-worker[73229]: goroutine 281 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc000f12150, 0xc000324b40)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/publisher.go:67 +0x145
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:419 +0x101b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 282 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0xc000d68c60)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/[email protected]/datastore.go:172 +0xe5
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/ipfs/go-ds-badger2.NewDatastore
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/[email protected]/datastore.go:160 +0x65d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 286 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage/stores.(*Local).reportHealth(0xc001568120, 0x2e65480, 0xc000793a40)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/stores/local.go:330 +0x11b
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/filecoin-project/lotus/extern/sector-storage/stores.(*Local).open
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/stores/local.go:242 +0x1ae
Oct 12 16:41:40  lotus-worker[73229]: goroutine 284 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91d58, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc001202898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc001202880, 0xc001554000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc001202880, 0xc001554000, 0x1000, 0x1000, 0x55433c, 0xc000a50c38, 0x5830a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc000010b30, 0xc001554000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).Read(0xc000c90900, 0xc001554000, 0x1000, 0x1000, 0xc0015642a0, 0xc000a50d40, 0x51e735)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1922 +0x77
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc000d68d80)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc000d68d80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).readLoop(0xc000c90900)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2083 +0x1a8
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1743 +0xc77
Oct 12 16:41:40  lotus-worker[73229]: goroutine 285 [select]:
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).writeLoop(0xc000c90900)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2382 +0xf7
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1744 +0xc9c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 288 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: main.glob..func5.5(0x2e65480, 0xc000793a40, 0xc0003502a0)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:434 +0x65
Oct 12 16:41:40  lotus-worker[73229]: created by main.glob..func5
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:433 +0x1ca5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 306 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91b88, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc001137618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc001137600, 0xc0023b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc001137600, 0xc0023b2000, 0x1000, 0x1000, 0x55433c, 0xc0001a7c38, 0x5830a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc0003fa108, 0xc0023b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).Read(0xc0004625a0, 0xc0023b2000, 0x1000, 0x1000, 0xc000253560, 0xc0001a7d40, 0x51e735)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1922 +0x77
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc001205b60)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc001205b60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).readLoop(0xc0004625a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2083 +0x1a8
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1743 +0xc77
Oct 12 16:41:40  lotus-worker[73229]: goroutine 307 [select]:
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).writeLoop(0xc0004625a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2382 +0xf7
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1744 +0xc9c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 291 [select]:
Oct 12 16:41:40  lotus-worker[73229]: main.glob..func5.7(0xc001568120, 0x2e65480, 0xc000793a40, 0xc000d938a0, 0x2e94f28, 0xc000615000, 0xc000d643f0, 0xc000044198, 0x11, 0xc000319680)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:527 +0x411
Oct 12 16:41:40  lotus-worker[73229]: created by main.glob..func5
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:487 +0x248f
Oct 12 16:41:40  lotus-worker[73229]: goroutine 308 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91aa0, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc0001f8998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc0001f8980, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc0001f8980, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x7f18eca91aa8, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc0003fa110, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*connReader).Read(0xc001094d80, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:800 +0x1b9
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc001205c20)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc001205c20, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).readRequest(0xc0003da500, 0x2e65480, 0xc000c42840, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:983 +0xec5
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).serve(0xc0003da500, 0x2e65528, 0xc000c42840)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:1878 +0x705
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Server).Serve
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:3013 +0x39b
Oct 12 16:41:40  lotus-worker[73229]: rax    0x1004
Oct 12 16:41:40  lotus-worker[73229]: rbx    0x2
Oct 12 16:41:40  lotus-worker[73229]: rcx    0x4
Oct 12 16:41:40  lotus-worker[73229]: rdx    0x4d0dad0
Oct 12 16:41:40  lotus-worker[73229]: rdi    0x0
Oct 12 16:41:40  lotus-worker[73229]: rsi    0x3fa9eac
Oct 12 16:41:40  lotus-worker[73229]: rbp    0x7f1917ad5160
Oct 12 16:41:40  lotus-worker[73229]: rsp    0x7f189d7f7c50
Oct 12 16:41:40  lotus-worker[73229]: r8     0x0
Oct 12 16:41:40  lotus-worker[73229]: r9     0x0
Oct 12 16:41:40  lotus-worker[73229]: r10    0x7f189d7f6c60
Oct 12 16:41:40  lotus-worker[73229]: r11    0x0
Oct 12 16:41:40  lotus-worker[73229]: r12    0x4d08d48
Oct 12 16:41:40  lotus-worker[73229]: r13    0x0
Oct 12 16:41:40  lotus-worker[73229]: r14    0x4d08b38
Oct 12 16:41:40  lotus-worker[73229]: r15    0x7f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: rip    0x20bffbc
Oct 12 16:41:40  lotus-worker[73229]: rflags 0x10202
Oct 12 16:41:40  lotus-worker[73229]: cs     0x33
Oct 12 16:41:40  lotus-worker[73229]: fs     0x0
Oct 12 16:41:40  lotus-worker[73229]: gs     0x0

Repo Steps

Build lotus with FIL_PROOFS_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"
export FIL_PROOFS_GPU_FRAMEWORK=cuda
Do PC2

jennijuju · 2021-10-12T15:37:02Z

jennijuju
Oct 12, 2021
Maintainer

You should use BELLMAN_CUDA_NVCC_ARGS instead.

Where did you get the instruction to set FIL_PROOFS_CUDA_NVCC_ARGS? we should update that

3 replies

kenshyx Oct 12, 2021
Author

from here filecoin-project/lotus#7443 (comment)

jennijuju Oct 12, 2021
Maintainer

OMG WHO IS THAT OZZY PERSON WHY HE'S GIVING FALSE INSTRUCTIONS!

jennijuju Oct 12, 2021
Maintainer

(he corrected it😜

vmx · 2021-10-12T15:46:15Z

vmx
Oct 12, 2021
Collaborator

Setting

BELLMAN_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"
NEPTUNE_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"

should make it work.

0 replies

cryptonemo · 2021-10-12T15:50:03Z

cryptonemo
Oct 12, 2021
Maintainer

I'm concerned by the SIGILL: illegal instruction -- are we sure it's not a basic "this isn't compiled properly for the native platform" issue?

1 reply

dignifiedquire Oct 12, 2021

This seems to be the criticial part GpuError("Cuda(NoBinaryForGpu)"). The cuda kernel is not compiled for the right architecture, which @vmx instructions should fix.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA PC2 on Tesla V100S panic #1522

{{title}}

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

CUDA PC2 on Tesla V100S panic #1522

kenshyx Oct 12, 2021

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Repo Steps

Replies: 3 comments · 4 replies

jennijuju Oct 12, 2021 Maintainer

kenshyx Oct 12, 2021 Author

jennijuju Oct 12, 2021 Maintainer

jennijuju Oct 12, 2021 Maintainer

vmx Oct 12, 2021 Collaborator

cryptonemo Oct 12, 2021 Maintainer

dignifiedquire Oct 12, 2021

kenshyx
Oct 12, 2021

Replies: 3 comments 4 replies

jennijuju
Oct 12, 2021
Maintainer

kenshyx Oct 12, 2021
Author

jennijuju Oct 12, 2021
Maintainer

jennijuju Oct 12, 2021
Maintainer

vmx
Oct 12, 2021
Collaborator

cryptonemo
Oct 12, 2021
Maintainer