OOM - How much ram is required for running 64 GiB sectors?

[jennijuju]

@kernelogicL tried to pledging two 64GB sectors and it killed his 256GB RAM miner with OOM.

Another miner also mentioned that when running two PC1 at once on 3900x with 128G RAM and have FIL_PROOFS_USE_MULTICORE_SDR=1 on, the worker was also killed by the OOM killer.

My question is, how much ram does running these proof require?

[kernelogic]

Update: once I turn off FIL_PROOFS_USE_MULTICORE_SDR, I can seal 2 64GB sectors at the same time without being OOM killed - so definitely some difference with this variable.

[cryptonemo]

@neondragon Have you dried disabling the mlock calls? For many people they were not succeeding anyway, and in that case it maps them unlocked, which may solve your issue and properly hit swap.

[neondragon]

@cryptonemo That could very plausibly explain why some people don't encounter the issue. I've seen it on machines with nothing else running where mlock was likely to succeed.

I've discovered something else I was unaware of. I've been running the tests as root, and according to https://linux.die.net/man/2/setrlimit

Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process may lock, and this limit instead governs the amount of memory that an unprivileged process may lock.

This may explain why my attempts to limit locked memory (ulimit -l, LimitMEMLOCK in systemd service) were unsuccessful.

I'll test with the mlock calls disabled and also as an unprivileged user and report back.

[kernelogic]

How to disable mlock calls? Where is it?

[neondragon]

@kernelogic I don't know, I grepped the code for 'mlock' and tried changing the one instance I found in storage-proofs/porep/src/stacked/vanilla/memory_handling.rs, but the process was still OOM killed.

I did find two other ways to get it to work though!

1. Run as a non-root user. 'ulimit -l' must not be unlimited.
2. Run as root using capsh to remove the memory lock capability from the process, like this
   capsh --drop=cap_ipc_lock --print -- -c "FIL_PROOFS_USE_MULTICORE_SDR=1 TMPDIR=/srv/lotus/tmp-bench FIL_PROOFS_MAXIMIZE_CACHING=1 RUST_LOG=debug /opt/lotus/lotus-bench sealing --storage-dir /srv/lotus/seal4_1/bench --skip-commit2 --skip-unseal --num-sectors 2 --parallel 2 --sector-size 32GiB"

Both of these methods are working for me, the process is using all the 128GB RAM and ~6.8GB swap.

I'm not completely sure if it can be called a bug in lotus or not. It seems like lotus shouldn't try to mlock more memory than the system's physical memory size. But, the mlock limit is a Linux feature. At any rate, it's really confusing!

[cryptonemo]

@kernelogic Check here: https://github.com/filecoin-project/rust-fil-proofs/blob/master/storage-proofs-porep/src/stacked/vanilla/memory_handling.rs#L262

I recommend just not running as root as @neondragon suggested.

[cryptonemo]

Addressed by #1413