Peak Memory Usage for dnf5

[toppk]

Is there any theory for how much memory dnf5 will consume for basic operations like "dnf5 upgrade" or "dnf5 install $FOO_PKG"?

I was hoping that dnf5 would get this under control, but even after the latest libsolv improvement, dnf5 memory usage still way too use on a 1gb vm without triggering oom.

For example, I'm wondering is it conceptually possible for dnf5 to run with less than 50MB? Is that based on the number of packages, repos? I can easily imagine that depsolving is creating tons of datastructures to try to model things, and that there's a tradeoff between cpu and memory, but it would seem that for all tiny use cases (embedded and micro vms) that memory is the hard resource that dnf needs to work around, and cpu is a secondary consideration.

[kontura]

The peak definitely happens when the repos are first loaded by libsolv. After the cache (solv) files are created it gets compressed requiring less memory on subsequent runs.
This means it is dependent on the repo size (number of packages),

In addition with dnf5 it can happen that it will be loading multiple repos at the same time which will need more memory but the user can configure the repos and work with only one at a time.

Right now 50MB doesn't seem possible, even in the best scenario I could not get below ~215 MiB.

I just did some testing in a F38 container:
with the following repos enabled:
fedora, fedora-cisco-openh264, fedora-modular, updates, updates-modular, copr:copr.fedorainfracloud.org:rpmsoftwaremanagement:dnf-nightly, copr:copr.fedorainfracloud.org:rpmsoftwaremanagement:dnf5-unstable, copr:copr.fedorainfracloud.org:rpmsoftwaremanagement:test-utils
running dnf5 install zoxide:

• without cache I got peak usage at 711.5 MiB.
• with generated cache the peak was at 250.2 MiB.

With just the updates repo, running dnf5 install zoxide:

• without cache I got peak usage at 440.4 MiB.
• with generated cache the peak was at 215.6 MiB.

With just the fedora repo, running dnf5 install zoxide:

• without cache I got peak usage at 673.6 MiB.
• with generated cache the peak was at 220.7 MiB.

(For context the updates repo has: 17466 pkgs and fedora repo has: 69222.)

With the copr:copr.fedorainfracloud.org:rpmsoftwaremanagement:test-utils (which is basically empty - 2 pkgs):

• without cache I got peak usage at 295.7 MiB.
• with generated cache the peak was at 215.6 MiB.

It does still seem too high to me, especially the last example. I will investigate further.

[kontura]

Previously I was doing the memory measuring with valgrind --tool=massif --pages-as-heap=yes dnf5 install zoxide...

Now I was testing running dnf5 in a VM with limited memory and it is running for me even with a limit set to 400 megabytes (with default fedora 38 repos and clean cache).
Also I was able to run dnf5 install with a small repo (copr:copr.fedorainfracloud.org:rpmsoftwaremanagement:test-utils) with less than 200 megabytes.

It seems like 1gb vm should be enough?

[j-mracek]

I tried following:

sudo /usr/bin/time -v /usr/bin/dnf5 install acpi -y

Maximum resident set size (kbytes): 277372

sudo /usr/bin/time -v /usr/bin/dnf5 upgrade -y
(Upgrade of 135 packages including kernel-core)

Maximum resident set size (kbytes): 373596

I am suggesting that processing larger transaction or bigger RPMs might require additional RAM.

[j-mracek]

Anyway this is huge improvement in comparison to microdnf that requires 550 MB.