GPU memory management

[KSepetanc]

I found out that solving the ExaModels AC OPF documentation example in a loop results in possible memory leak. All of 6 GB of GPU's VRAM is taken in about 33 solves.

Below is part relevant part of the code and full code to reproduce can be downloaded: gpu memory leak.zip

m = ac_power_model("pglib_opf_case13659_pegase.m",backend=CUDABackend())

for i in 1:50
    println(i)
    madnlp(m,linear_solver=MadNLPGPU.CuCholeskySolver)
end

Configuration:
Windows,
CPU: AMD R9 5950x
GPU GTX 1060 6GB

[sshin23]

GPU memory usage is not necessarily optimized on MadNLP. Curious if this is CuCholeskySolver's issue or general MadNLP issue. Have you tried CUDSSSolver (the default solver for CUDA)?

[KSepetanc]

I have tried, but it fails with an error reported in 333.

I run dual boot of Windows and Linux and on Linux there is much less severe memory issue, but still existent.

[KSepetanc]

With package update as suggested in 333 issue, I've managed to run OPF with CUDSSSolver.
I the memory issue is still present. Memory usage goes slowly up with repeated runs of MadNLP until failure due to full VRAM.

[sshin23]

Converting this as a discussion, as memory management within GPU is not something we have direct control over

[KSepetanc]

Do you think we could have some command to clear taken GPU memory taken by MadNLP without stopping Julia sesion?

[sshin23]

I'd suggest we start from https://cuda.juliagpu.org/stable/usage/memory/

Maybe you can try calling GC.gc() or CUDA.reclaim() after each MadNLP call?

[KSepetanc]

@sshin23 memory management issue is largely resolved with cuDSS 0.3 update. There is still build-up with repetitive running in the same session, but it is much smaller now. It is about 100 MB per run for 78k network. Also, less total GPU memory is used compared to v0.1.

To free GPU memory usage I need to call CUDA.reclaim(); GC.gc(); CUDA.reclaim();
Single call won't free all memory. However, freeing memory usage does not reset the memory build-up with consecutive running. It will still use about 100 MB more starting from previous build-up. So the issue is now largely mitigated, but not fully resolved yet. For now, this is good enough for me. Hope you release the master branch soon as I am working on some local package too and I don't know how to add master revision as dependency for custom package.

[sshin23]

@KSepetanc Thanks for letting us know; indeed we're observing improvement in memory management. The compatibility is updated in the recent release of MadNLPGPU