Memory leak?

[vonProteus]

What happened ?

while running app it consume ~8.4G after ~15 minutes

What did you expect to happen ?

i thing ram usage is little excessive and i didn't notice reduction in usage of ram only increasing

OS version

docker

App version

mansuf/mangadex-downloader@sha256:33312a64791de9434e6f2fcd2f423862e667a75fa3d30f47fa912ba0417cdac2

Installation origin

Other

Installation origin (other sources)

docker mansuf/mangadex-downloader

Reproducible command

args: --language en --save-as cbz --delay-requests "1.5" --force-http --progress-bar-layout=none /tmp/mangadex-urls.txt

Additional context

mangadex-urls.txt in i have ~50 links and ~4000 cbz files ~50GB
and this is next run (when i check for new parts)
in this particular run i didn't get any new parts

[vonProteus]

is it normal?

[mansuf]

Is the memory increased slowly or it increased to 8GB immediately ? And you said for checking new chapters only right ? So i assume there is no download involved, only checking files.

[vonProteus]

it increase slowly and no new chapters after completion of crone job

[mansuf]

Can you give me your mangadex-urls.txt ? So i can replicate this on my device

[vonProteus]

I'd rather not.
I don't want to share all my links.

sory

[mansuf]

No problem, keep the good stuff in there ( ͡° ͜ʖ ͡°)

Anyway, i don't have the exact problem is, but there is some possibilities.

The first one is when verifying files some file pointer may not closed properly
The second is the cache, usually at fetching user, author, groups, and cover_art. This cache is to prevent repeated requests to MangaDex API.

I can test download dozens of popular manga in cbz format and see if this problem can be replicated. But downloading those takes time sadly, so it would take a while to get the problem. While waiting you can contribute too by running a memory profiler for the application, if you found the problem you can report it here and i will fix it 👍 .

[vonProteus]

how to run memory profiler?

[mansuf]

You can use tool like guppy3 to run a memory profiler

[mansuf]

I cannot find any issue with memory increased up to 1GB or more on my device. But i found some issue that cause memory increased only 50-100MB not 1GB or more. The issue caused by SQL migrations is keeps running indefinitely. Maybe install this patch 61ec2d7 will fix the issue ? Let me know if this patch is not fixed the issue yet

pip uninstall mangadex-downloader

pip install git+https://github.com/mansuf/mangadex-downloader.git@61ec2d7d04899c59e97a84044d1098ad8078c17e

I'm using memory-profiler for watching memory usage. The results can be seen below

Command used:

python -m mangadex_downloader "https://mangadex.org/title/32d76d19-8a05-4db0-9fc2-e0b0648fe9d0"

Before patch

After patch