Multithread option is much slower than simply using GNU parallel to spawn multiple jpegoptim processes

[catharsis71]

My normal use cases for jpegoptim involve using find | parallel jpegoptim utilizing GNU parallel to spawn multiple jpegoptim processes. I tried out the new multithreading in 1.5.0 however it seems much slower than what you get with parallel.

testing on a 4-core ARM VPS:

Single-core performance test:

root@uk2:/var/www/fanart/unsorted/1# time jpegoptim-test -w 1 -n --nofix -q *.jpg

real    0m19.419s
user    0m18.808s
sys     0m0.607s

4-core performance test (only a 60% time reduction when I'd expect a ~75% time reduction):

root@uk2:/var/www/fanart/unsorted/1# time jpegoptim-test -w 4 -n --nofix -q *.jpg

real    0m7.806s
user    0m19.547s
sys     0m10.565s

4-core test using GNU parallel to spawn multiple jpegoptim processes (72% time reduction versus single core test):

root@uk2:/var/www/fanart/unsorted/1# time ls *.jpg | parallel -n 100 jpegoptim-test -w 1 -n --nofix -q

real    0m5.346s
user    0m18.978s
sys     0m1.138s

Using a larger set of files:

Single-core:

root@uk2:/var/www/fanart# time find -iname *.jpg | xargs jpegoptim-test -w 1 -n --nofix -q

real    1m6.324s
user    1m5.109s
sys     0m1.222s

4-core (only a 58% time reduction, expected ~75%):

root@uk2:/var/www/fanart# time find -iname *.jpg | xargs jpegoptim-test -w 4 -n --nofix -q

real    0m27.818s
user    1m8.227s
sys     0m34.509s

4-core using GNU parallel (70% time reduction versus single core test):

root@uk2:/var/www/fanart# time find -iname *.jpg | parallel -n 100 jpegoptim-test -w 1 -n --nofix -q

real    0m19.970s
user    1m6.209s
sys     0m6.347s

Using an even larger set of files:

Single-core:

root@uk2:/var/www# time find -iname *.jpg | xargs jpegoptim-test -w 1 -n --nofix -q

real    1m32.259s
user    1m29.135s
sys     0m3.145s

Four-core (only 55% time reduction, expected around ~75% reduction):

root@uk2:/var/www# time find -iname *.jpg | xargs jpegoptim-test -w 4 -n --nofix -q

real    0m41.669s
user    1m35.060s
sys     1m7.442s

Four-core with GNU parallel (73% time reduction versus single core test):

root@uk2:/var/www# time find -iname *.jpg | parallel -n 100 jpegoptim-test -w 1 -n --nofix -q

real    0m24.542s
user    1m30.582s
sys     0m5.328s

Note: I don't explicitly tell parallel how many threads to use because it automatically matches the number of system cores (4). I do use "-n 100" because spawning a jpegoptim process for every file is wasteful, and 1 process per 100 files is much better. So parallel spawns 4 jpegoptim processes each processing 100 files, and when each batch finishes it starts another until all are done.

[catharsis71]

Testing on a 12-core / 24-thread AMD64 desktop system:

Single-thread test:

$ time find -iname *.jpg | xargs jpegoptim-150 -w 1 -n --nofix -q

real    2m51.032s
user    2m27.781s
sys     0m11.922s

12-thread test (69% time reduction):

$ time find -iname *.jpg | xargs jpegoptim-150 -w 12 -n --nofix -q

real    0m53.561s
user    3m7.172s
sys     3m58.734s

24-thread test (67% time reduction):

$ time find -iname *.jpg | xargs jpegoptim-150 -w 24 -n --nofix -q

real    0m55.822s
user    3m17.906s
sys     4m11.031s

24-thread GNU parallel test (93% time reduction):

$ time find -iname *.jpg | parallel -n 100 jpegoptim-150 -w 1 -n --nofix -q

real    0m12.536s
user    3m38.297s
sys     0m32.641s

[catharsis71]

Summary:

On a 4-core ARM VPS, using GNU parallel to spawn multiple jpegoptim processes reduces execution time by 70-73%, whereas using jpegoptim's new built-in multithread support only reduces execution time by 55-60%

On a 12-core / 24-thread AMD desktop, using GNU parallel to spawn multiple jpegoptim processes reduces execution time by 93%, whereas using jpegoptim's new built-in multithread support only reduces execution time by 67-69%

[tjko]

@catharsis71 , interesting findings, thanks!
I have always thought it would be better to use something like GNU parallel , but its not available on all platforms, and seems like large number of users would much rather have this built-in, based on requests for the feature...

How many images (roughly) you used to run these test? These are probably rather small JPEGs (do you know what is the average size?)

I would guess that this is likely due to the fact that with -w option, jpegoptim will fork new process for each image. So processing large number of small images likely yield pretty high "overhead"....

It would be interesting to se results when parallel was run with option -n 1....

[catharsis71]

The files are various sizes but capped at 2MB post-optimization

I'll do some more testing using a directory with 1861 JPGs, 329MB, so somewhere around 180KB per file, using the 4-core system

# time jpegoptim-test -n -q --nofix -w 1 *.jpg

real    0m19.365s
user    0m19.007s
sys     0m0.355s

# time jpegoptim-test -n -q --nofix -w 4 *.jpg

real    0m7.655s
user    0m19.885s
sys     0m10.108s

# time ls *.jpg | parallel jpegoptim-test -n -q --nofix -w 1

real    0m9.759s
user    0m22.256s
sys     0m12.039s

# time ls *.jpg | parallel -n 5 jpegoptim-test -n -q --nofix -w 1

real    0m6.384s
user    0m20.041s
sys     0m3.973s

# time ls *.jpg | parallel -n 25 jpegoptim-test -n -q --nofix -w 1

real    0m5.650s
user    0m19.391s
sys     0m1.591s

# time ls *.jpg | parallel -n 100 jpegoptim-test -n -q --nofix -w 1

real    0m5.295s
user    0m19.194s
sys     0m0.799s

# time ls *.jpg | parallel -n 200 jpegoptim-test -n -q --nofix -w 1

real    0m6.386s
user    0m19.137s
sys     0m0.650s

So the -w multiprocessing does actually outperform parallel IF you're not using a -n option to parallel (equivalent to -n 1), since there's a lot of overhead spawning a process for every single file

But parallel pulls ahead using even a small -n value of 5, and continues getting faster up to around -n 100, then slows down a bit if you go beyond that

If all files required an equal time to process, probably parallel would be fastest using n = 466 (total number of files divided by 4 rounding up), so that only 4 total processes would need to be spawned, but in reality you end up with unequal workloads and on process will take longer to finish than the others

[tjko]

I guess could always implement something like the -n option in parallel, then jpegoptim should (in theory) perform comparably.
It not trivial change to implement, so I probably won't get around to do it immediately, so for now for absolutely best performance probably best keep using GNU parallel.