scan speed low

[chennqqi]

time http -f localhost:3993/scan [email protected]

HTTP/1.1 200 OK
Content-Length: 129
Content-Type: application/json; charset=UTF-8
Date: Thu, 24 Aug 2017 06:17:10 GMT

{
    "clamav": {
        "engine": "0.99.2", 
        "error": "", 
        "infected": true, 
        "known": "6303004", 
        "result": "Eicar-Test-Signature", 
        "updated": "20170813"
    }
}


real	0m14.414s
user	0m0.244s
sys	0m0.041s

scan a simple txt cost more than 14s seconds

[blacktop]

this is related to this: malice-plugins/avast#2 (comment)

The reason being that every scan creates a new docker container, which means that the virus-signature DB needs to be read into memory each time.

A way to speed it up would be to start a container and then keep sending it samples. I believe if you used the plugin as a web-service it might scan faster https://github.com/malice-plugins/clamav/blob/master/docs/web.md

However, there is a bit of a security trade off as running on scan per container means that if the malware can corrupt the docker container it will only break that container, but it will be discarded and not used a second time.

FYI, these plugins aren't hardened yet, but they will be. Meaning I will turn on all the available docker security features to limit the risk the the host running docker in case of malware that tries to attack docker itself.

[chennqqi]

Thank you for your response. I certainly used the plugin as a web-service.
time http -f localhost:3993/scan [email protected]
or some http api call..
but each request cost more than 14s seconds.

[blacktop]

Ah yes sorry about that I am more tired than I thought. So I went into the container to just run clamav by itself without my Golang app frontend proxy service and saw this:

/malware # clamscan EICAR
LibClamAV Warning: Cannot dlopen libclamunrar_iface: file not found - unrar support unavailable
LibClamAV Warning: **************************************************
LibClamAV Warning: ***  The virus database is older than 7 days!  ***
LibClamAV Warning: ***   Please update it as soon as possible.    ***
LibClamAV Warning: **************************************************
EICAR: Eicar-Test-Signature FOUND

----------- SCAN SUMMARY -----------
Known viruses: 6303004
Engine version: 0.99.2
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 11.608 sec (0 m 11 s)
/malware # clamscan EICAR
LibClamAV Warning: Cannot dlopen libclamunrar_iface: file not found - unrar support unavailable
LibClamAV Warning: **************************************************
LibClamAV Warning: ***  The virus database is older than 7 days!  ***
LibClamAV Warning: ***   Please update it as soon as possible.    ***
LibClamAV Warning: **************************************************
EICAR: Eicar-Test-Signature FOUND

----------- SCAN SUMMARY -----------
Known viruses: 6303004
Engine version: 0.99.2
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 11.779 sec (0 m 11 s)

This means clamav is JUST slow. So the way to speed it up would be to run a lot of them in parallel and then use a reverse proxy like nginx in front of them OR you could run a bunch in a docker-swarm.

[chennqqi]

@blacktop the main cost of scan progress is load clam-av db.
scan multi file in each post would be better.
or using <github.com/mirtchovski/clamav> in docker.

[DDB-en]

Is it possible to use clamdscan instead of clamscan in web api?!
that's too fast, for example less than 1 second for EICAR:
Time: 0.010 sec (0 m 0 s)
clamdscan is Daemon Client and no need to read whole the signature database everytime!