clamav-devel December 2011 archive
Main Archive Page > Month Archives  > clamav-devel archives
clamav-devel: [Clamav-devel] [PATCH] reduce memory usage when sc

[Clamav-devel] [PATCH] reduce memory usage when scanning zip files

From: Bram <bram-pynzni-qriry_at_nospam>
Date: Wed Dec 21 2011 - 16:07:46 GMT
To: clamav-devel@lists.clamav.net

Hi,

In the thread
http://lurker.clamav.net/message/20111212.083121.71a17ccd.en.html I
asked about memory usage when scanning a zip file.

I decided to have a look on how to change the behaviour and wrote a
patch for it.
I don't know if this patch is desirable or acceptable but I'm
submitting it anyway.
Patch is against 0.97.3 but should apply cleanly on git master.

To test the patch I tested 3 (large) files:

*File 1*

ZIP file of 477MB which contains 14 files.
The largest file in the zip is a CAB file of 489MB.
This CAB file contains:
* 57079 files that are smaller than 1MB,
* 112 files that are between 1MB and 10MB,
* 9 files that are larger then 10 MB (the largest being 132MB)

Original version:

----------- SCAN SUMMARY -----------
Known viruses: 1096211
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 990.25 MB
Data read: 476.21 MB (ratio 2.08:1)
Time: 103.864 sec (1 m 43 s)

Patched version:

----------- SCAN SUMMARY -----------
Known viruses: 1096215
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 990.25 MB
Data read: 476.21 MB (ratio 2.08:1)
Time: 102.853 sec (1 m 42 s)

*File 2*

The CAB file extract out of the above zip file (489MB).
This CAB file contains:
* 57079 files that are smaller than 1MB,
* 112 files that are between 1MB and 10MB,
* 9 files that are larger then 10 MB (the largest being 132MB)

Original version:

----------- SCAN SUMMARY -----------
Known viruses: 1096211
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 498.80 MB
Data read: 489.75 MB (ratio 1.02:1)
Time: 51.487 sec (0 m 51 s)

Patched version:

----------- SCAN SUMMARY -----------
Known viruses: 1096215
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 498.80 MB
Data read: 489.75 MB (ratio 1.02:1)
Time: 51.564 sec (0 m 51 s)

*File 3*

Zip file of 401MB that contains 1 file, with random data, of 400 MB

Original version:

----------- SCAN SUMMARY -----------
Known viruses: 1096220
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 804.19 MB
Data read: 400.06 MB (ratio 2.01:1)
Time: 81.984 sec (1 m 21 s)

Patched version:

----------- SCAN SUMMARY -----------
Known viruses: 1096220
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 804.19 MB
Data read: 400.06 MB (ratio 2.01:1)
Time: 82.960 sec (1 m 22 s)

Best regards,

Bram

_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net