clamav-devel December 2011 archive
Main Archive Page > Month Archives  > clamav-devel archives
clamav-devel: Re: [Clamav-devel] scanning large zip files ->

Re: [Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

From: aCaB <acabng_at_nospam>
Date: Mon Dec 12 2011 - 13:16:37 GMT
To: ClamAV Development <clamav-devel@lists.clamav.net>

On 12/12/11 09:31, Bram wrote:
> Questions:
> Is it expected that the RSS memory increases with approximately the size
> of the zip file before extracting it?

No. The RSS memory size will increase at most by the smaller of
- compressed internal file size
- max-filesize (the file is skipped if the value's exceeded)
- max-scansize (the scan is aborted if the value's exceeded)

> Is this necessary? (Tools such as unzip are able to decompress the file
> without loading the entire file into memory).

Not necessary but it makes the code much simpler in the default case
scenario as, with the default settings, the allocated memory is way
below CLI_MAX_ALLOCATION.
I agree that the implicit relation between the alloc size and the
max-(scan|file)size setting could be better documented.
But frankly I don't see a point in refactoring the code to handle
insanely large values of max-XXXsize in a way that is memory efficient.
After all we're targeting malware which is on avg 2-300 KB in size.

> What is the purpose of fmap/fmap_need_ptr_once?

That's basically an alloc and read.

HtH,
--aCaB
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net