The zlib convenience API we've been using so far has two problems:
- On Windows-64, where sizeof(long) == 4, the use of ulong for sizes
meant that we could not compress data compressable on other 64-bit
platforms (Unix). While zstream also uses ulong, being a stream API,
it allows feeding data in chunks. The total_in and total_out members
are only required for gzip compression and are otherwise just
informational. They're unsigned, so their overflow does not cause
UB. In summary, using zstream + deflate() allows us to compress more
than 4GiB of data even on Windows-64.
- On all platforms, we always allocated the output buffer in such a
way as to accommodate the pathological case of random, incompressible
data, so the output buffer was larger than the input. Using zstream
+ deflate(), we can start with a smaller buffer, then let zlib pick
up where it left off when it ran out of output buffer space, saving
memory in the common case that compression meaningfully reduces the
size. To avoid the first few rounds of reallocations, we continue to
use zlib's compressBound() for input less than 256KiB.
This completely fixes the compression side of QTBUG-106542 and
QTBUG-104972.
Pick-to: 6.4 6.3 6.2
Fixes: QTBUG-104972
Fixes: QTBUG-106542
Change-Id: Ia7e6c38403906b35462480fd611b482f05a5c59c
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2fd990b386