QuickLZ

QuickLZ is a toolbox for fast lossless compression and decompression of files and data.

This download makes these tools available at the Matlab prompt.

The download contains mex files that have been tested with Matlab versions 2006b and 2010b, and include source code. Usage is free under GPL conditions as described in the accompanying file quicklz.h.

Acknowledgement

Thanks to Lasse Mikkel Reinhold for his permission to use his code.

Short example using observed road traffic flows

The matrix "flows" contains observed vehicle counts from the Dutch National HighWay System for 2016 fifteen minute periods (21 days), and 5917 cross sections. The matrix is stored as "uint6" to economize on memory as much as possible without compressing data:

>> whos flows
Name          Size                   Bytes  Class     Attributes
flows      2016x5917            23.857.344  uint16

Using the zlib based utility "compress", we may compress the matrix:

>> tic;flowsZlib=compress(flows);toc
Elapsed time is 2.505811 seconds.

>> whos flowsZlib
Name                  Size                 Bytes  Class    Attributes
flowsZlib      15647569x1             15.647.569  int8

This results in a compressed size that is about 65% of the original. Compression took about 2.5 seconds.

If we repeat the experiment with the new quicklz utility, we get the following result:

>> tic;flowsQuicklz=compressQuicklz(flows);toc
Elapsed time is 0.143340 seconds.

>> whos flowsQuicklz
Name                     Size                 Bytes  Class    Attributes
flowsQuicklz      19394833x1             19.394.833  int8

For QuickLZ compressed size is about 81% of the original. Compression took about 0.14 second.

Both Zlib based and QuickLz based algortitms only reach moderate compression for this particular type of data. If we repeat the experiments for typical "status" data with a more repetitive nature, we obtain these results:

Data Size (bytes) Compression Rate (%) Compression time (sec)
Original data 11928672 - -
Zlib compressed data 182399 98.5% 0.23 sec
QuickLZ compressed data 443150 96.3% 0.022 sec

Compression level

Both zlib and quicklz were applied using compression level 1. Unlike zlib, the quicklz compression level and compression time are heavily impacted by the compression that is selected. Quicklz can be used with 3 compression levels, Selecting level 3 improves compression significantly, but even more so the execcution time, making the method less distinctive from zlib. To use quicklz in the niche were it excels, compression level 1 is preferable.

Typical application area

Modelit utilizes QuickLz as a part of the deflated vector object (deflatev) and deflated matrix object (deflate) that is built on top of that. These objects use two strategies to handle extremely large datasets: segmentation and compression. Segmentation divides large vectors into segments. This removes the need to store  all data in contiguous memory. Compression reduces the total need for memory.  The deflated vector object can be used without compression,  with fast compression using quicklz or with full compression using zlib.