Help Memory Error
Processing large data sets can lead to the error message "HELP MEMORY". For users of Matlab on a Windows 32 bit PC this error occurs if the if the size of an individual array exceeds the size of the largest contiguous free block of Memory. In practice this limits the size of data sets that can be held in memory to 1.5 GB.
When confronted with the error "HELP MEMORY" one can do a number of things:
- choose a more efficient storage of data, for example int16 instead of double;
- use sparse matrices instead of full matrices;
- change the representation of data so that you do not need contiguous memory;
- save part of the data on disk instead of memory;
- switch to the 64 bit Matlab version.
The Modelit deflate utility compresses a matrix in memory. The contents of a compressed matrix can be accessed as any other matrix. Typically, compression reduces the contents with 85%. But matrices that contain highly repetitive data can be compressed to much smaller sizes.
Note: in this example the Matrix L compresses to about 12.5%. This is somewhat of a worst case because L is populated with fully random logicals. If instead L is populated with more repetitive data, the compression rate will increase to over 97%:
The deflate utility is used in TRIP. This package is used to prepare a historical database of traffic data for the on-line route planner/travel time predictor TripCast. Thanks to in memory compression, all traffic detector data for a full year can be kept in memory simultaneously. This greatly facilitates the necessary computations to check, supplement and aggregate these data.
Download source code
The source code of the deflated matrix utility is free to use and included in a zip archive:
When publishing work involving the software, please cite as follows:
Zijpp, N.J. van der (2016), Modelit Deflated Matrix utility for Matlab (Version yyyymmdd) [Software]. Available from www.modelit.nl
(with yyyymmdd the creation date of the version you have been using)