kdb+ has built-in optional file compression as of version 2.7.
Q) How do I compress a file?
A) Use the -19! operator. e.g.
- logicalBlockSize is a power of 2 between 12 and 20 (pageSize or allocation granularity to 1MB - pageSize for AMD64 is 4kB, sparc is 8kB. Windows seems to have a default allocation granularity of 64kB). When choosing the logicalBlockSize, consider the minimum of all the platforms that will access the files directly - otherwise you may encounter "disk compression - bad logicalBlockSize". Note that this argument affects both compression speed and compression ratio (larger blocks can be slower and better compressed)
- compressionAlgorithm is one of the following, 0 - none, 1 - kdb+ ipc, 2 - gzip, 3 - snappy () (since 3.4)
- compressionLevel is between 0 and 9 (valid only for gzip, use 0 for other algorithms)
Return value - the compression level achieved, as a %.
q)`:test set asc 10000000?100; / create a test data file q) / compress input file test, to output file ztest, using a block size of 128kB (2 xexp 17), gzip level 6 q)-19!(`:test;`:ztest;17;2;6) 99.87667 q)get[`:test]~get`:ztest / check the compressed data is the same as the uncompressed data 1b
Q) How do I save directly to a compressed file?
A) Since v2.8 2011.11.21, kdb+ supports streaming file compression, i.e. the data is compressed as it is written to disk.
This is achieved through the overriding of "set", in that the lhs target of set can be a list describing the file or splay target, with the compression parameters. For example
(`:ztest;17;2;6) set asc 10000?`3 (`:zsplay/;17;2;6) set .Q.en[`:.;(sym:asc 10000?`3;time:.z.p+til 10000;price:10000?1000.;size:10000?100)]
kdb+ compressed files/splays can also be appended to. e.g.
q)(`:zippedTest;17;2;6) set 100000?10;`:zippedTest upsert 100000?10;-21!`:zippedTest
Appending to files with an attribute (e.g. `p# on sym) causes the whole file to be read and rewritten. Currently, unless .z.zd is set accordingly, this would be rewritten in a non-compressed format regardless of its original format.
'compress new files' mode - active if .z.zd (zip defaults) present and valid. .z.zd can be set to an int vector of (blockSize;algo;zipLevel) to apply to all new files which do not have a file extension. .e.g
q).z.zd:(17;2;6);`:zfile set asc 10000?`3 -19!x and (`:file;size;algo;level) set x take precedence over .z.zd
To reset to not compress new files, use \x, e.g.
Since kdb+v2.8 2011.11.28, the zip params can be a dictionary of filenames!zipParams. The null ` entry in the dict is the default zipParams, or if no ` specified then default will be do not compress. e.g.
q)(`:splay/;``a`b!((17;2;9);(17;2;6);(17;2;6)))set(a:asc 1000000?10;b:asc 1000000?10;c:asc 1000000?10)
Do not use streaming compression with log files, as in the event of a crash, the log file will be unusable as it will be missing meta information from the end of the file. Streaming compression maintains the last block in memory and compresses/purges it as needed or latest on close of file handle.
Q) How do I decompress a file?
A) just read the file, e.g.
If you want to store it again decompressed, then use
`:uncompressedFile set get `:compressedFile
Q) I want to use algorithm #2 (gzip). Do I need additional libraries??
A) Yes, but these may already be installed on your system. It binds dynamically to zlib (more info at http://zlib.net). For windows we chose to be compatible with the pre-built zlib dlls from http://www.winimage.com/zLibDll/index.html. For linux and solaris you may find it convenient to install zlib using your package manager software, or consult your system administrator for assistance. Note that you will require the matching 32 or 64bit libs for kdb+, i.e. 32bit libs for 32bit kdb+, 64bit libs for 64bit kdb+.
Q) I want to use algorithm #3 (snappy). Do I need additional libraries??
A) Yes, but these may already be installed on your system, and can be utilized in kdb+v3.4 onwards. It binds dynamically to snappy (installation info at http://google.github.io/snappy). kdb+ will look for the following files on the respective OSs - windows:snappy.dll, osx:libsnappy.dylib, other:libsnappy.so.1. Consult your system administrator for assistance. Note that you will require the matching 32 or 64bit libs for kdb+, i.e. 32bit libs for 32bit kdb+, 64bit libs for 64bit kdb+. To install snappy on osx, the simplest method is to use macports:
$sudo port install snappy +universal $export LD_LIBRARY_PATH=/opt/local/lib
Q) When compressing data, if the source file and the target file are on the same drive, it might run slowly. Why?
A) The compression routine is reading from the source file, compressing the data and writing to the target file. The disk is likely being given many seek requests. Try to locate the targetFile to a different physical disk, this will reduce the number of seeks needed.
Q) How can I tell if a file is compressed?
q)-21!`:testfile compressedLength | 2100716 uncompressedLength| 8000300 algorithm | 2 logicalBlockSize | 17
Q) Does the compression format support random access to the data?
Q) Does the compression require column data to be compressed using the -19! operation, or will running gzip on column data also work?
A) Yes, you have to use the -19! operation as the file format is different to gzip.
Q) Is it be possible to compress only some partitions, or even only some columns for a table within a partition? (For instance, for a database partitioned by date.)
A) Yes, you can choose which files to compress, and which algorithm/level to use per file; the same kdb+ process can read compressed and uncompressed files. So for files that don't compress well, or have an access pattern that does not perform well with compression, you could leave those uncompressed.
Q) How fast is the decompression?
A) A single thread with full use of a core can decompress approx 300MB/s, depending on data/algorithm and level.
Q) Is there a performance impact?
A) People are often concerned about the cpu overhead associated with compression, but the actual cost is difficult to calculate. On the one hand, compression does trade cpu utilization for disk space savings. And up to a point, if you're willing to trade more CPU time, you can get more space savings. But by reducing the space used, you end up doing less disk I/O, which can improve overall performance if your workload is bandwidth-limited. The only way to really know the impact of compression on your disk utilization and system performance is to run your workload with different levels of compression and observe the results.
Q) When is decompression done and how long is decompressed data cached?
A) Files are mapped/unmapped on demand during a query, only the areas of the file that are touched are decompressed, i.e. it supports random access. Decompressed data is cached for the lifetime that a file is mapped. Since v2.7 2011.04.30 columns are mapped for the duration of the select.
Q) Say you're querying by date and sum over a date partitioned table, with each partition parted by sym - will the query just decompress parts of the column data for the syms in the query predicate?
Q) Can I append to a compressed file?
A) Yes, since kdb+ v2.8 2011.11.21. Appending to compressed enum files was blocked in v3.0 2012.05.17 due to potential concurrency issues, hence these files should not be compressed.
Q) Which is better, ZFS compression or built-in kdb+ compression?
A) Currently, ZFS compression probably has an edge due to keeping more decompressed data in cache which is available to all processes.
Q) Is the compression or decompression multithreaded?
A) The reading or writing of a compressed file must NOT be performed concurrently from multiple threads. However, multiple files can be read or written from their own threads concurrently (one file per thread). For example, a par.txt'd historical database with slave threads will be using the decompression in a multithreaded mode.
Q) What's the difference among different logicalBlockSize (pageSize to 1MB) and compressionLevel (1 to 9 for gzip)? What's a standard recommendation?
A) The logicalBlockSize represents how much data is taken as a compression unit, and consequently the minimum size of a block to decompress. e.g. using a logicalBlockSize of 128kB, a file of size 128000kB would be cut into 100 blocks, and each block compressed independently of the others. Later, if a single byte is requested from that compressed file, a minimum of 128kB would be decompressed to access that byte. Fortunately those types of access patterns are rare, and typically you would be extracting clumps of data that make a logical block size of 128kB quite reasonable. Ultimately, you should experiment with what suits your data, hardware and access patterns best. A good balance for taq data and typical taq queries is to use algorithm 1 (the same algorithm as used for ipc compression) with 128kB blockSizes. For those who can accept slower performance but better compression, they can choose gzip with compression level 6.
Q) hcount`:compressedFile returns the uncompressed file length. Is this intentional?
A) Yes. In our defense, ZFS has similar issues - http://blog.buttermountain.co.uk/2008/05/10/zfs-compression-when-du-and-ls-appear-to-disagree
Compressed file size can be obtained from -21!`:compressedFile
Q) Is there a limit on the number of compressed files that can be open simultaneously?
A) For releases prior to v3.1 2013.02.21 the limit is 4096 files; releases since then are limited by the environment/OS only (e.g. ulimit -n). There is no practical internal limit on the number of uncompressed files.
n.b. If using compressed files, please note kdb+ 3.2 onwards uses 2 file descriptors per file, so you may need to increase the ulimit that was used in prior versions
Q) Does kdb+ file compression use more memory than working with non-compressed files?
A) Due to the nature of working with vectors, kdb+ allocates enough space to decompress the whole vector, regardless of how much it finally uses. This reservation of memory is required as there is no backing store for the decompressed data, unlike with mapped files of non-compressed data which can always read the pages from file again should they have been dropped. However, this is reservation of memory only, and can be accommodated by increasing the swap space available - even though the swap should never actually be written to, the OS has to be kept happy that in the worst case scenario of decompressing the data in full, it could swap it out if needed. If you experience wsfull even when sufficient swap space is configured, check whether you have any soft/hard limits imposed with ulimit -v. Also be aware of your memory overcommit settings on linux, /proc/sys/vm/overcommit_memory and /proc/sys/vm/overcommit_ratio - these control how careful linux is when allocating address space wrt available physical memory plus swap.
Q) What compression ratios should I expect?
A) Using real nyse trade data, we observed the gzip algorithm at level 9 compressing to 15% of original size, and the ipc compression algorithm compressing to 33% of original size.
Q) Do you have any benchmarking tips?
A) Perform your benchmarks on the same hardware setup as you would use for production and be aware of the disk cache - flush the cache before each test. The disk cache can be flushed on linux using
sync ; sudo echo 3 | sudo tee /proc/sys/vm/drop_caches
and on osx, the OS command 'purge' can be used.
Q) Can I use a hardware accelerator card to improve compression performance?
A) Yes. For example, the AHA367-PCIe 10.0 Gbits/sec GZIP Compression/Decompression Accelerator Card.
kdb+ v2.7 can use this card via the zlib shared library api. The card may be obtained directly from aha.com -
This card was observed to be compatible with kdb+ 2.7 2010.08.24 on Linux 2.6.32-22-generic SMP Intel i5 750 @ 2.67GHz 8GB RAM.
Using sample NYSE quote data from 2010.08.05, 482 million rows, compression ratios and timings were observed as below.
The uncompressed size of the data was 12GB, which compressed to 1.7GB, yielding a compression ratio 7:1 (the card currently has a fixed compression level).
The time taken to compress the data was 65077mS with the AHA card enabled versus 552506mS using zlib compression in pure software. i.e. using the AHA card took 12% of the time to compress the same amount of data to the same level, achieving approx. a 10x speed up, using just one channel only. For those wishing to execute file compression in parallel using the peach command, all 4 channels on the card can be used.
The AHA zlib shared lib can be run in 3 modes - compression+decompression, compression only or decompression only. With kdb+ using just a single channel of the card, the decompression performance of the card was slightly slower than as in software, although when kdb+ was used in a multithreaded mode, increased overall performance was observed due to all 4 channels being used thereby freeing up the main cpu.
AHA also offer other cards, the AHA360PCIe (1 channel) and the AHA363PCIe (2 channel), however these have not yet been tested for compatibility.
Installation is very straightforward - unpack and plug-in the card, compile and load the driver, compile and install the zlib shared library. As a reference, it took less than 30 minutes from opening the box to having kdb+ use it for compression. A very smooth installation.
Runtime troubleshooting for the AHA 367 card
If you see the error message
aha367 - ahagz_api.c: open() call failed with error: 2 on device /dev/aha367_board
this likely means the kernel module has not been loaded. Remedy - go to the <aha install dir>/bin and execute sudo ./load_module and select the 367 card option.
Another accelerator card vendor (untested) - http://www.indranetworks.com/SCMX3.html
Q) Do I need to tweak the kernel settings in order to work with compressed files?
A) On linux, perhaps - it really depends on the size and number of compressed files you have open at any time, and the access patterns used. For example, random access to a compressed file will use many more kernel resources than sequential access.
See Cookbook/Production_Notes#Compression for further information.
Q) Can I run kdb+ under gdb (the gnu debugger), and use compressed files?
A) You should only ever need to run gdb if you are debugging your own custom shared libs loaded into kdb+.
gdb will intercept SIGSEGV which should be passed to kdb+; to tell it to do so, issue the following command at the gdb prompt
(gdb) handle SIGSEGV nostop noprint
Q) I use kdb+ with R Server for Q, i.e. R as shared library loaded into kdb+. Does this work with compressed files?
A) R uses signal handling to detect stack overflows. This conflicts with kdb+'s use of signals for handling compressed files, causing kdb+ to crash. R's use of signals can be suppressed by setting the variable R_SignalHandlers (declared in Rinterface.h) to 0 when compiling the relevant R library.
Q) I'd like to use Transparent Huge Pages (THP) on Linux. Is this compatible with the kdb+ file compression?
A) No. We have had reports of systems stalling due to THP fragmentation. Disable THP either with the shell command
echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled
or more permanently via grub at boot time
n.b. kdb+ must be restarted to pick up the new setting.