Skip to content

Compression and Encryption APIs

Functionality for the setting of compression and encryption configuration when handling on-disk data.

Warning

This functionality is provided in it's present form as a BETA Feature and is subject to change. To enable this functionality for testing please following configuration instructions here setting PYKX_BETA_FEATURES='true'

CompressionAlgorithm

Bases: Enum

The compression algorithm to be used when compressing a DB partition/column.

Presently the supported algorithms are qipc, gzip, snappy, lz4hc and zstd. These algorithms support different compression levels denoting the agressivness of compression in each case.

algorithm levels
none 0
q IPC 0
gzip 0-9
snappy 0
lz4hc 0-16
zstd -7-22

Encrypt 

Encrypt(path=None, password=None)

Initialize a class object which is used to control the use of encryption with PyKX.

Parameters:

Name Type Description Default
path

Location of a users encryption key file as an 'str' object

 None
password

Password which had been set for encryption file

 None
Example 
>>> import pykx as kx
>>> encrypt = kx.Encrypt('/path/to/mykey.key', 'mySuperSecretPassword')

load_key 

load_key()
Example 
>>> import pykx as kx
>>> encrypt = kx.Encrypt('/path/to/mykey.key', 'mySuperSecretPassword')
>>> encrypt.load_key()

Load the encyption key within your process, note this will be a global load.

Compress 

Compress(algo=CompressionAlgorithm.none, block_size=2**17, level=None)

Initialize a class object which is used to control encryption within PyKX.

Parameters:

Name Type Description Default
algo

Compression algorithm to be used when applying compression, this must be one of:

  • kx.CompressionAlgorithm.none
  • kx.CompressionAlgorithm.ipc
  • kx.CompressionAlgorithm.gzip
  • kx.CompressionAlgorithm.snappy
  • kx.CompressionAlgorithm.lz4hc
 CompressionAlgorithm.none
block_size

Must be a port of 2 between 12 and 20 denoting the pageSize or allocation granularity to 1MB, see here for more information.

 2**17
level

The degree to which compression will be applied, when non zero values are supported for a supported algorithm larger values will result in higher compression ratios.

 None
Example 
>>> import pykx as kx
>>> comp = kx.Compress(kx.CompressionAlgorithm.gzip, level=5)

global_init 

global_init(encrypt=False)

Globally initialise compression settings, when completed any persistence operation making use of kx.q.set will be compressed based on the user specified compression settings

Parameters:

Name Type Description Default
encrypt

A kx.Encrypt object denoting if and using what credentials encryption is to be applied.

 False
Example 
>>> import pykx as kx
>>> comp = kx.Compress(kx.CompressionAlgorithm.gzip, level=2)
>>> kx.q.z.zd
pykx.Identity(pykx.q('::'))
>>> comp.global_init()
>>> kx.q.z.zd
pykx.LongVector(pykx.q('17 2 2'))