SSL/TLS¶

Since V3.4t 2016.05.12, kdb+ can use Secure Sockets Layer (SSL)/Transport Layer Security (TLS) to encrypt connections using the OpenSSL libraries.

Configuration¶

OpenSSL library¶

Ensure that your OS has the latest OpenSSL libraries installed, and that they are in your LD_LIBRARY_PATH (Unix), DYLD_LIBRARY_PATH (MacOS), or PATH (Windows).

OpenSSL version¶

Beginning with v4.1t 2022.03.25, kdb+ will try to load versioned shared libraries for OpenSSL. It will load the first library that it can locate from the lists below:

Linux macOS Windows

libssl.so libssl.so.3 libssl.1.1 libssl.so.1.0 libssl.so.10 libssl.so.1.0.2 libssl.so.1.0.1 libssl.1.0.0

libssl.3.dylib libssl.1.1.dylib

both libssl and libcrypto are loaded, the library names in priority order are

	libssl	libcrypto
`w64`	`libssl-3-x64.dll` `libssl-1_1-x64.dll`	`libcrypto-3-x64.dll` `libcrypto-1_1.dll`
`w32`	`libssl-3.dll` `libssl-1_1.dll`	`libcrypto-3.dll` `libcrypto-1_1.dll`

The Windows build was tested with the pre-compiled libs Win32 OpenSSL v1.1.1L Light, Win64 OpenSSL v1.1.1L Light.

Prior to V4.1t 2022.03.25, kdb+ would load the following files:

Linux, Solaris macOS Windows

libssl.so

libssl.dylib

libssl-1_1-x64.dll   libcrypto-1_1-x64.dll    w64 build
libssl-1_1.dll       libcrypto-1_1.dll        w32 build

Keys/Certificates¶

Since TLS uses certificates, prior to enabling TLS in a kdb+ server, ensure that you have the necessary certificates in place.

The minimum for a TLS-enabled server is to provide a certificate and its associated key, both in PEM format.

To locate these files, q uses the default path as reported by the openssl version -d command as a base, e.g.

$ openssl version -d
OPENSSLDIR: "/opt/local/etc/openssl"

Configuration of keys/certificates and checks performed is controlled by the following environment variables

KX first

Since V3.6, kdb+ gives preference to the KX_ prefix for the SSL_* environment variables to avoid clashes with other OpenSSL based products.

For example, the value for

getenv `KX_SSL_CERT_FILE

has a higher precedence than

getenv `SSL_CERT_FILE

for determining config.

SSL_CERT_FILE¶

The certificate file. This must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root).

Default value is <OPENSSLDIR>/server-crt.pem

SSL_CA_CERT_FILE¶

A file containing certificate authority (CA) certificates in PEM format. The file can contain several CA certificates identified by

-----BEGIN CERTIFICATE-----
... (CA certificate in base64 encoding) ...
-----END CERTIFICATE-----

sequences. Text is allowed before, between, and after the certificates; it can be used, for example, for descriptions of the certificates.

Default value is <OPENSSLDIR>/cacert.pem

SSL_CA_CERT_PATH¶

A directory containing certificate authority (CA) certificates in PEM format.

Default value is <OPENSSLDIR>

SSL_KEY_FILE¶

Private key in PEM format.

Default value is <OPENSSLDIR>/server-key.pem

SSL_CIPHER_LIST¶

The default cipher list is set to the Intermediate compatibility (default) as recommended by Mozilla.org. You may override this, to reduce the list to whatever your IT security policy requires.

Default value is

ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-
AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-
SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-
AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:
ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:
ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:
DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-
RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-
RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:
AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS

A good source for what is generally recommended can be found at Ciphersuites.

If you select a set which is not compatible with the peer process, you’ll observe a message at the q console similar to the following.

140735201689680:error:1408A0C1:SSL routines:ssl3_get_client_hello:no
shared cipher:s3_srvr.c:1417:

SSL_VERIFY_CLIENT¶

Controls the processing of certificates from a client. Uses the certificates from SSL_CA_CERT_FILE or SSL_CA_CERT_PATH to verify the client’s certificate.

Can be set to one of the following values:

NO (default) kdb+ does not request nor validate the certificate from a client
YES server requests a client certificate and disconnects the client if the provided certificate is missing or invalid
REQUESTONLY (since 4.1t 2024.02.07) server requests a client certificate but allows the connection if the client certificate is missing or invalid
IFPRESENT (since 4.1t 2024.02.07) server requests a client certificate and terminates the connection if an invalid certificate is provided; the server continues if the certificate is missing or valid

SSL_VERIFY_SERVER¶

Controls the processing of certificates from a server. Default value is YES.

Checks the X509 certificate the peer presented by verifying the server’s certificate against a trusted source, using the certificates from SSL_CA_CERT_FILE or SSL_CA_CERT_PATH to verify the server’s certificate. Expired certificates will fail the verification process.

If the verification process fails or no server certificate provided, the TLS/SSL handshake is immediately terminated with an alert message containing the reason for the verification failure.

Setting to NO does not terminate the connection due to a failure verifying the certificate.

Checking Configuration¶

Configured TLS settings for a kdb+ process can be viewed with (-26!)[].

Certificates¶

If you don’t have a certificate, you can create a self-signed certificate using the openssl program. An example script (makeCerts.sh) to do so follows; customize as necessary.

mkdir $HOME/certs && cd $HOME/certs

# create private key for CA (certificate authority)
openssl genrsa -out ca-private-key.pem 2048
# create X509 certificate for CA (certificate authority)
openssl req -x509 -new -nodes -key ca-private-key.pem -sha256 -days 365 -out ca-cert.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=FoobarCA

# create server private key
openssl genrsa -out server-private-key.pem 2048
# create servers certificate signing request (CSR)
# CSR contains the common name(s) you want your certificate to secure, information about your company, and your public key (taken from provided private key)
openssl req -new -sha256 -key server-private-key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out server.csr
# create X509 certificate for the server (signed by CA)
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-private-key.pem -CAcreateserial -out server-cert.pem -days 365 -sha256

# create client private key
openssl genrsa -out client-private-key.pem 2048
# create clients certificate signing request (CSR)
# CSR contains the common name(s) you want your certificate to secure, information about your company, and your public key (taken from provided private key)
openssl req -new -sha256 -key client-private-key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out client.csr
# create X509 certificate for the client (signed by CA)
openssl x509 -req -in client.csr -CA ca-cert.pem -CAkey ca-private-key.pem -CAcreateserial -out client-cert.pem -days 365 -sha256

Using this script the server settings can be configured as:

$ export SSL_CERT_FILE=$HOME/certs/server-cert.pem
$ export SSL_KEY_FILE=$HOME/certs/server-private-key.pem
$ export SSL_CA_CERT_FILE=$HOME/certs/ca-cert.pem

with the client as:

$ export SSL_CERT_FILE=$HOME/certs/client-cert.pem 
$ export SSL_KEY_FILE=$HOME/certs/client-private-key.pem
$ export SSL_CA_CERT_FILE=$HOME/certs/ca-cert.pem

mkcert is a simple tool for making locally-trusted development certificates.

Secure your certificates

Store your certificates outside of the directories accessible from within kdb+, otherwise remote users can easily steal your server’s key file!

TLS Server Mode¶

When the certificates are in place, and the environment variables are set, TLS Server Mode can be enabled through the command-line option -E.

TLS Client Mode¶

TLS connections can be opened to TLS-enabled servers with

q)h:hopen`:tcps://hostname:port[:username:password]

Clients can also request secure HTTP (HTTPS) and WebSockets (WSS) connections.

If you don't wish to verify a server’s certificate, set

$ export SSL_VERIFY_SERVER=NO

To allow verification of certificates which were not issued by you, you can import the CA bundle from reputable sources, e.g.

$ curl https://curl.se/ca/cacert.pem > $HOME/certs/cabundle.pem
$ export SSL_CA_CERT_FILE=$HOME/certs/cabundle.pem

If you open the downloaded cabundle.pem with a text editor you’ll see a list of certificates, and you can append your own self-signed ca.pem to this file if you wish.

If there is an issue in loading the CA certificate, an error similar to the following will be printed at the q console

q).Q.hg`$":https://www.kx.com"
140735201689680:error:02001002:system library:fopen:No such file or directory:bss_file.c:175:fopen('/opt/local/etc/openssl/cacert.pem','r')
140735201689680:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:178:
140735201689680:error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib:by_file.c:253:
'conn. OS reports: Protocol not available

Connection information¶

Extra protocol details for a connection handle are available via .z.e, including information about whether the current handle's TLS certificate was successfully verified.

Suitability and restrictions¶

Currently we would recommend TLS be considered only for long-standing, latency-insensitive, low-throughput connections. The overhead of hopen on localhost appears to be 40-50× that of a plain connection, and once handshaking is complete, the overhead is ~1.5× assuming your OpenSSL library can utilize AES-NI.

OpenSSL 1.1 is supported since V4.0 2020.03.17.

Thread Support¶

The following associated features are not implemented for TLS

multithreaded input mode
use within secondary threads
hopen timeout (implemented in V3.5)

Since 4.1t 2023.11.10 this was altered to allow use on any thread where messaging was previously supported: i.e.

incoming connections in multithreaded input queue mode (mtiqm)
one-shot sync requests within peach or mtiqm socket thread
https client requests within peach or mtiqm socket thread