SSL/TLS¶

V3.4t 2016.05.12 can use Secure Sockets Layer (SSL)/Transport Layer Security (TLS) to encrypt connections using the OpenSSL libraries.

Prerequisites¶

Ensure that your OS has the latest OpenSSL libraries installed, and that they are in your LD_LIBRARY_PATH (Unix), or PATH (Windows).

Versioned libraries¶

Beginning with v4.1t 2022.03.25, Kdb+ will try to load versioned shared libraries for OpenSSL. It will load the first library that it can locate from the lists below:

Linux macOS Windows

libssl.so libssl.so.3 libssl.1.1 libssl.so.1.0 libssl.so.10 libssl.so.1.0.2 libssl.so.1.0.1 libssl.1.0.0

libssl.3.dylib libssl.1.1.dylib

both libssl and libcrypto are loaded, the library names in priority order are

	libssl	libcrypto
`w64`	`libssl-3-x64.dll` `libssl-1_1-x64.dll`	`libcrypto-3-x64.dll` `libcrypto-1_1.dll`
`w32`	`libssl-3.dll` `libssl-1_1.dll`	`libcrypto-3.dll` `libcrypto-1_1.dll`

The Windows build was tested with the pre-compiled libs Win32 OpenSSL v1.1.1L Light, Win64 OpenSSL v1.1.1L Light.

Prior to V4.1t 2022.03.25, kdb+ would load the following files:

Linux, Solaris macOS Windows

libssl.so

libssl.dylib

libssl-1_1-x64.dll   libcrypto-1_1-x64.dll    w64 build
libssl-1_1.dll       libcrypto-1_1.dll        w32 build

Certificates¶

Since TLS uses certificates, prior to enabling TLS in a kdb+ server, ensure that you have the necessary certificates in place. The minimum for a TLS-enabled server is to provide a certificate and its associated key, both in PEM format. To locate these files, q will use the default path as reported by the openssl version -d command as a base, e.g.

$ openssl version -d
OPENSSLDIR: "/opt/local/etc/openssl"

This default can be overridden by setting the environment variables SSL_CERT_FILE and SSL_KEY_FILE to the full path to your certificate and key files. e.g.

$ export SSL_CERT_FILE=$HOME/certs/server-crt.pem
$ export SSL_KEY_FILE=$HOME/certs/server-key.pem

KX first

Since V3.6, kdb+ gives preference to the KX_ prefix for the SSL_* environment variables to avoid clashes with other OpenSSL based products.

For example, the value for getenv`KX_SSL_CERT_FILE has a higher precedence than getenv`SSL_CERT_FILE for determining config.

If you don’t have a certificate, you can create a self-signed certificate using the openssl program. An example script to do so follows, which you should customize as necessary

$ more makeCerts.sh
mkdir -f $HOME/certs && cd $HOME/certs

# Create CA certificate
openssl genrsa 2048 > ca-key.pem
openssl req -new -x509 -nodes -days 3600 \
    -key ca-key.pem -out ca.pem -extensions usr_cert \
    -subj '/C=US/ST=New York/L=Brooklyn/O=Example Brooklyn Company/CN=examplebrooklyn.com'

# Create server certificate, remove passphrase, and sign it
# server-crt.pem = public key, server-key.pem = private key
openssl req -newkey rsa:2048 -days 3600 -nodes \
    -keyout server-key.pem -out server-req.pem -extensions usr_cert \
    -subj '/C=US/ST=New York/L=Brooklyn/O=Example Brooklyn Company/CN=myname.com'
openssl rsa -in server-key.pem -out server-key.pem
openssl x509 -req -in server-req.pem -days 3600 -CA ca.pem -CAkey ca-key.pem \
    -set_serial 01 -out server-crt.pem -extensions usr_cert

# Create client certificate, remove passphrase, and sign it
# client-crt.pem = public key, client-key.pem = private key
openssl req -newkey rsa:2048 -days 3600  -nodes \
    -keyout client-key.pem -out client-req.pem -extensions usr_cert \
    -subj '/C=US/ST=New York/L=Brooklyn/O=Example Brooklyn Company/CN=myname.com'
openssl rsa -in client-key.pem -out client-key.pem
openssl x509 -req -in client-req.pem -days 3600 -CA ca.pem -CAkey ca-key.pem \
    -set_serial 01 -out client-crt.pem -extensions usr_cert

mkcert is a simple tool for making locally-trusted development certificates.

Secure your certificates

Store your certificates outside of the directories accessible from within kdb+, otherwise remote users can easily steal your server’s key file!

TLS cipher List¶

The default cipher list is set to the Intermediate compatibility (default) as recommended by Mozilla.org, and you may override this via the environment variable SSL_CIPHER_LIST, to reduce the list to whatever your IT security policy requires. A good source for what is generally recommended can be found at Ciphersuites.

e.g.

$ export SSL_CIPHER_LIST='ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY
1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES2
56-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES
256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AE
S128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384
:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES1
28-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-
RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES12
8-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'

If you select a set which is not compatible with the peer process, you’ll observe a message at the q console similar to the following.

140735201689680:error:1408A0C1:SSL routines:ssl3_get_client_hello:no 
shared cipher:s3_srvr.c:1417:

TLS Server Mode¶

Once the certificates are in place, and the environment variables set, TLS Server Mode can be enabled through the command-line option -E 0 (plain), 1 (plain & TLS), 2 (TLS only). e.g.

$ q -u 1 -E 1 -p 5000

starts a server which will listen for plain and TLS connections on port 5000, restricting remote access to the pwd and below.

General TLS settings for a kdb+ process can be viewed with (-26!)[].

q)(-26!)[]
SSLEAY_VERSION   | OpenSSL 1.0.2g  1 Mar 2016
SSL_CERT_FILE    | /Users/kdb/certs/server-crt.pem
SSL_CA_CERT_FILE | /Users/kdb/certs/ca.pem
SSL_CA_CERT_PATH | /Users/kdb/certs/
SSL_KEY_FILE     | /Users/kdb/certs/server-key.pem
SSL_CIPHER_LIST  | ALL
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| YES

All keys except SSLEAY_VERSION in the result from (-26!)[] are initialized from environment variables.

The environment variable SSL_VERIFY_CLIENT controls the processing of certificates from a client, and uses the certificates from SSL_CA_CERT_FILE or SSL_CA_CERT_PATH to verify the client’s certificate. Can be set to one of the following values:

NO (default) kdb+ does not request nor validate the certificate from a client
YES server requests a client certificate and disconnects the client if the provided certificate is missing or invalid
REQUESTONLY (since 4.1t 2024.02.07) server requests a client certificate but allows the connection if the client certificate is missing or invalid
IFPRESENT (since 4.1t 2024.02.07) server requests a client certificate and terminates the connection if an invalid certificate is provided; the server continues if the certificate is missing or valid

Extra protocol details for a handle h are available via .z.e, including information about whether the current handle's TLS certificate was successfully verified.

TLS Client Mode¶

TLS client mode is always enabled, and TLS Connections can be opened to TLS-enabled servers with

q)h:hopen`:tcps://hostname:port[:username:password]

Clients can also request secure HTTP (HTTPS) and WebSockets (WSS) connections via

q)(`$":https://127.0.0.1:5000")"GET /login.html http/1.1\r\nhost:www.kx.com\r\n\r\n"
and for websockets
q) r:(`$":wss://127.0.0.1:5000")"GET / HTTP/1.1\r\nHost: 127.0.0.1:5000\r\n\r\n"

By default, kdb+ will try to verify the server’s certificate against a trusted source, using the certificates from SSL_CA_CERT_FILE or SSL_CA_CERT_PATH to verify the server’s certificate. If you don't wish to verify a server’s certificate, set

$ export SSL_VERIFY_SERVER=NO

To allow verification of certificates which were not issued by you, you can import the CA bundle from reputable sources, e.g.

$ curl https://curl.se/ca/cacert.pem > $HOME/certs/cabundle.pem
$ export SSL_CA_CERT_FILE=$HOME/certs/cabundle.pem

If you open the downloaded cabundle.pem with a text editor you’ll see a list of certificates, and you can append your own self-signed ca.pem to this file if you wish.

In the interests of not interrupting service, verification of certificates accepts expired certificates.

If there is an issue in loading the CA certificate, an error similar to the following will be printed at the q console

q).Q.hg`$":https://www.kx.com"
140735201689680:error:02001002:system library:fopen:No such file or directory:bss_file.c:175:fopen('/opt/local/etc/openssl/cacert.pem','r')
140735201689680:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:178:
140735201689680:error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib:by_file.c:253:
'conn. OS reports: Protocol not available

Testing your client configuration¶

You can test your client configuration with

q)(`$":howsmyssl.html")0:enlist .Q.hg`$":https://www.howsmyssl.com";

And then open the resulting file with your browser, e.g. on macOS use the open command

q)\open howsmyssl.html

Unix Domain Socket¶

When TLS mode enabled, any Unix Domain Socket (UDS) connection wishing to use TLS must use the handle :unixs:// (rather than :unix://) to encrypt its communication

q)h:hopen`:unixs://12345

Suitability and restrictions¶

Currently we would recommend TLS be considered only for long-standing, latency-insensitive, low-throughput connections. The overhead of hopen on localhost appears to be 40-50× that of a plain connection, and once handshaking is complete, the overhead is ~1.5× assuming your OpenSSL library can utilize AES-NI.

OpenSSL 1.1 is supported since V4.0 2020.03.17.

Thread Support¶

The following associated features are not implemented for TLS

multithreaded input mode
use within secondary threads
hopen timeout (implemented in V3.5)

Since 4.1t 2023.11.10 this was altered to allow use on any thread where messaging was previously supported: i.e.

incoming connections in multithreaded input queue mode (mtiqm)
one-shot sync requests within peach or mtiqm socket thread
https client requests within peach or mtiqm socket thread