Skip to content

Querying over QIPC

Prerequisites

  • You have access to a running version of kdb Insights Enterprise. For more information, refer to the installation guide.
  • You have installed the kdb Insights Command Line (KXI CLI) following the installation guide.

Authentication and Authorization

Acquire an access token

First, acquire an INSIGHTS_TOKEN from a client that has the insights.query.* roles:

You can create a new user to do this with the KXI CLI:

Administrator password

The administrator password mentioned below is defined during the installation of your kdb Insights Enterprise deployment. For more information, refer to the Administration Passwords documentation.

kxi user create john@kx.com --password mySecretPassword123! --admin-password <adminPassword> --not-temporary
INSIGHTS_ROLES="insights.client.create,insights.query.*"
kxi user assign-roles john@kx.com --roles $INSIGHTS_ROLES --admin-password <adminPassword> 
export INSIGHTS_TOKEN=$(kxi auth print-token)

Acquire the endpoint by enrolling a client

Step 1 - Enroll a client

kxi client enrol --name query-client --insert-topic ext-taxi
The following response is returned:

{
  "message": "success",
  "detail": "Client enrolled",
  "url": "5ed6e5b7c80c8e35d07249d12f32d9eb",
  "config_url": "https://my-insights.kx.com/informationservice/details/5ed6e5b7c80c8e35d07249d12f32d9eb"
}

Step 2 - Curl the authenticated kdb Insights Enterprise client URL (KXI_CONFIG_URL)

Copy the value returned for the config_url key returned when the client is enrolled and run curl against it:

KXI_CONFIG_URL=https://my-insights.kx.com/informationservice/details/5ed6e5b7c80c8e35d07249d12f32d9eb
curl $KXI_CONFIG_URL | jq
{
  "name": "query-client",
  "topics": {
    "insert": "ext-taxi",
    "query": "requests"
  },
  "ca": "-----BEGIN CERTIFICATE-----\nMIID .. ==\n-----END CERTIFICATE-----\n",
  "cert": "-----BEGIN CERTIFICATE-----\nMII ... ==\n-----END CERTIFICATE-----\n",
  "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEp ... ==\n-----END CERTIFICATE-----\n",
  "insert": {
    "insert": [
      ":hostname-0:5000",
      ":hostname-1:5000",
      ":hostname-2:5000"
    ],
    "query": []
  },
  "query": [
    "k8s-insights-875d1037f3.elb.us-east-2.amazonaws.com:6050"
  ]
}

Step 3 - Define your query endpoint

export INSIGHTS_QUERY=k8s-insights-875d1037f3.elb.us-east-2.amazonaws.com:6050

Step 4 - Define your SSL certificates

You can use Secure Sockets Layer (SSL)/Transport Layer Security (TLS) to encrypt connections using the OpenSSL libraries. You can define your certificates using the code below. You can find more information on securing this traffic on the kdb+ and TLS page.

PATH_TO_CERTS=/tmp/certs
mkdir -p /tmp/certs
curl $KXI_CONFIG_URL | jq .ca | xargs printf > $PATH_TO_CERTS/ca.pem
export KX_SSL_CA_CERT_FILE=$PATH_TO_CERTS/ca.pem

Query

Use the username token and the INSIGHTS_TOKEN as the tcps password, with the hostname and port of the load balancer:

h:hopen `$":tcps://",getenv[`INSIGHTS_QUERY],":token:",getenv `INSIGHTS_TOKEN;
h (`.kxi.getMeta;()!();`;()!())

Considerations

  • Access tokens can expire, and using this with PyKX/q probably means we should get the tokens with .kurl and maintain them...
  • The Load Balancers for AWS will close h due to inactivity at a mere 30s, by default. You need to keep making requests to use this OR update the AWS settings. See https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html
  • There is no readiness check for SSL currently, so its possible to query the GW before the SSL port is up. You will get 'hop. OS reports: Protocol error.