Skip to content

Querying over QIPC

This page explains how to configure and query external IPC in a kdb Insights Enterprise database environment.

QIPC (kdb+/q Inter-Process Communication) is a mechanism that allows processes running on different machines or on the same machine to communicate with each other using the kdb+/q database system.

In the context of kdb+/q, IPC refers to the exchange of data between multiple processes or instances of the q language, such as when querying or transferring data between a server and client or between different services in a distributed environment.

QIPC involves querying over TCP/IP sockets. For more information, refer to the IPC documentation.

Prerequisites

  • You have access to a running version of kdb Insights Enterprise. For more information, refer to the installation guide.
  • You have installed the kdb Insights Command Line Interface (KXI CLI) following the installation guide.

Authentication and authorization

Acquire an access token

First, acquire an INSIGHTS_TOKEN from a client that has the insights.query.* roles.

You can create a new user to do this using the KXI CLI, as follows:

kxi user create john@kx.com --password mySecretPassword123! --admin-password <adminPassword> --not-temporary
INSIGHTS_ROLES="insights.client.create,insights.query.*"
kxi user assign-roles john@kx.com --roles $INSIGHTS_ROLES --admin-password <adminPassword> 
export INSIGHTS_TOKEN=$(kxi auth print-token)

Administrator password

The administrator password mentioned below is defined during the installation of your kdb Insights Enterprise deployment. For more information, refer to the Administration Passwords documentation.

Acquire the endpoint

Follow the steps below to acquire the endpoint by enrolling a client.

Step 1 - Enroll a client

Run the following command to enroll a client:

kxi client enrol --name query-client --insert-topic ext-taxi
This returns the following response:

{
  "message": "success",
  "detail": "Client enrolled",
  "url": "5ed6e5b7c80c8e35d07249d12f32d9eb",
  "config_url": "https://my-insights.kx.com/informationservice/details/5ed6e5b7c80c8e35d07249d12f32d9eb"
}

Step 2 - Curl the authenticated kdb Insights Enterprise client URL (KXI_CONFIG_URL)

Copy the value returned for the config_url key after the client is enrolled, and then run curl to make a request to that URL:

KXI_CONFIG_URL=https://my-insights.kx.com/informationservice/details/5ed6e5b7c80c8e35d07249d12f32d9eb
curl $KXI_CONFIG_URL | jq
{
  "name": "query-client",
  "topics": {
    "insert": "ext-taxi",
    "query": "requests"
  },
  "ca": "-----BEGIN CERTIFICATE-----\nMIID .. ==\n-----END CERTIFICATE-----\n",
  "cert": "-----BEGIN CERTIFICATE-----\nMII ... ==\n-----END CERTIFICATE-----\n",
  "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEp ... ==\n-----END CERTIFICATE-----\n",
  "insert": {
    "insert": [
      ":hostname-0:5000",
      ":hostname-1:5000",
      ":hostname-2:5000"
    ],
    "query": []
  },
  "query": [
    "k8s-insights-875d1037f3.elb.us-east-2.amazonaws.com:6050"
  ]
}

Step 3 - Define your query endpoint

Run the following command to define your query endpoint:

export INSIGHTS_QUERY=k8s-insights-875d1037f3.elb.us-east-2.amazonaws.com:6050

Step 4 - Define your SSL certificates

You can use Secure Sockets Layer (SSL)/Transport Layer Security (TLS) to encrypt connections using the OpenSSL libraries.

You can define your certificates using the code below. Refer to the kdb+ and TLS page for more information on securing this traffic.

PATH_TO_CERTS=/tmp/certs
mkdir -p /tmp/certs
curl $KXI_CONFIG_URL | jq .ca | xargs printf > $PATH_TO_CERTS/ca.pem
export KX_SSL_CA_CERT_FILE=$PATH_TO_CERTS/ca.pem

Query

Use the username token and the INSIGHTS_TOKEN as the tcps password, with the hostname and port of the load balancer, as follows:

h:hopen `$":tcps://",getenv[`INSIGHTS_QUERY],":token:",getenv `INSIGHTS_TOKEN;
h (`.kxi.getMeta;()!();`;()!())

Considerations

  • Access tokens can expire: When using PyKX/q, it's recommended to retrieve tokens with .kurl and manage their lifecycle to ensure continuous authentication.

  • AWS Load Balancers idle timeout: By default, AWS Load Balancers terminate connections after 30 seconds of inactivity. To avoid this, either ensure regular requests are made to keep the connection alive or update the idle timeout settings in AWS. For more details, refer to the AWS documentation on idle timeout.