Skip to content

KX Insights Installation

The section details the steps to deploy the KX Insights Platform in a pre-configured Kubernetes Cluster. It will walk through

Once installed the user will be able to deploy workloads which can be designed and developed to meet their specific needs and requirements.

For users wishing to upgrade their KX Insights deploy, refer to the upgrade guide

Prerequisites

Cluster and shared services

Before installing the KX Insights Platform it is necessary to have an appropriate Kubernetes cluster in place with the necessary shared services available.

These include

  • Virtual network, public, private and bastian resources
  • Certificate manager
  • Ingress controller
  • Persistent storage resources

The KX Insights Platform provides a number of Terraform scripts which enable the quick deployment of the necessary cluster services on all 3 main cloud providers. More details here.

KX Insights namespaces

The KX Insights Platform can be deployed to the default namespace or a uniquely created namespace. However two deployments of KX Insights cannot be on the same namespace. When deploying KX Insights you should check the active namespace to ensure there are no current deployments.

helm ls

This will return any instances of the KX Insights Platform currently deployed to the active namespace. Further details on Kubernetes namespaces and contexts are available here.

The Kxi Operator is deployed to its own namespace kxi-operator and requires only to be installed once per cluster. When deployed, it services all deployments of the KX Insights Platform and reconciles all Assembly workloads deployed to the appropriate instance of the KX Insights Platform based on the active namespace of the client who deployed the assembly workload.

DNS

In order for external clients to interact with the KX Insights Platform, a DNS record must be setup with your DNS provider. For examples on setting up DNS records with the cloud providers, please see here.

Pre-install checklist

  1. Access to your cluster and Kubernetes tooling available

    The user must have access to the Kubernetes cluster along with a number of third party tools to interact with the cluster. Details here

  2. The KXI command line interface installed

    Having the KXI CLI installed simplifies install and will be used in the following instructions. Details here.

  3. KX Insights compatible license

    The install process requires a KX Insights license. To obtain a valid KX Insights license, speak to your KX Sales representative.

    KX Insights provides the ability to self-generate licenses. For information on this, please refer to the documentation here.

  4. Access to the appropriate helm/docker registries for charts and images

    The install will prompt for repository locations and access details

The KXI CLI provides an interactive installer that can be used to deploy the KX Insights Platform on a Kubernetes cluster. It will prompt and walk the user through the steps to install the KX Insights Platform. To ensure a smooth process, it is recommended to have a number of variables to hand

Variable Example Further Details
INSIGHTS_HOSTNAME kxi-insights.domain.com DNS Hostname setup
HELM_REPO https://nexus.dl.kx.com/repository/kx-insights-charts (Optional) Helm repo details
DOCKER_REPO registry.dl.kx.com (Optional) Docker repo details
ACCESS_USER my-user Helm/Docker repo details
ACCESS_PASS my-pass Helm/Docker repo details
KX_LIC_PATH /home/app/q/kc.lic Path to KX License
CA_CERT_PATH /home/app/certs/ca.crt (Optional) Path to ca.crt file
CA_KEY_PATH /home/app/certs/ca.crt (Optional) Path to ca.key file
INSIGHTS_VERSION 1.1.1 Version of insights to install
OPERATOR_VERSION 1.1.0 Version of insights operator to install

Installing the KX Insights Platform

To install the KX Insights Platform two commands are run; the first kxi install setup will generate a values.yaml config file for us to use for repeatable future installs; followed by a kxi install run --filepath values.yaml command providing the values config file as an argument to deploy the KX Insights Platform to the cluster.

Note

It is possible to combine these two commands in the one kxi install run, but to ensure clarity, this walkthrough separates them.

All options prompted to the user can alternativly be supplied by command line arguments or cli configuration. See the options precedence for a description of the order in which these are read.

Run setup
kxi install setup
Hostname

Provide the hostname defined by $INSIGHTS_HOSTNAME DNS record as discussed above

Checking ingress configuration...
Please enter the hostname for the installation: kxi-insights.example.com
Using Let's Encrypt TLS certificates
Ingress certification

The KX Insights Platform requires ingress to be enabled in order to route external client requests into the application via UI or REST API. There are two supported modes; auto-generated (default) or self-managed.

Auto-generated

In this mode the KX Insights Platform will use cert manager to issue certificates via Let's Encrypt. These certs will be automatically generated and be used by the ingress controller to secure traffic. This is the default mode.

Self-managed

In some cases it may be preferable to self-manage certificates, especially if there is a preference to use a cert authority not supported by cert manager, e.g. DigiCert. This requires additional setup and maintenance as the certificates aren't automatically generated. Guidance on manually creating these files and secret can be found in the Manual Install section.

To use self-managed certs, the --ingress-cert and --ingress-key options may be passed to kxi install setup as follows:

kxi install setup --ingress-cert <CA_CRT_PATH> --ingress-key <CA_KEY_PATH>
...
Checking ingress configuration...
Using ingress.cert <CA_CRT_PATH> from command line option
Using ingress.key <CA_KEY_PATH> from command line option
Secret kxi-ingress-cert successfully created

Renewals

Certificate renewals need to be handled manually. The command to create the secret above should be repeated with the existing ca.key and a newly issued ca.crt.

Chart Repository details

If installing from repositories, the defaults can be used along with your KX-provided credentials. If installing from a local repository, please submit the required details.

Checking chart details...
Please enter a name for the chart repository to set locally [kx-insights]: 
Please enter the chart repository URL to pull charts from [https://nexus.dl.kx.com/repository/kx-insights-charts]: 
Please enter the username for the chart repository: <ACCESS_USER>
Please enter the password for the chart repository (input hidden): <ACCESS_PASS>
Re-enter to confirm (input hidden): <ACCESS_PASS>
License information

If an existing license secret exists it will be automatically used. Otherwise, provide the $KX_LIC_PATH to your license file to generate the secret on your cluster

Checking license details...
Please enter the path and filename of your kdb+ license: <KX_LIC_PATH>
Secret kxi-license successfully created
Image Repository details

Defaults may be used if utilizing KX repositories along with access details provided by KX. Or if downloaded to an alternative location, provide those details at this point.

Image repository
Please enter the image repository to pull images from [registry.dl.kx.com]: 
Please enter the username for registry.dl.kx.com: <ACCESS_USER>
Please enter the password for <ACCESS_USER> (input hidden): <ACCESS_PASS>
Re-enter to confirm (input hidden): <ACCESS_PASS>
Secret kxi-nexus-pull-secret successfully created
Client certification

The KX Insights Platform uses mutual TLS for clients publishing data into the system. This requires the system to be configured with a certificate issuer to generate unique keys and certs for each client. The installer will create this certificate issuer when one does not already exist. Guidance on manually creating these files and secret can be found in the Manual Install section.

Checking client certificate issuer...
Secret kxi-certificate successfully created
Keycloak Authentication

Keycloak is used as the identity and access management platform for the KX Insights Platform.

In the base install of the KX Insights Platform, there are four passwords that are required.

Password Description
admin-password This is the password for the Keycloak Admin UI
management-password This is the password for the WildFly management user
postgresql-postgres-password This is the password for the database superuser
postgresql-password This is the password for the database user that Keycloak uses

The installer will prompt you for these four passwords unless you have an existing secret that contains them.

Checking keycloak configuration...
Please enter the Keycloak Admin password (input hidden): 
Re-enter to confirm (input hidden): 
Please enter the Keycloak Admin password (input hidden): 
Re-enter to confirm (input hidden): 
Please enter the Keycloak WildFly Management password (input hidden): 
Re-enter to confirm (input hidden): 
Secret kxi-keycloak successfully created
Please enter the Postgresql postgres password (input hidden): 
Re-enter to confirm (input hidden): 
Please enter the Postgresql user password (input hidden): 
Re-enter to confirm (input hidden): 
Secret kxi-postgresql successfully created
Do you want to set a secret for the gui service account explicitly [y/N]: 
Randomly generating client secret for gui and setting in values file, record this value for reuse during upgrade
Persisting option guiClientSecret to file ~/.insights/cli-config
Do you want to set a secret for the operator service account explicitly [y/N]: 
Randomly generating client secret for operator and setting in values file, record this value for reuse during upgrade
Persisting option operatorClientSecret to file ~/.insights/cli-config
Service Accounts setup

The KX Insights Platform requires internal service accounts to allow services to communicate via authenticated channels. These secrets can be auto generated or they can be user defined. The installer will prompt if they want to provide this and, if so, ask for the details.

Do you want to set a secret for the gui service account explicitly [y/N]: y
Please enter the secret (input hidden): 
Re-enter to confirm (input hidden): 
Do you want to set a secret for the operator service account explicitly [y/N]: y
Please enter the secret (input hidden):
Re-enter to confirm (input hidden): 
Setup complete

At this point you should have a generated values.yaml config file. This file is also persisted as an install values secret.

Secret kxi-install-config successfully created

KX Insights installation setup complete

Helm values file for installation saved in values.yaml
Deploy KX Insights Platform

Now the values.yaml has been generated, it can be used to deploy KX Insights to the active cluster

kxi install run --filepath values.yaml --version $INSIGHTS_VERSION --operator-version $OPERATOR_VERSION

This command will install the KX Insights Platform to the currently active namespace within the client's kubeconfig. More info on creating and managing namespaces is available here.

It will also install the KXI Operator to the kxi-operator namespace if it is not installed. If it is installed it will skip installing the KXI Operator. If you wish to re-install the KXI Operator, it requires that it be deleted before installing the KX Insights Platform.

Info

There are further configuration and customizations available for the KX Insights Platform. Details on how to use these is available in the Configuration section.

KX Insights Platform Versions

To view the list of available versions, use helm

helm search repo kx-insights/insights --versions

Post install

Once the KX Insights Platform is running you should see pods deployed similar to below output. All pods should enter a Running state after a short period of time.

kubectl get pods -l app.kubernetes.io/instance=insights

Applying an assembly

After installing the base system, an assembly can be deployed for data capture.

Deploying an assembly requires an Assembly CR file defining the data pipeline and the configuration required within a yaml file. Full details on how to build assemblies and the specification is available here.

A number of sample assemblies have been released and provided as samples to ingest data and run workloads. These are documented under Ingesting data.

Each of which can be deployed by a single command, from which the KX Insights Platform Operator Service will manage the lifecycle of that pipeline. All use cases give further detail on how to interact appropriately with data pipelines, both to publish data into the pipeline, and query data out of the KX Insights Platform.

Example assembly deployment command

kubectl apply -f sdk_sample_assembly.yaml

The label insights.kx.com/app currently gets set to the name of the assembly so this can be used as a filter to view the resources. For example if you deployed an assembly called sdk-sample-assembly, you could view the pods associated with the assembly by running the following...

kubectl get pods -l insights.kx.com/app=sdk-sample-assembly

Note

As before all pods should eventually enter a Running state.