kdb Insights Enterprise Installation

The section details the steps to deploy kdb Insights Enterprise in a pre-configured Kubernetes Cluster. It will walk through

Prerequisites
Installing kdb Insights Enterprise
Applying an assembly

Once installed, the user will be able to deploy workloads that can be designed and developed to meet their specific needs and requirements.

For users wishing to upgrade their kdb Insights Enterprise deploy, refer to the upgrade guide

Prerequisites

Cluster and shared services

Before installing kdb Insights Enterprise it is necessary to have an appropriate Kubernetes cluster in place with the necessary shared services available.

These include

Virtual network, public, private and bastian resources
Certificate manager
Ingress controller
Persistent storage resources

The kdb Insights Enterprise provides a number of Terraform scripts which enable the quick deployment of the necessary cluster services on all 3 main cloud providers. More details here.

kdb Insights Enterprise namespaces

The kdb Insights Enterprise can be deployed to the default namespace or a uniquely created namespace. However two deployments of kdb Insights Enterprise cannot be on the same namespace. When deploying kdb Insights Enterprise you should check the active namespace to ensure there are no current deployments.

helm ls

This will return any instances of kdb Insights Enterprise currently deployed to the active namespace. Further details on Kubernetes namespaces and contexts are available here.

The Kxi Operator is deployed to its own namespace kxi-operator and requires only to be installed once per cluster. When deployed, it services all deployments of kdb Insights Enterprise and reconciles all Assembly workloads deployed to the appropriate instance of kdb Insights Enterprise based on the active namespace of the client who deployed the assembly workload.

DNS

In order for external clients to interact with kdb Insights Enterprise, a DNS record must be setup with your DNS provider. For examples on setting up DNS records with the cloud providers, please see here.

Pre-install checklist

Access to your cluster and Kubernetes tooling available

The user must have access to the Kubernetes cluster along with a number of third party tools to interact with the cluster. Details here
The KXI command line interface installed

Having the KXI CLI installed simplifies install and will be used in the following instructions. Details here.
kdb Insights compatible license

The install process requires a kdb Insights license. To obtain a valid kdb Insights license, speak to your KX Sales representative.

kdb Insights provides the ability to self-generate licenses. For information on this, please refer to the documentation here.
Access to the appropriate helm/docker registries for charts and images

The install will prompt for repository locations and access details

The KXI CLI provides an interactive installer that can be used to deploy kdb Insights Enterprise on a Kubernetes cluster. It will prompt and walk the user through the steps to install kdb Insights Enterprise. To ensure a smooth process, it is recommended to have a number of variables to hand

Variable	Example	Further Details
INSIGHTS_HOSTNAME	kxi-insights.domain.com	DNS Hostname setup
HELM_REPO	https://nexus.dl.kx.com/repository/kx-insights-charts	(Optional) Helm repo details
DOCKER_REPO	registry.dl.kx.com	(Optional) Docker repo details
ACCESS_USER	my-user	Helm/Docker repo details
ACCESS_PASS	my-pass	Helm/Docker repo details
KX_LIC_PATH	/home/app/q/kc.lic	Path to KX License
CA_CERT_PATH	/home/app/certs/ca.crt	(Optional) Path to ca.crt file
CA_KEY_PATH	/home/app/certs/ca.crt	(Optional) Path to ca.key file
INSIGHTS_VERSION	1.1.1	Version of insights to install
OPERATOR_VERSION	1.1.0	Version of insights operator to install

Installing kdb Insights Enterprise

To install kdb Insights Enterprise two commands are run; the first kxi install setup will generate a values.yaml config file for us to use for repeatable future installs; followed by a kxi install run --filepath values.yaml command providing the values config file as an argument to deploy kdb Insights Enterprise to the cluster.

Note

It is possible to combine these two commands in the one kxi install run, but to ensure clarity, this walkthrough separates them.

All options prompted to the user can alternativly be supplied by command line arguments or cli configuration. See the options precedence for a description of the order in which these are read.

Run setup

kxi install setup

Hostname

Provide the hostname defined by $INSIGHTS_HOSTNAME DNS record as discussed above

Checking ingress configuration...
Please enter the hostname for the installation: kxi-insights.example.com
Using Let's Encrypt TLS certificates

Ingress certification

The kdb Insights Enterprise requires ingress to be enabled in order to route external client requests into the application via UI or REST API. There are two supported modes; auto-generated (default) or self-managed.

Auto-generated

In this mode kdb Insights Enterprise will use cert manager to issue certificates via Let's Encrypt. These certs will be automatically generated and be used by the ingress controller to secure traffic. This is the default mode.

Self-managed

In some cases it may be preferable to self-manage certificates, especially if there is a preference to use a cert authority not supported by cert manager, e.g. DigiCert. This requires additional setup and maintenance as the certificates aren't automatically generated. Guidance on manually creating these files and secret can be found in the Manual Install section.

To use self-managed certs, the --ingress-cert and --ingress-key options may be passed to kxi install setup as follows:

kxi install setup --ingress-cert <CA_CRT_PATH> --ingress-key <CA_KEY_PATH>
...
Checking ingress configuration...
Using ingress.cert <CA_CRT_PATH> from command line option
Using ingress.key <CA_KEY_PATH> from command line option
Secret kxi-ingress-cert successfully created

Renewals

Certificate renewals need to be handled manually. The command to create the secret above should be repeated with the existing ca.key and a newly issued ca.crt.

Chart Repository details

If installing from repositories, the defaults can be used along with your KX-provided credentials. If installing from a local repository, please submit the required details.

Checking chart details...
Please enter a name for the chart repository to set locally [kx-insights]: 
Please enter the chart repository URL to pull charts from [https://nexus.dl.kx.com/repository/kx-insights-charts]: 
Please enter the username for the chart repository: <ACCESS_USER>
Please enter the password for the chart repository (input hidden): <ACCESS_PASS>
Re-enter to confirm (input hidden): <ACCESS_PASS>

License information

If an existing license secret exists it will be automatically used. Otherwise, provide the $KX_LIC_PATH to your license file to generate the secret on your cluster

Checking license details...
Please enter the path and filename of your kdb+ license: <KX_LIC_PATH>
Secret kxi-license successfully created

Image Repository details

Defaults may be used if utilizing KX repositories along with access details provided by KX. Or if downloaded to an alternative location, provide those details at this point.

Image repository
Please enter the image repository to pull images from [registry.dl.kx.com]: 
Please enter the username for registry.dl.kx.com: <ACCESS_USER>
Please enter the password for <ACCESS_USER> (input hidden): <ACCESS_PASS>
Re-enter to confirm (input hidden): <ACCESS_PASS>
Secret kxi-nexus-pull-secret successfully created

Client certification

The kdb Insights Enterprise uses mutual TLS for clients publishing data into the system. This requires the system to be configured with a certificate issuer to generate unique keys and certs for each client. The installer will create this certificate issuer when one does not already exist. Guidance on manually creating these files and secret can be found in the Manual Install section.

Checking client certificate issuer...
Secret kxi-certificate successfully created

Keycloak Authentication

Keycloak is used as the identity and access management platform for kdb Insights Enterprise.

In the base install of kdb Insights Enterprise, there are four passwords that are required.

Password	Description
`admin-password`	This is the password for the Keycloak Admin UI
`management-password`	This is the password for the WildFly management user
`postgresql-postgres-password`	This is the password for the database superuser
`postgresql-password`	This is the password for the database user that Keycloak uses

The installer will prompt you for these four passwords unless you have an existing secret that contains them.

Checking keycloak configuration...
Please enter the Keycloak Admin password (input hidden): 
Re-enter to confirm (input hidden): 
Please enter the Keycloak Admin password (input hidden): 
Re-enter to confirm (input hidden): 
Please enter the Keycloak WildFly Management password (input hidden): 
Re-enter to confirm (input hidden): 
Secret kxi-keycloak successfully created
Please enter the Postgresql postgres password (input hidden): 
Re-enter to confirm (input hidden): 
Please enter the Postgresql user password (input hidden): 
Re-enter to confirm (input hidden): 
Secret kxi-postgresql successfully created
Do you want to set a secret for the gui service account explicitly [y/N]: 
Randomly generating client secret for gui and setting in values file, record this value for reuse during upgrade
Persisting option guiClientSecret to file ~/.insights/cli-config
Do you want to set a secret for the operator service account explicitly [y/N]: 
Randomly generating client secret for operator and setting in values file, record this value for reuse during upgrade
Persisting option operatorClientSecret to file ~/.insights/cli-config

Service Accounts setup

The kdb Insights Enterprise requires internal service accounts to allow services to communicate via authenticated channels. These secrets can be auto generated or they can be user defined. The installer will prompt if they want to provide this and, if so, ask for the details.

Do you want to set a secret for the gui service account explicitly [y/N]: y
Please enter the secret (input hidden): 
Re-enter to confirm (input hidden): 
Do you want to set a secret for the operator service account explicitly [y/N]: y
Please enter the secret (input hidden):
Re-enter to confirm (input hidden):

Setup complete

At this point you should have a generated values.yaml config file. This file is also persisted as an install values secret.

Secret kxi-install-config successfully created

KX Insights installation setup complete

Helm values file for installation saved in values.yaml

Deploy kdb Insights Enterprise

Now the values.yaml has been generated, it can be used to deploy kdb Insights Enterprise to the active cluster

kxi install run --filepath values.yaml --version $INSIGHTS_VERSION --operator-version $OPERATOR_VERSION

This command will install kdb Insights Enterprise to the currently active namespace within the client's kubeconfig. More info on creating and managing namespaces is available here.

It will also install the KXI Operator to the kxi-operator namespace if it is not installed. If it is installed it will skip installing the KXI Operator. If you wish to re-install the KXI Operator, it requires that it be deleted before installing kdb Insights Enterprise.

Info

There are further configuration and customizations available for kdb Insights Enterprise. Details on how to use these is available in the Configuration section.

kdb Insights Enterprise Versions

To view the list of available versions, use helm

helm search repo kx-insights/insights --versions

Post install

Once kdb Insights Enterprise is running you should see pods deployed similar to below output. All pods should enter a Running state after a short period of time.

kubectl get pods -l app.kubernetes.io/instance=insights

Applying an assembly

After installing the base system, an assembly can be deployed for data capture.

Deploying an assembly requires an Assembly CR file defining the data pipeline and the configuration required within a yaml file. Full details on how to build assemblies and the specification is available here.

A number of sample assemblies have been released and provided as samples to ingest data and run workloads. These are documented under Ingesting data.

Each of which can be deployed by a single command, from which kdb Insights Enterprise Operator Service will manage the lifecycle of that pipeline. All use cases give further detail on how to interact appropriately with data pipelines, both to publish data into the pipeline, and query data out of kdb Insights Enterprise.

Example assembly deployment command

kxi assembly deploy --filepath sdk_sample_assembly.yaml

The label insights.kx.com/app currently gets set to the name of the assembly so this can be used as a filter to view the resources. For example if you deployed an assembly called sdk-sample-assembly, you could view the pods associated with the assembly by running the following...

kubectl get pods -l insights.kx.com/app=sdk-sample-assembly

Note

As before all pods should eventually enter a Running state.