kdb Insights Enterprise Installation
The section details the steps to deploy kdb Insights Enterprise in a pre-configured Kubernetes Cluster. It will walk through
Once installed, the user will be able to deploy workloads that can be designed and developed to meet their specific needs and requirements.
For users wishing to upgrade their kdb Insights Enterprise deploy, refer to the upgrade guide
Cluster and shared services
Before installing kdb Insights Enterprise it is necessary to have an appropriate Kubernetes cluster in place with the necessary shared services available.
- Virtual network, public, private and bastian resources
- Certificate manager
- Ingress controller
- Persistent storage resources
The kdb Insights Enterprise provides a number of Terraform scripts which enable the quick deployment of the necessary cluster services on all 3 main cloud providers. More details here.
kdb Insights Enterprise namespaces
The kdb Insights Enterprise can be deployed to the default namespace or a uniquely created namespace. However two deployments of kdb Insights Enterprise cannot be on the same namespace. When deploying kdb Insights Enterprise you should check the active namespace to ensure there are no current deployments.
This will return any instances of kdb Insights Enterprise currently deployed to the active namespace. Further details on Kubernetes namespaces and contexts are available here.
The Kxi Operator is deployed to its own namespace
kxi-operator and requires only to be installed once per cluster. When deployed, it services all deployments of kdb Insights Enterprise and reconciles all Assembly workloads deployed to the appropriate instance of kdb Insights Enterprise based on the active namespace of the client who deployed the assembly workload.
In order for external clients to interact with kdb Insights Enterprise, a DNS record must be setup with your DNS provider. For examples on setting up DNS records with the cloud providers, please see here.
Access to your cluster and Kubernetes tooling available
The user must have access to the Kubernetes cluster along with a number of third party tools to interact with the cluster. Details here
The KXI command line interface installed
Having the KXI CLI installed simplifies install and will be used in the following instructions. Details here.
kdb Insights compatible license
The install process requires a kdb Insights license. To obtain a valid kdb Insights license, speak to your KX Sales representative.
kdb Insights provides the ability to self-generate licenses. For information on this, please refer to the documentation here.
Access to the appropriate helm/docker registries for charts and images
The install will prompt for repository locations and access details
The KXI CLI provides an interactive installer that can be used to deploy kdb Insights Enterprise on a Kubernetes cluster. It will prompt and walk the user through the steps to install kdb Insights Enterprise. To ensure a smooth process, it is recommended to have a number of variables to hand
|INSIGHTS_HOSTNAME||kxi-insights.domain.com||DNS Hostname setup|
|HELM_REPO||https://nexus.dl.kx.com/repository/kx-insights-charts||(Optional) Helm repo details|
|DOCKER_REPO||registry.dl.kx.com||(Optional) Docker repo details|
|ACCESS_USER||my-user||Helm/Docker repo details|
|ACCESS_PASS||my-pass||Helm/Docker repo details|
|KX_LIC_PATH||/home/app/q/kc.lic||Path to KX License|
|CA_CERT_PATH||/home/app/certs/ca.crt||(Optional) Path to ca.crt file|
|CA_KEY_PATH||/home/app/certs/ca.crt||(Optional) Path to ca.key file|
|INSIGHTS_VERSION||1.1.1||Version of insights to install|
|OPERATOR_VERSION||1.1.0||Version of insights operator to install|
Installing kdb Insights Enterprise
To install kdb Insights Enterprise two commands are run; the first
kxi install setup will generate a
values.yaml config file for us to use for repeatable future installs; followed by a
kxi install run --filepath values.yaml command providing the values config file as an argument to deploy kdb Insights Enterprise to the cluster.
It is possible to combine these two commands in the one
kxi install run, but to ensure clarity, this walkthrough separates them.
kxi install setup
Provide the hostname defined by
$INSIGHTS_HOSTNAME DNS record as discussed above
Checking ingress configuration... Please enter the hostname for the installation: kxi-insights.example.com Using Let's Encrypt TLS certificates
The kdb Insights Enterprise requires ingress to be enabled in order to route external client requests into the application via UI or REST API. There are two supported modes; auto-generated (default) or self-managed.
In this mode kdb Insights Enterprise will use cert manager to issue certificates via Let's Encrypt. These certs will be automatically generated and be used by the ingress controller to secure traffic. This is the default mode.
In some cases it may be preferable to self-manage certificates, especially if there is a preference to use a cert authority not supported by cert manager, e.g. DigiCert. This requires additional setup and maintenance as the certificates aren't automatically generated. Guidance on manually creating these files and secret can be found in the Manual Install section.
To use self-managed certs, the
--ingress-key options may be passed to
kxi install setup as follows:
kxi install setup --ingress-cert <CA_CRT_PATH> --ingress-key <CA_KEY_PATH> ... Checking ingress configuration... Using ingress.cert <CA_CRT_PATH> from command line option Using ingress.key <CA_KEY_PATH> from command line option Secret kxi-ingress-cert successfully created
Certificate renewals need to be handled manually. The command to create the secret above should be repeated with the existing
ca.key and a newly issued
Chart Repository details
If installing from repositories, the defaults can be used along with your KX-provided credentials. If installing from a local repository, please submit the required details.
Checking chart details... Please enter a name for the chart repository to set locally [kx-insights]: Please enter the chart repository URL to pull charts from [https://nexus.dl.kx.com/repository/kx-insights-charts]: Please enter the username for the chart repository: <ACCESS_USER> Please enter the password for the chart repository (input hidden): <ACCESS_PASS> Re-enter to confirm (input hidden): <ACCESS_PASS>
If an existing license secret exists it will be automatically used. Otherwise, provide the $KX_LIC_PATH to your license file to generate the secret on your cluster
Checking license details... Please enter the path and filename of your kdb+ license: <KX_LIC_PATH> Secret kxi-license successfully created
Image Repository details
Defaults may be used if utilizing KX repositories along with access details provided by KX. Or if downloaded to an alternative location, provide those details at this point.
Image repository Please enter the image repository to pull images from [registry.dl.kx.com]: Please enter the username for registry.dl.kx.com: <ACCESS_USER> Please enter the password for <ACCESS_USER> (input hidden): <ACCESS_PASS> Re-enter to confirm (input hidden): <ACCESS_PASS> Secret kxi-nexus-pull-secret successfully created
The kdb Insights Enterprise uses mutual TLS for clients publishing data into the system. This requires the system to be configured with a certificate issuer to generate unique keys and certs for each client. The installer will create this certificate issuer when one does not already exist. Guidance on manually creating these files and secret can be found in the Manual Install section.
Checking client certificate issuer... Secret kxi-certificate successfully created
Keycloak is used as the identity and access management platform for kdb Insights Enterprise.
In the base install of kdb Insights Enterprise, there are four passwords that are required.
||This is the password for the Keycloak Admin UI|
||This is the password for the WildFly management user|
||This is the password for the database superuser|
||This is the password for the database user that Keycloak uses|
The installer will prompt you for these four passwords unless you have an existing secret that contains them.
Checking keycloak configuration... Please enter the Keycloak Admin password (input hidden): Re-enter to confirm (input hidden): Please enter the Keycloak Admin password (input hidden): Re-enter to confirm (input hidden): Please enter the Keycloak WildFly Management password (input hidden): Re-enter to confirm (input hidden): Secret kxi-keycloak successfully created Please enter the Postgresql postgres password (input hidden): Re-enter to confirm (input hidden): Please enter the Postgresql user password (input hidden): Re-enter to confirm (input hidden): Secret kxi-postgresql successfully created Do you want to set a secret for the gui service account explicitly [y/N]: Randomly generating client secret for gui and setting in values file, record this value for reuse during upgrade Persisting option guiClientSecret to file ~/.insights/cli-config Do you want to set a secret for the operator service account explicitly [y/N]: Randomly generating client secret for operator and setting in values file, record this value for reuse during upgrade Persisting option operatorClientSecret to file ~/.insights/cli-config
Service Accounts setup
The kdb Insights Enterprise requires internal service accounts to allow services to communicate via authenticated channels. These secrets can be auto generated or they can be user defined. The installer will prompt if they want to provide this and, if so, ask for the details.
Do you want to set a secret for the gui service account explicitly [y/N]: y Please enter the secret (input hidden): Re-enter to confirm (input hidden): Do you want to set a secret for the operator service account explicitly [y/N]: y Please enter the secret (input hidden): Re-enter to confirm (input hidden):
At this point you should have a generated
values.yaml config file. This file is also persisted as an install values secret.
Secret kxi-install-config successfully created KX Insights installation setup complete Helm values file for installation saved in values.yaml
Deploy kdb Insights Enterprise
values.yaml has been generated, it can be used to deploy kdb Insights Enterprise to the active cluster
kxi install run --filepath values.yaml --version $INSIGHTS_VERSION --operator-version $OPERATOR_VERSION
It will also install the KXI Operator to the
kxi-operator namespace if it is not installed. If it is installed it will skip installing the KXI Operator. If you wish to re-install the KXI Operator, it requires that it be deleted before installing kdb Insights Enterprise.
There are further configuration and customizations available for kdb Insights Enterprise. Details on how to use these is available in the Configuration section.
kdb Insights Enterprise Versions
To view the list of available versions, use
helm search repo kx-insights/insights --versions
Once kdb Insights Enterprise is running you should see pods deployed similar to below output.
All pods should enter a
Running state after a short period of time.
kubectl get pods -l app.kubernetes.io/instance=insights
Applying an assembly
After installing the base system, an assembly can be deployed for data capture.
Deploying an assembly requires an Assembly CR file defining the data pipeline and the configuration required within a yaml file. Full details on how to build assemblies and the specification is available here.
A number of sample assemblies have been released and provided as samples to ingest data and run workloads. These are documented under Ingesting data.
Each of which can be deployed by a single command, from which kdb Insights Enterprise Operator Service will manage the lifecycle of that pipeline. All use cases give further detail on how to interact appropriately with data pipelines, both to publish data into the pipeline, and query data out of kdb Insights Enterprise.
Example assembly deployment command
kxi assembly deploy --filepath sdk_sample_assembly.yaml
insights.kx.com/app currently gets set to the name of the assembly
so this can be used as a filter to view the resources. For example if you deployed
an assembly called
sdk-sample-assembly, you could view the pods associated with the assembly by running the following...
kubectl get pods -l insights.kx.com/app=sdk-sample-assembly
As before all pods should eventually enter a