Skip to content

KX Insights Architecture

Introduction

KX Insights is built as a cloud native microservice application which leans on Kubernetes to orchestrate the deployment and scalability of the application. This principle allows KX Insights to focus on business logic and solid OpenAPI interfaces between services to meet the key requirements of a fast, secure and scalable data capture and streaming Platform.

KX Insights Platform

Cluster Setup and Prerequisites

KX Insights Platform Installation requires the following prerequisites:

  • Virtual network, public/private subnets
  • Bastion Host
  • Managed Kubernetes Cluster
  • Certificate manager based on cert-manager
  • Ingress Controller based on ingress-nginx
  • Persistent storage resources

Terraform scripts to deploy these necessary shared resources have been released and can be used on all cloud providers. Full details on these resources and how to setup a supporting cluster can be found here

KX Insights Core Components

There are a number of microservices and components within KX Insights which collaborate to offer the key feature set of KX Insights Platform. Being a cloud native application built on Kubernetes, it utilizes a variety of cloud technologies requiring a Kubernetes Cluster to be provided along with a number of pre-requisites to be in place to allow the application to be installed.

Base KX Insights Platform Application

Deploying KX Insights Platform is achieved through Helm and Helm charts. These charts are released to the KX Insights repository which can be added to your helm repository configuration. Full details in how to configure helm to connect to the KX Insights Platform helm repo can be found here.

The base KX Insights install includes framework components necessary to provide:

  • dynamic scalability,

  • data discovery,

  • access through authenticated and authorized endpoints.

At a high level these core services include:

  • The KX Insights Operator - Data pipeline deployment manager
  • Stream Processor Coordination - Streaming Data and transformations
  • Discovery & Information Services - Data and client discovery repositories
  • KX Insights User Interface
  • Authentication & authorization Services
  • Service Gateway - Authenticated query access point
  • Resource coordinator & Aggregator - Data Management services

Installation of the KX Insights base chart along with the deployment of an Assembly CR is detailed here.

KX Insights Operator

The KX Insights Operator offers the ability to deploy, persist and transform pipelines dynamically within the KX Insights applications. Data can be distributed, sharded or replicated through a Kubernetes custom resource definition (CRD) Assembly file. Clients can build their own Assembly CR pipelines to meet their requirements; sources; schemas and sizing. Full details on the KX Operator; Assembly CRD and configuration can be found here

Authentication & Authorization

KX Insights handles the full authentication and authorization life-cycles for both human and programmatic users. It can be integrated with OAuth compliant IDP's and allow authenticated endpoints for programmatic applications to both publish and query data over secure TLS protected endpoints. Clients can be enrolled and provide unique identifiers to allow data to be published through to specific data pipelines safe in the guarantee the Reliable Transport (RT) message pipelines will ensure all data is guaranteed for delivery and available for recovery at each point in the pipeline.

Full details about client registration and how network connections are authorized and secure is available here

User Interface

KX Insights UI is an easy-to-use data visualization tool for KX Insights. Users can import and query their data, generate custom tables and charts, build data pipelines, all with support for streaming data. In addition, users can create customizable views.

Data Persistence and Retrieval

The Insights Platform persists data from upstream data pipelines in an efficient, fault-tolerant way. Insights allows the user to define a configuration for how their data is stored via an assembly (further documentation on assemblies can be found here). Insights will also migrate data across user-defined tiers spanning on-disk and cloud object storage, allowing users to organize their data efficiently and without on-going intervention.

Insights provides an authenticated, scalable query path for retrieval of in-memory and on-disk data, all without the end user needing to understand the underlying storage configuration. It provides users with an out-of-the-box API for querying data while also allowing for user-defined, custom APIs to be executed against the Data Access processes with simple aggregation. Details of the supported APIs can be found here.

Stream Processor Coordinator

The Coordinator is a persistent service provided in the platform responsible for submitting, managing, listing, and tearing down streaming jobs based on configuration and performance parameters. It will dynamically spawn Controllers (who then create Workers) based on the provided configuration.

More information on the Stream Processor and writing streaming jobs can be found below.

Stream Processor Overview

Reliable Transport

RT is the native message transport for KX Insights, and it is designed to satisfy both extremely high availability and high performance requirements of the KX Insights Platform.