KX Insights Architecture
KX Insights is built as a cloud native microservice application which leans on Kubernetes to orchestrate the deployment and scalability of the application. This principle allows KX Insights to focus on business logic and solid OpenAPI interfaces between services to meet the key requirements of a fast, secure and scalable data capture and streaming Platform.
Cluster Setup and Prerequisites
KX Insights Platform Installation requires the following prerequisites:
- Virtual network, public/private subnets
- Bastion Host
- Managed Kubernetes Cluster
- Certificate manager based on cert-manager
- Ingress Controller based on ingress-nginx
- Persistent storage resources
Terraform scripts to deploy these necessary shared resources have been released and can be used on all cloud providers. Full details on these resources and how to setup a supporting cluster can be found here
KX Insights Core Components
There are a number of microservices and components within KX Insights which collaborate to offer the key feature set of KX Insights Platform. Being a cloud native application built on Kubernetes, it utilizes a variety of cloud technologies requiring a Kubernetes Cluster to be provided along with a number of pre-requisites to be in place to allow the application to be installed.
Base KX Insights Platform Application
Deploying KX Insights Platform is achieved through
Helm and Helm charts. These charts are released to the KX Insights repository which can be added to your helm repository configuration. Full details in how to configure helm to connect to the KX Insights Platform helm repo can be found here.
The base KX Insights install includes framework components necessary to provide:
access through authenticated and authorized endpoints.
At a high level these core services include:
- The KX Insights Operator - Data pipeline deployment manager
- Stream Processor Coordination - Streaming Data and transformations
- Discovery & Information Services - Data and client discovery repositories
- KX Insights User Interface
- Authentication & authorization Services
- Service Gateway - Authenticated query access point
- Resource coordinator & Aggregator - Data Management services
Installation of the KX Insights base chart along with the deployment of an Assembly CR is detailed here.
KX Insights Operator
The KX Insights Operator offers the ability to deploy, persist and transform pipelines dynamically within the KX Insights applications. Data can be distributed, sharded or replicated through a Kubernetes custom resource definition (CRD)
Assembly file. Clients can build their own
Assembly CR pipelines to meet their requirements; sources; schemas and sizing. Full details on the KX Operator; Assembly CRD and configuration can be found here
Authentication & Authorization
KX Insights handles the full authentication and authorization life-cycles for both human and programmatic users. It can be integrated with OAuth compliant IDP's and allow authenticated endpoints for programmatic applications to both publish and query data over secure TLS protected endpoints. Clients can be enrolled and provide unique identifiers to allow data to be published through to specific data pipelines safe in the guarantee the Reliable Transport (RT) message pipelines will ensure all data is guaranteed for delivery and available for recovery at each point in the pipeline.
Full details about client registration and how network connections are authorized and secure is available here
KX Insights UI is an easy-to-use data visualization tool for KX Insights. Users can import and query their data, generate custom tables and charts, build data pipelines, all with support for streaming data. In addition, users can create customizable views.
Data Persistence and Retrieval
The Insights Platform persists data from upstream data pipelines in an efficient, fault-tolerant way. Insights allows the user to define a configuration for how their data is stored via an assembly (further documentation on assemblies can be found here). Insights will also migrate data across user-defined tiers spanning on-disk and cloud object storage, allowing users to organize their data efficiently and without on-going intervention.
Insights provides an authenticated, scalable query path for retrieval of in-memory and on-disk data, all without the end user needing to understand the underlying storage configuration. It provides users with an out-of-the-box API for querying data while also allowing for user-defined, custom APIs to be executed against the Data Access processes with simple aggregation. Details of the supported APIs can be found here.
Stream Processor Coordinator
The Coordinator is a persistent service provided in the platform responsible for submitting, managing, listing, and tearing down streaming jobs based on configuration and performance parameters. It will dynamically spawn Controllers (who then create Workers) based on the provided configuration.
More information on the Stream Processor and writing streaming jobs can be found below.
RT is the native message transport for KX Insights, and it is designed to satisfy both extremely high availability and high performance requirements of the KX Insights Platform.