kdb Insights Enterprise Architecture
kdb Insights Enterprise is built as a cloud native microservice application which leans on Kubernetes to orchestrate the deployment and scalability of the application. This principle allows kdb Insights Enterprise to focus on business logic and solid OpenAPI interfaces between services to meet the key requirements of a fast, secure and scalable data capture and streaming solution.
Cluster Setup and Prerequisites
kdb Insights Enterprise Installation requires the following prerequisites:
- Virtual network, public/private subnets
- Bastion Host
- Managed Kubernetes Cluster
- Certificate manager based on cert-manager
- Ingress Controller based on ingress-nginx
- Persistent storage resources
Terraform scripts to deploy these necessary shared resources have been released and can be used on all cloud providers. Full details on these resources and how to setup a supporting cluster can be found here
kdb Insights Components
There are a number of microservices and components within kdb Insights which collaborate to offer the key feature set of kdb Insights Enterprise. Being a cloud native application built on Kubernetes, it utilizes a variety of cloud technologies requiring a Kubernetes Cluster to be provided along with a number of pre-requisites to be in place to allow the application to be installed.
Base kdb Insights Enterprise Application
Deploying kdb Insights Enterprise is achieved through
Helm and Helm charts. These charts are released to kdb Insights Enterprise repository which can be added to your helm repository configuration. Full details in how to configure helm to connect to kdb Insights Enterprise helm repo can be found here.
The base kdb Insights Enterprise install includes framework components necessary to provide:
access through authenticated and authorized endpoints.
At a high level these core services include:
- The kdb Insights Operator - Data pipeline deployment manager
- Stream Processor Coordination - Streaming Data and transformations
- Discovery & Information Services - Data and client discovery repositories
- kdb Insights Enterprise User Interface
- Authentication & authorization Services
- Service Gateway - Authenticated query access point
- Resource coordinator & Aggregator - Data Management services
Installation of the kdb Insights Enterprise base chart along with the deployment of an Assembly CR is detailed here.
kdb Insights Operator
The kdb Insights Operator offers the ability to deploy, persist and transform pipelines dynamically within kdb Insights Enterprise applications. Data can be distributed, sharded or replicated through a Kubernetes custom resource definition (CRD)
Assembly file. Clients can build their own
Assembly CR pipelines to meet their requirements; sources; schemas and sizing. Full details on the KX Operator; Assembly CRD and configuration can be found here
Authentication & Authorization
kdb Insights Enterprise handles the full authentication and authorization life-cycles for both human and programmatic users. It can be integrated with OAuth compliant IDP's and allow authenticated endpoints for programmatic applications to both publish and query data over secure TLS protected endpoints. Clients can be enrolled and provide unique identifiers to allow data to be published through to specific data pipelines safe in the guarantee the Reliable Transport (RT) message pipelines will ensure all data is guaranteed for delivery and available for recovery at each point in the pipeline.
Full details about client registration and how network connections are authorized and secure is available here
kdb Insights Enterprise UI is an easy-to-use data visualization tool for kdb Insights Enterprise. Users can import and query their data, generate custom tables and charts, build data pipelines, all with support for streaming data. In addition, users can create customizable views.
Data Persistence and Retrieval
kdb Insights Enterprise persists data from upstream data pipelines in an efficient, fault-tolerant way. kdb Insights Enterprise allows the user to define a configuration for how their data is stored via an assembly (further documentation on assemblies can be found here). kdb Insights Enterprise will also migrate data across user-defined tiers spanning on-disk and cloud object storage, allowing users to organize their data efficiently and without on-going intervention.
kdb Insights Enterprise provides an authenticated, scalable query path for retrieval of in-memory and on-disk data, all without the end user needing to understand the underlying storage configuration. It provides users with an out-of-the-box API for querying data while also allowing for user-defined, custom APIs to be executed against the Data Access processes with simple aggregation. Details of the supported APIs can be found here.
Stream Processor Coordinator
The Coordinator is a persistent service provided in kdb Insights Enterprise responsible for submitting, managing, listing, and tearing down streaming jobs based on configuration and performance parameters. It will dynamically spawn Controllers (who then create Workers) based on the provided configuration.
More information on the Stream Processor and writing streaming jobs can be found below.
RT is the native message transport for kdb Insights Enterprise, and it is designed to satisfy both extremely high availability and high performance requirements of kdb Insights Enterprise.