KX Insights

Stream processor

KX Insights

Home
Home
- About
- kdb+ and q
- PyKX
- KX Community
- KX Academy
- KX
Core
Core
- About
- Release notes
  Release notes
  - 3.2.0 (Latest)
  - Previous Releases
    Previous Releases
    
    3.1.1
    
    3.1.0
    
    3.0.1
    
    3.0.0
    
    2.1.0
    
    2.0.10
    
    2.0.9
    
    2.0.8
    
    2.0.7
    
    2.0.6
    
    2.0.5
    
    2.0.4
    
    2.0.3
    
    2.0.2
    
    2.0.1
    
    2.0.0
    
    1.0.6
    
    1.0.5
    
    1.0.4
    
    1.0.3
    
    1.0.2
    
    1.0.1
    
    1.0.0
- Prerequisites
- Contents
- Quick start
  Quick start
  - Overview
  - Packaging
  - REST
    REST
    
    Client
    
    Server
  - Logging
  - Object storage
  - Google Cloud Platform
    Google Cloud Platform
    
    Big Query
- Packaging
  Packaging
  - About
  - Examples
    Examples
    
    About the examples
    
    Basic Tick
    
    Hello Python
    
    Hello C
  - Labelling
- REST API
  REST API
  - Client
    Client
    
    About
    
    Workflows
    
    Examples
    Examples
    
    Async
    
    Follow redirects
    
    Response headers
    
    Timeouts
    
    Azure API Management
    
    GCP Identity Aware Proxy
  - Server
    Server
    
    About
    
    API reference
    
    Examples
    Examples
    
    customers
    
    queryclient
    
    queryserver
    
    queryworker
    
    OpenAPI Sample
- Logging
  Logging
  - About
  - API reference
- Object storage
  Object storage
  - About
  - Caching
  - Examples
- SQL
  SQL
  - About
  - pgwire
- Google Cloud Platform
  Google Cloud Platform
  - BigQuery
    BigQuery
    
    About
    
    Main
    
    Discovery
    
    Query
    
    Projects
    
    Datasets
    
    Tables
    
    Tabledata
    
    Helpers
    
    Configuration
    
    API
    
    Troubleshooting
Microservices
Microservices
- About
- Release notes
  Release notes
  - 1.1.3 (Latest)
  - Previous Releases
    Previous Releases
    
    1.1.2
    
    1.0.0
    
    0.10.0
    
    0.9.2
    
    0.9.1
    
    0.9.0
    
    0.8.0
- Prerequisites
- Artifacts
- Quickstart
  Quickstart
  - Service Discovery
  - Stream Processor
    Stream Processor
    
    Docker
    
    Kubernetes
  - Dashboards
- Configuration
  Configuration
  - Assembly Configuration
- Downloads
  Downloads
  - Artifacts
  - Docker registry
- Service Discovery
  Service Discovery
  - About
  - Prerequisites
  - Running
  - Configuration
  - OpenAPI
- Data Access
  Data Access
  - About
  - Service Gateway
    Service Gateway
    
    About
    
    Configuration
    
    Interface
    
    OpenAPI
  - Data Access Process
    Data Access Process
    
    About
    
    Configuration
    
    Monitoring
    
    Interface
    
    OpenAPI
  - Interface
  - Header
  - Codes
  - Metadata
  - APIs
    APIs
    
    GetMeta
    
    GetData
    
    QSQL
    
    Ping
    
    SQL
  - OpenAPI
  - Examples
    Examples
    
    S3 Historical Data
  - Troubleshooting
- Storage Manager
  Storage Manager
- Stream Processor
  Stream Processor
  - About Streaming Data
  - Quickstart
    Quickstart
    
    Docker
    
    Kubernetes
  - Writing
  - Running
  - Configuration
  - Examples
    Examples
    
    Static file
    
    Batch S3 ingestion
    
    Streaming Kafka ingestion
    
    Kafka with TLS
    
    PostgreSQL Querying
    
    Stateful operators
    
    Enriching streams
    
    Windowing on event time
    
    Windowing on processing time
    
    kdb+ tick (callback)
  - Concepts
    Concepts
    
    Checkpoints and recovery
    
    Scaling
    
    State
  - Release Notes
- OpenAPI
  OpenAPI
- Dashboards
  Dashboards
  - Overview
  - Introduction
  - Get started
  - Workspace
  - Components
    Components
    
    3D Chart
    
    Accordion
    
    Analyst Visual
    
    Bipartite Chart
    
    Bitmap
    
    Blob
    
    Breadcrumbs
    
    Button
    
    Canvas Chart
    
    ChartGL
    
    Code Editor
    
    Contour
    
    Data Filter
    
    Data Form
    
    Data Grid
    
    Date Picker
    
    Drop Down List
    
    Editable List
    
    External Content
    
    Financial Chart
    
    Flex Panel
    
    Form Builder
    
    Graph
    
    Layout
    
    Map
    
    Mapbox
    
    Pie Chart
    
    Pivot Grid
    
    Playback
    
    Radar Chart
    
    Selection Controls
    
    Range Slider
    
    Scatter
    
    Sunburst
    
    Tab Control
    
    Text
    
    Text Input
    
    Trade
    
    Tree Map
    
    TreeView
    
    Vega Chart
    
    Video
    
    Visual Query Builder
  - Data sources
  - View States
  - Actions
  - Highlight Rules
  - Publishing
  - Templates
    Templates
    
    Overview
    
    Arrays
    
    Comparison
    
    Dates
    
    Math
    
    Misc
    
    Numbers
    
    Strings
    
    Utils
  - SDK
    SDK
    
    Introduction
    
    Basics
    
    Data Sources
    
    View States
    
    API Reference
    API Reference
    
    ChangeSetInterface
    
    ComponentInterface
    
    DataModelInterface
    
    ErrorInterface
    
    PagingInfoInterface
    
    ViewModelInterface
  - Theme
    Theme
    
    Style
    
    Custom logo
  - Support
    Support
    
    Release notes
    
    Bug Reporting
- Assembly
  Assembly
  - Assembly configuration
  - Tick based example
- Machine Learning
  Machine Learning
  - About
  - Quickstart
    Quickstart
    
    Docker
    
    Kubernetes
  - Examples
    Examples
    
    Model Generation & Deployment
Platform
Platform
- About
- Release notes
  Release notes
  - 1.1.2 (Latest)
  - Previous Releases
    Previous Releases
    
    1.1.1
    
    1.1.0
    
    1.1.0
    
    0.10.0
    
    0.9.0
    
    0.8.0
  - Changes on 2022-07-15
  - Changes on 2022-06-28
  - Changes on 2022-06-07
  - Changes on 2022-03-14
  - Changes on 2022-02-01
  - Changes on 2021-12-09
  - Changes on 2021-10-27
- Architecture
- Infrastructure
  Infrastructure
  - Overview
  - Cloud Provider
    Cloud Provider
    
    GCP
    
    AWS
    
    Azure
- Install Platform
  Install Platform
  - Interacting with Kubernetes
  - DNS setup
  - Installing the CLI
  - Installing Insights
  - Upgrading Insights
  - Advanced
    Advanced
    
    Manual Install
- Security & authorization
  Security & authorization
  - Authentication
  - Client Registration
- Building Workloads
  Building Workloads
- Building Workloads (UI)
  Building Workloads (UI)
- Ingesting data
  Ingesting data
  - Overview
  - Import from UI
  - Creating Pipelines in UI
    Creating Pipelines in UI
    
    Overview
    
    Readers
    
    Writers
    
    Encoders
    
    Functions
    
    Transform
    
    Decoders
    
    Windows
    
    Machine Learning
  - Using KX SDKs
  - Batch S3 ingestion
  - Kafka
  - Kafka with TLS
  - PostgreSQL query
  - Diagnostics in UI
- Persisting data
  Persisting data
  - Reference Data
  - Object Storage
- Querying data
  Querying data
- Analyzing data
  Analyzing data
  - Batch S3 ingestion and preprocessing
  - Machine Learning Clustering on Kafka ingest
- Visualize data
  Visualize data
  - User Interface
  - Designing Reports
- Monitor
  Monitor
- Configuration
  Configuration
  - Overview
  - Global Configuration
    Global Configuration
    
    Configuring license secrets
    
    Providing custom image values
    
    Discovery configuration
    
    Kubernetes ingress configuration
    
    Setting up metrics
    
    Global Side Car configuration
    
    Global logging configuration
    
    Configuring Scaling
    
    Image Security Context
    
    Additional Configuration
    
    RT Stream Archival
  - Service Configuration
    Service Configuration
    
    Client Controller (RT ingestion)
    
    Keycloak (Authentication Service)
    
    Information service (Ingestion discovery)
    
    KXI Controller (Assembly Management)
- Command line interface
  Command line interface
  - Configuration
  - Reference
- REST API
  REST API
  - Reference
- Reliable Transport
  Reliable Transport
- SDKs
  SDKs
  - Installing the Java SDK
  - Installing the ODBC driver SDK
- Azure Marketplace
  Azure Marketplace
- Guided Walkthrough
  Guided Walkthrough
  - About
  - Ingest & Deploy
    Ingest & Deploy
    
    Build Database
    
    Object Storage (Weather)
    
    Kafka (Subway)
    
    SQL Database (Health)
    
    Protocol Buffer (Crime)
  - Report
    Report
    
    Build Dashboard
    
    Maps
  - Troubleshooting
APIs
APIs
- Overview
- OpenAPI
  OpenAPI
- Stream Processor
  Stream Processor
  - Overview
  - q Interface
    q Interface
    
    Overview
    
    Configuring Operators
    
    General
    
    Lifecycle
    
    Operators
    
    Data Structures
    
    Readers
    
    Decoders
    
    Encoders
    
    Data Transforms
    
    Stats
    
    State
    
    Windows
    
    Writers
    
    Machine Learning
  - Python Interface
    Python Interface
    
    Overview
    
    General
    
    Lifecycle
    
    Operators
    
    Readers
    
    Decoders
    
    Encoders
    
    Data Transforms
    
    State
    
    Windows
    
    Writers
    
    Machine Learning
  - OpenAPI
    OpenAPI
    
    Coordinator
    
    Controller
    
    Worker
- Machine Learning
  Machine Learning
  - About
  - q Interface
    q Interface
    
    Analytics
    Analytics
    
    About
    
    ML Analytics API
    ML Analytics API
    
    Introduction
    
    ML Toolkit
    
    Online Models
    Online Models
    
    Introduction
    
    Stochastic Gradient Descent
    Stochastic Gradient Descent
    
    Stochastic Gradient Descent
    
    Linear Regression
    
    Logistic Classification
    
    Secure Updates
    
    Sequential K Means
    
    Variadic Functionality
    Variadic Functionality
    
    Introduction
    
    Function Calls
    
    Clustering models
    
    Statistical models
    
    Time series models
    
    Online models
    
    Registry
    Registry
    
    About
    
    Cloud Integration
    
    Registry API
    Registry API
    
    Storing
    
    Loading
    
    Deleting
    
    Examples
    Examples
    
    Basic Examples
  - Python Interface
    Python Interface
    
    About
    
    Registry
    Registry
    
    About
    
    Cloud Integration
    
    Registry API
    Registry API
    
    Storing
    
    Loading
    
    Deleting
    
    Examples
    Examples
    
    Basic Python Examples
Licensing
Licensing
- About
- Installation
- Quick start
- Licensing workflow
- Service accounts
- Usage data
- Access control
- Walkthroughs
  Walkthroughs
  - Containers
  - Kubernetes
- Appendix
  Appendix
  - Identity
  - Offline enrolment
Extras
Extras