Versions of kdb Insights Enterprise before v1.5 use assemblies; an assembly is comprised of:
- A database to store and access your data.
- A schema to convert imported data to a format compatible with kdb Insights Enterprise (using kdb+ technology).
- A stream to push event data into a database.
- Pipelines to read data from source and write it to a kdb Insights Enterprise database.
Every assembly is labelled with its name.
Build a Database
Build a Database is a step-by-step wizard to create a pre-configured database and schema to store your data on kdb Insights Enterprise. Build a database is accessible from a tile under "Discover kdb Insights Enterprise" on the Overview page.
An open-source, scalable, simplified storage solution for data pipelines.
A software development kit to support C/C++ applications.
"cli" is the command line interface. The cli runs Insights processes and is an alternative to the UI for power users.
The console is where results from ad hoc queries run in the Scratchpad are presented.
The console is part of the Query window.
Dashboards is an interactive visualization tool that runs in your browser. You can query, transform, share and present live data insights. Dashboards is integrated into kdb Insights Enterprise as Views.
A database is a data store built on kdb+ technology. A database offers rdb (real-time database storage), idb (interval data storage) and at least one hdb (historic database storage) - sub-tiers of a hdb may be on the database too.
A database also includes:
- A schema to convert imported data to a kdb+ compatible format.
- A stream to help push event data to the database.
- Optional pipelines to import data to the platform
A step-by-step guide to help you build a database. At the end of a wizard building process you will have a fully functional database to store your data.
Docker is a platform-as-a-service, delivering software to consumers via "containers".
The entity-tree is a dynamic menu, always available in the left margin of the kdb Insights Enterprise user interface. The content of the menu changes depending on the interaction in the platform. On the Overview page, for example, the entity-tree shows a list of databases, pipelines, queries and views. On the pipeline page, the entity-tree lists the nodes used to build data pipelines to import data from source and transform it to a format compatible with a kdb Insights Enterprise database.
A hdb is a mount for storing historic data on a database. A historic database is the final destination for interval data.
An idb is a mount for storing interval data on a database. It takes data from a real-time database (rdb), stores this data for a set period, e.g. 10 minutes, before the data is written to a historic database.
A step-by-step process for building a pipeline to import, transform and write data to your database.
A software development kit for developing applications in Java.
Kdb+ is an ultra-fast time series columnar database.
Keycloak is an open-source, single-sign-on authentication service and management tool. It offers enhanced user security built from existing protocols and can support authentication via social platform providers like Google, Facebook or GitHub.
An open database connectivity driver for connecting kdb+ databases.
Kubernetes is an open-source tool for bundling and managing clusters of containerized applications.
Kurl is an easy-to-use cloud integration, registering Azure, Amazon, and Google Cloud Platform authentication information.
A Label is required by a database. Every created database has a default label,
kxname; additional labels can be added to the database. Labels are a filter option in the query tab.
Machine learning is a branch of artificial intelligence (AI) that focuses on the use of data and algorithms to imitate how people learn, with the goal of improving accuracy.
A mounted database is ready for use; a database can have a hdb, idb and/or rdb mounts.
Nodes are used by pipelines to read, write and transform data from its source to the database.
Each node has a defined function and set of properties to edit. Some nodes allow for
Machine Learning nodes offer more advanced manipulations of imported data before writing to the database.
A data storage system managing data as objects, compared to a file hierachy or block based storage architecture. Object storage is used for unstructured data, eliminating the scaling limitations of traditional file storage. Limitless scale is the reason object storage is the storage of the cloud; Amazon, Google and Microsoft all employ object storage as their primary storage.
Results from a database query are written to an Output Variable. The Output Variable can be queried in the scratchpad using
A package is a storage location for code, metadata and information for describing an application.
When a data table is written to a database it must be partitioned to be compatible with a kdb+ time series database.
Partitioning is handled by a Timestamp column, and defined in a schema. Every table must have a Timestamp column.
Pipelines are a linked set of processes to read data from its source, transform it to a format compatible with kdb Insights Enterprise, then write it to a database for later querying.
Pipelines can be created using the Import Wizard or a visual pipeline builder. The pipeline builder offers a set of nodes to help read, writer or transform data; nodes are connected together in a workspace to form a linked chain of events or pipeline template. Additional machine learning nodes are available for more advanced data interactions.
Pipelines can be deployed individually or associated with a database; pipelines associated with a database will be deployed and activated when the database is deployed.
The Pipeline Template is the layout of the nodes that together make a Pipeline.
Postgres (PostgreSQL), is an open-source relational database management system.
pgwire is a PostgresSQL client library, used to implement a Postgres wire protocol server that connects to kdb Insights Core.
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data - think XML, but smaller, faster, and simpler. Data structure is first defined before specially generated source code read-and-writes structured data, to-and-from a variety of data streams, using a variety of programming languages.
PyKX is the interface between the programming language, q, the time-series columnar database, kdb+, data, and Python.
q is the programming language used to query a kdb+ database.
q/SQL is a collection of SQL-like functions for interacting with a kdb+ database.
Interact with your data by building queries in the Query window. Build queries with filters, or execute with
SQL code. Queries reference the name of the table generated by the pipeline and results written to an Output Variable for use by the Scratchpad.
Ad hoc queries in the scratchpad use
python with the Output Variable; results are outputted to the console as a table or chart.
A reader is typically the first node in a pipeline. It feeds or imports data from an external data source to kdb Insights Enterprise. Data read from an external source needs to be decoded (in most cases), and transformed, before it can written to a kdb Insights Enterprise database.
The kdb Insights Reliable Transport (RT) is a microarchitecture for ensuring the reliable streaming of messages.
REST, Representational State Transfer, is a software architectural style that describes the architecture of the web.
Real-time event data is stored on an rdb mount of the database, before it's written to the interval database (idb).
An RT Bridge is a q process which enables applications outside of Insights which deliver data to a q process, to deliver the same data to Insights without further modification.
An RT stream is the kdb Insights Enterprise deployment of a Reliable Transport cluster.
Service Discovery is the process of automatically detecting devices and services on a computer network, thus reducing the need for manual configuration by users and administrators.
A schema is how data is converted from its source format, to a format compatible with a kdb+ database. Every data table has its own schema.
Scratchpad is part of the Query window. With scratchpad you can make ad hoc queries against an Output Variable generated by a query against a table in the database.
You can also create data tables directly in the scratchpad editor. The scratchpad editor supports
Results from a scratchpad query are presented in the console, or as a table or chart.
kdb Insights Enterprise uses a number of software development kits (SDK) to help you organize your data on the platform.
SDKs are available for
SQL (Structured Query Language) is a standard language for accessing databases, and is supported by kdb+ databases.
Streams is how event data is written to a database. Event data is typically real-time as may be generated by a price or sensor feed.
Real-time data is stored on an real-time database (rdb), moved to an interval database (idb), before the data is written to an historic (hdb) database.
The KX Stream Processor is a steam processing service for transforming, validating, processing, enriching and analyzing real-time data in context.
Some high level text about terraform scripts and how we use them.
A Transform node is required for most pipelines. A transform node takes imported data and transforms it to a kdb+ format suitable for storage on the database.
UI is the User Interface for kdb Insights Enterprise.
Views is how you build visualizations in kdb Insights Enterprise. Views are powered by KX Dashboards technology.
A Writer node is an essential part of any pipeline. The writer nodes takes the transformed (kdb+) data you have read from its source and writes it to a kdb+ database.