Querying
The Service Gateway provides query access to the kdb Insights Database. Queries can be called via REST or QIPC, SQL, or through custom APIs.
Within the kdb Insights Database, a set of query processors (known as Data Access Processors) are available, each holding a particular temporal range of data from real-time to historical for a particular set of labels. A Data Access Process responsible for a particular temporal range may have multiple redundant copies for both availability and query processing throughput.
Map-reduce
Data within the kdb Insights Database can be queried through REST or QIPC, SQL, or via custom user-developed APIs. In all cases, queries take the form of a distributed map-reduce. The Gateway splits the request and distributes partial requests to available Data Access Processes that can service portions of the requested time range of the query. All Data Access Processes that are participating in the query process their segment and are immediately available for processing further requests. When a partial request is processed, it is sent to an available Aggregator. Once an aggregator receives all partial requests required to satisfy a query, the Aggregator reduces the partial requests into the final response, performing any necessary combination and aggregation of the partial requests.
Note
The Aggregator will hold all partial responses in memory, and perform aggregation in memory. When writing queries, the map-reduce nature of the query-processing should be kept in mind to avoid running into out-of-memory errors in Data Access Processes (prior to aggregation) or within Aggregators (while collecting pre-aggregated responses).
Query responses
All query responses contain a metadata header as the first element of the response with information regarding the execution of the query. In particular, the following fields of the metadata header help with observing query errors:
logCorr
- all processes will log their actions for this query with this log correlator, allowing logs to be searched effectivelyai
- a human-readable error message indicating the nature of the query problemrc
- a response code, amenable to automated queries to trigger appropriate actionsac
- an application code, amenable to automated queries to trigger appropriate actions
For successful queries, the rc
and ac
will both be 0
, and the second element of the response will be populated with the query result.