Within the kdb Insights Database, a set of query processors (known as Data Access Processors) are available, each holding a particular temporal range of data from real-time to historical for a particular set of labels. A Data Access Process responsible for a particular temporal range may have multiple redundant copies for both availability and query processing throughput.
Data within the kdb Insights Database can be queried through REST or QIPC, SQL, or via custom user-developed APIs. In all cases, queries take the form of a distributed map-reduce. The Gateway splits the request and distributes partial requests to available Data Access Processes that can service portions of the requested time range of the query. All Data Access Processes that are participating in the query process their segment and are immediately available for processing further requests. When a partial request is processed, it is sent to an available Aggregator. Once an aggregator receives all partial requests required to satisfy a query, the Aggregator reduces the partial requests into the final response, performing any necessary combination and aggregation of the partial requests.
The Aggregator will hold all partial responses in memory, and perform aggregation in memory. When writing queries, the map-reduce nature of the query-processing should be kept in mind to avoid running into out-of-memory errors in Data Access Processes (prior to aggregation) or within Aggregators (while collecting pre-aggregated responses).
All query responses contain a metadata header as the first element of the response with information regarding the execution of the query. In particular, the following fields of the metadata header help with observing query errors:
logCorr- all processes will log their actions for this query with this log correlator, allowing logs to be searched effectively
ai- a human-readable error message indicating the nature of the query problem
rc- a response code, amenable to automated queries to trigger appropriate actions
ac- an application code, amenable to automated queries to trigger appropriate actions
For successful queries, the
ac will both be
0, and the second element of the response will be populated with the query result.