A unified access point for data spread across multiple processes
The Service Gateway (SG) provides a unified access point for data spread across multiple processes, here referred to as data access processes (DAPs). DAPs can be user-defined q processes or processes within the KX Insights Data Access microservice.
To invoke a publicly exposed API, the Service Gateway component
- provides the protocols and interfaces to compose and issue the request
- oversees all activities related to the fulfilment of the request until a response can be presented to the caller (even if indicating that request was unable to be fulfilled)
You can also use Service Gateway to offer such a service, providing similar interfaces, protocols and configuration definitions to make it available to other components of the system.
Data in the system is dispersed across different locations. The set of processes with access to particular data is located at some arbitrarily distant positions in the system architecture. Over time, that set of processes will change, since processes will come and go, and data will drift through various storage tiers. Similarly, the ability to perform certain activities may not be defined just by locale of data, but also by the locale of the runtime logic required to perform the activity.
The Service Gateway component and associated elements abstract away this complexity by defining concepts and interfaces by which a service provider can describe the functional and data access requirements for the request, including mechanisms to allow the details and context of a particular request instance to contribute to the expression of these requirements.
At a high level, the Service Gateway component is responsible for:
- Accepting and validating service requests
- Identifying the locale for service execution based on metadata discernible from a combination of previously known configuration information, dynamic system topology information, and data contained in the request instance itself (notably its arguments and potentially other aspects of accompanying out-of-band information)
- Selecting one or more specific processes to execute the request
- Shepherding the request to the selected target process(es) and triggering execution
- Triggering any aggregation of partial results from multiple execution processes where appropriate
- Shepherding results back to original caller
Data may be distributed across DAPs temporally (e.g. RDB, IDB, HDB) or along user-defined labels (e.g. region, asset class) or a combination thereof. The SG
- provides a portal for making data-retrieval API calls
- routes to the appropriate DAPs
- aggregates the results
- and returns them to the client process that issued the request
The process architecture is as follows. (Only arrows representing communication with external processes are shown.)
|Client||Client process (external). Connects to Gateway (GW) process to issue API calls.|
|GW||Gateway (internal). Receives API requests from clients, forwards to appropriate DAPs, returns results.|
|RC||Resource Coordinator (internal). Makes routing decisions based on DAP data purviews and availability.|
|Agg||Aggregator (internal). Aggregates responses from DAPs.|
|DAP||Data Access Process (external). Has access to data.|
Unless otherwise specified, the arrows in the diagram represent asynchronous q IPC communication between processes. Query flow is as follows.
|1||Client makes API request to a GW. This can be sync or async.|
|2||GW sends request to the appropriate DAPs.|
|3a||Each DAP, upon completion, returns response to the Aggregator designated for the request.|
|3b||Each DAP, upon completion, notifies the RC that it is available to service requests.|
|4||GW sends response back to client.|
The SG also offers a REST interface. This adds two new processes to the architecture.
The REST processes are enabled and disabled via configuration.
In the beta the REST interface supports only pre-configured APIs.
If using the SG with the KX Insights Data Access (DA) microservice, there are no other requirements. Otherwise, the DAPs require implementations of the APIs that match the DA’s API names and return types.
A DAP's purview is a description of the data it provides, both temporally and along user-defined labels.
Every request specifies a temporal range and labels of interest. The SG partitions the request across DAPs that can satisfy it, aggregates the results from each, and returns the consolidated response to the client that made the call.
With the exception of the pre-configured APIs, the only supported aggregation function is
Future versions will have more.