Skip to content

REST-server library

The REST-server library can be used to expose REST-style facade to a new or existing KDB+ based system.

In a nutshell, it allows the developer to map operation/path combinations (endpoints) to q functions (handlers). Then when a HTTP request is received it matches it to an endpoint, and when one is found it processes the request according to its definition, and executes its handler.

The operation is arbitrary, but typically one of HTTP methods defined by REST specification: GET, POST (create), PUT (update), or DELETE.

The path is the part of the URL that comes after the host and port (e.g. /customers). It identifies the resource subject of the operation. The path is arbitrary and may contain variables (which are treated as parameters to the endpoint), for example /db/{table}/schema is a valid path that has a variable named table.

Usage

To use the library, the developer performs the following 2 (and 1 optional) steps: 1. Initialize the framework, using .com_kx_rest.init function. 2. Register the endpoints, using .com_kx_rest.register function. 3. On receipt of a request to .z.ph or .z.pp, forward execution to .com_kx_rest.process to process the incoming request and return processing result.

The following is an example: To simplify the examples, we assume .rest namespace exists which aliases .com_kx_rest:

.rest:.com_kx_rest; / Alias namespace for convenience - typically done once at beginning of file

.rest.init[enlist[`autoBind]!enlist[1b]]; / Initialize

//
// Register endpoints.
//
.rest.register[`get;"/customers";
 "Returns all customers";
 .db.getAllCustomers;
 .rest.reg.data[`i;-6h;0b;0;"Offset to first row"],
 .rest.reg.data[`cnt;-6h;0b;10;"Number of rows to return"]
 ];

.rest.register[`get;"/customers/{id}";
 "Returns one or more customers by their IDs";
 .db.getCustomersById;
 .rest.reg.data[`id;6h;1b;0;"One or more customer IDs"]
 ];

In the above example: we initialized the framework with autoBind option set to 1b, then created the following 2 endpoints: 1. get /customers returns all customers. It expects 2 parameters: i (integer) is the offset of first row to return, and cnt (integer) is the number of rows to return. Both parameters are optional, and have default values of 0 and 10 respectively. The endpoint is mapped to .db.getAllCustomers function which can possibly be defined as follows:

.db.getAllCustomers:{
 x[`arg;`cnt]#select from customers where i>=x[`arg;`i]
 }
  • The above endpoint can be queried as follows:
curl 'localhost:8080/customers'
curl 'localhost:8080/customers?i=10&cnt=10'
  1. get /customers/{id} returns one or more customers given their IDs. It expects one parameter: id (integer[]) is an input parameter in the form of a path variable (whose value is determined at matching time). The endpoint is mapped to .db.getcustomersByIdfunction which can possibly by defined as follows:
.db.getCustomersById:{
 select from customers where id in x[`arg;`id] //! should be in ID order
 }

Initialize the REST server framework

The framework is initialized using .com_kx_rest.init function. This function is overloaded such that it can be called niladically (e.g. .com_kx_rest.init[]) to perform minimal initialization, or monadically (with opts dictionary argument) to further perform specified steps (e.g. auto-binding to .z.ph and .z.pp in the above example).

When autoBind option is specified as 1b, an incoming request is delegated to the next handler (if applicable) if it cannot be matched to an endpoint. Conversely if autoBind is not specified (or specified as 0b), then a request that doesn't match an endpoint is rejected with 404 HTTP code.

Note that when auto-binding is disabled, then it is the application's responsibility to bind .com_kx_rest.process to .z.ph and .z.pp, as well as performing any chaining as needed.

Register the Endpoints

An endpoint is registered using .com_kx_rest.register function. The registration contains the following components: - The operation and path of the endpoint - A description summarizing the purpose of the endpoint - The handler function - Optional definition of user input, which includes: path variables, query string, headers, and request body (for post/put based endpoints). - Optional definition of result object

The operation is arbitrary, but typically is one of the following: get (for data retrieval), post (to create one or more objects), put (to update one or more objects), or delete (to delete an object).

The path is arbitrary, and may contain variables whose values are determined at matching time (e.g. /db/{tbl}/schema). The operation and path combination uniquely identifies the endpoint.

The handler function is responsible for performing the activity of the endpoint. It is explained in Request Processing section.

User input definitions specify the properties of expected input parameters which include path-variables and query-string (using .com_kx_rest.reg.data), headers (using .com_kx_rest.reg.header), and JSON based request body (using .com_kx_rest.reg.body). For example the request/db/{x}/{y}/z?i=0&cnt=10has the following input parameters:x,y,i, andcnt`. The developer can specify the name, data type, necessity, default value, and description of each parameter. The framework uses this information to: - Ensure that required parameters are supplied in the request, failing the request if any is missing. - Parse value using the expected data type.

The handler also has access to raw user input, which is useful when some or all of the input is not known at registration time.

Output definition specifies the schema of the result (using .com_kx_rest.reg.output). The definition is currently not used by the framework, but it is a good practice to specify it for future purposes. For example, a future version will support generating openAPI documents based on the registrations.

Request Processing

A HTTP request contains the following components: - HTTP method - Path - Query string - HTTP headers - Request body (applies to POST, PUT, and DELETE)

HTTP method generally specifies the REST operation, but in KDB+ it is limited to GET and POST. Thus, the framework looks for the operation in http-method HTTP header, and if not present falls back to GET (for .z.ph), or POST (for .z.pp). It is thus important to use a front-end API gateway (e.g. AWS HTTP API gateway) that can populate http-method HTTP header, and convert PUT, and DELETE HTTP methods to POST, leaving GET requests as is.

When a request is received, both operation and path are combined to find a matching endpoint, favouring exact matches over ones containing variables (e.g. /a/b/c vs /a/{x}/c).

User input is then processed, distinguishing between the following categories:

  • User input specified using .com_kx_rest.reg.data function. These contain path variables (e.g. id in /customers/{id}) and query-string parameters. Values are parsed according to their data-type. If a parameter is missing from the request, then its default value is used. But if the missing parameter is marked required, then the request fails with 400 HTTP status code (with names of missing required parameters included in the response). Values of input parameters are passed to the handler function as dictionary under arg key. it is worth noting that raw input parameters (as they appear in the request) are also passed to the handler function under rawArg key.

  • Request body (where applicable) is expected to be in JSON format. It is deserialized into a KDB+ data structure that using .j.k, and passed to the handler function under data key.

  • Request body (in case of post or put operation) is expected to be in JSON format. If the endpoint has body input specified using .com_kx_rest.reg.body function, then the elements of the defined object are parsed from the input (including nested objects) according to their datatypes and necessity settings. Similar to input parameters, the raw body is passed to the handler under rawData key.

  • HTTP headers are are passed as is to handler function under hdr key.

After input is collected, the handler function is invoked. If the function is declared to take arguments with same names as the endpoint parameters (or body if endpoint is comprised only of a body parameter), then the function is variadically invoked with its arguments mapped from the request input. Otherwise, the function is monadically invoked with a dictionary containing the following keys: - REST operation of the endpoint - The path of the endpoint - Values of processed input parameters - Raw input parameters present in the request - Value of processed body object, which is typically a dictionary if the body is of an object type, but can be of any type as specified by reg.body function. - Raw KDB+ form of the request body (if present) - HTTP headers

The handler function is expected to return its response in one of the following forms: - A KDB data structure (typically a dictionary or a table), which is serialized to JSON by the framework before being returned to the client - Result of .com_kx_rest.util.response function, which given the handler control over the HTTP status code, and content type of the response (available post release 1.0.0). - Result of .com_kx_rest.util.httpResponse function, which gives the handler total control over the response. - If there is a problem possibly with input, the handler must call .util.throw to signal an error.


API reference is here.