Manual EOD trigger

This page describes how to perform a manual end of day (EOD) trigger during the day, that may be necessary in certain situations.

kdb Insights automatically runs a full EOD writedown, however, you can also manually trigger any number of additional "partial" EODs to write to HDB all the data ingested so far, without waiting until the end of day. These manual EODs may be needed in order to prevent the HDB processes from running out of memory if a substantial amount of late data is ingested during the day.

Issue an EOD writedown request

An EOD writedown request can be issued by making the POST eod REST call on the Storage Manager.

Admin roles only are able to perform a manual EOD writedown

kdb Insights hostname and database name

In the examples below, $INSIGHTS_HOSTNAME is the kdb Insights hostname, $INSIGHTS_TOKEN is the access token, and $KX_DATABASE_NAME is the name of the database to which the EOD request should be issued.

curl -X POST \
    -L "https://$INSIGHTS_HOSTNAME/servicegateway/api/v1/database/$KX_DATABASE_NAME/eod" \
    -H "Authorization: Bearer $INSIGHTS_TOKEN"

Example responses

If the EOD request is accepted, the response has the following form:

{
    "status":"pending",
    "date":"2025-03-05",
    "seq":12
}

Status indicators

The requested EOD writedown is assigned a new EOD sequence number seq included in the response, which can be referenced in the EOD status request.

The request moves through the following statuses:

• pending: This status indicates that the Storage Manager has published the end of partial day signal to the stream. This signal first initiates the writedown for the final interval of the given partial day.
• queued: On completion, the EOD task is queued and the status changes to queued.
• processing: Then it is dispatched to the EOD process for writedown, where the status changes to processing.
• completed: If any processing step fails after the request has been accepted, it is attempted again until the writedown succeeds and the status finally changes to completed.

If a new EOD request is issued before the previous request is assigned the queued status, this new request is rejected with the following response:

{
    "status":"rejected"
}

Query the EOD writedown status

The status of any EOD writedown (full or partial) with a known sequence ID $seq can be queried through the GET REST call on the Storage Manager to the endpoint eod/$seq:

curl -X GET \
    -L "https://$INSIGHTS_HOSTNAME/servicegateway/api/v1/database/$KX_DATABASE_NAME/eod/2" \
    -H "Authorization: Bearer $INSIGHTS_TOKEN"

Example success response:

{
    "seq":2,
    "status":"completed",
    "date":"2025-03-05",
    "type":"partial",
    "startTS":"2025-03-04T22:43:34.336378227"
}

Example response if the requested EOD sequence ID does not exist (or if it was completed too long ago):

{
    "seq":2,
    "status":"not found"
}

The status of all recent EODs (up to 100) can be queried via the GET REST call to the eod endpoint on the Storage Manager:

curl -X GET \
    -L "https://$INSIGHTS_HOSTNAME/servicegateway/api/v1/database/$KX_DATABASE_NAME/eod" \
    -H "Authorization: Bearer $INSIGHTS_TOKEN"

The response is a list of properties for each accepted EOD in reverse chronological order:

[
    {
        "seq":3,
        "status":"processing",
        "date":"2025-03-05",
        "type":"partial",
    },
    {
        "seq":2,
        "status":"completed",
        "date":"2025-03-05",
        "type":"partial",
        "startTS":"2025-03-04T22:43:34.336378227"
    },
    {
        "seq":1,
        "status":"completed",
        "date":"2025-03-04",
        "type":"full",
        "startTS":"2025-03-04T01:23:45.678901234"
    }
]

Performance considerations

Any subsequent EOD writedowns triggered during the same day and the final full EOD writedown for that day are likely to write the data to the same HDB partition. The Storage Manager needs to merge the data, resort and re-apply the attributes each time (similarly to the late data processing at regular EODs). Therefore, the total time taken by the Storage Manager to write all data that arrived during a given day increases if it is done in multiple smaller portions (triggered by multiple manual EOD requests).