Capturing the ingestion timestamp¶

This page explains how to use a customizable lifecycle hook and the storage-manager package entrypoint to record when the Storage Manager receives table data from RT.

This walkthrough uses the table ingestion hooks mechanism to satisfy a common request: an ingestTime column that records when the table row was received by the Storage Manager.

Limitations

This mechanism doesn't apply to batch ingestion. The custom function runs only when the Storage Manager receives a table update from RT.
If the Storage Manager is restarted, it replays the RT messages starting from the last processed end-of-interval, assigning the new ingestion timestamps to the messages that may have been received before the restart.
The ingestion timestamp becomes available for queries only after the next end-of-interval write-down. Therefore, the field is empty when you query records in the RDB. After the write-down, the field contains the time at which the Storage Manager received the row, not the time of the write-down to the IDB.

This page assumes that you have a package called mypkg containing tables you want the ingest time added to.

Add an ingestTime timestamp field to each table of interest.
Create a file called: customReceiveSM.q in the root of the package.
Add a custom function to customReceiveSM.q:

The first line of the file registers the function with .kxdb.addHook[`data.receive.pre;;]. That adds the ingestion timestamp to the incoming table data.

The custom function takes two parameters, table name (symbol) and table data, and must return the modified table data that will be written to the database.
```
.kxdb.addHook[`data.receive.pre;;`.custom.sm.receivePre] each `table1`table2`table3; / Register the handler for each table where the ingestion timestamp needs to be recorded

.custom.sm.receivePre:{[tableName;tableData]
    update ingestTime:.z.p from tableData / Replace the ingested column value (likely null) with the current timestamp
    }
```
Make sure to replace the current list of tables in the first line with the names of your tables.

Note

If you use a name other than ingestTime for the timestamp field, update the function to use that name.
Map the file to the storage-manager entrypoint:
- Declare the custom code file directly in the package's manifest.yaml:
```
entrypoints:
    storage-manager: customReceiveSM.q
```
  Warning
  
  The relative path and file name must match the file you created previously.
- Or, add it with the CLI, which updates the manifest for you:
```
kxi package add --to mypkg entrypoint --name storage-manager --path customReceiveSM.q
```
  Warning
  
  The --path value must match the relative path and file name you created previously.
Checkpoint the updated package before pushing it to kdb Insights Enterprise to give it a higher version number than the deployed package. For example, the following command increments the patch version:
```
kxi package checkpoint mypkg --bump patch
```
If the package is already running in kdb Insights Enterprise, tear it down.
```
kxi pm teardown mypkg
```
Push the updated package to kdb Insights Enterprise and redeploy it.
```
kxi pm push mypkg
kxi pm deploy mypkg
```

Once the package is deployed, the ingestTime column is available to query once an end-of-interval event moves the record from the RDB to the IDB.