Internal process events¶

All Refinery processes use an 'event' library to allow custom functions to be executed when a specific event occurs.

Note

All events are local to the process only. See Triggering Remote Events below for details on how to trigger events on remote processes

There are a number of built-in events that can be used and custom events can be triggered as well (e.g. between domain specific and client specific code). You can bind a function to an event with .event.addListener:

q) .notify.func:{ -1 "Notified with arg: ",.Q.s1 x };
q) .event.addListener[`test.event; `.notify.func];
q) .event.fire[`test.event; .z.p];

The handlers for all events in a current process can be seen with .event.handlers. More detail can be found here.

Note

The handler function must be a function reference to a predefined function on the process.

.event.addListener will throw an FunctionDoesNotExistException if the function is not defined

Note

The handler function must accept the number of arguments as defined in the documentation

Boot events¶

The following events are available to every Refinery process (refinery_process) during the boot procedure. If any handler function in these events throws an exception, it will be thrown and the process will fail to start up.

Event name	Provided arguments	Description
`process.init.pre`	Dictionary	After all the process code has been loaded but before the initalisation function for the process type is run. The dictionary contains the following keys: * `procType`: The process type * `pipeline`: The name of the pipeline * `pipelineInstance`: The instance of the pipeline
`process.init.post`	Dictionary	After the process initialisation function has run. The dictionary contains the following keys: * `procType`: The process type * `pipeline`: The name of the pipeline * `pipelineInstance`: The instance of the pipeline

Core events¶

The following events are available to every Refinery-based process (process type refinery_process):

Event name	Provided arguments	Description
`port.open`	Integer (Handle)	An 'event' interface to the Delta `.ch.o` (handle open)
`port.close`	Integer (Handle)	An 'event' interface to the Delta `.ch.c` (handle close)
`process.exit`	Integer (Exit Code)	An 'event' interface to the Delta `.ch.exit` (on process exit)
`process.disconnect`	Symbol (Process Name)	Similar to `port.close` but provides the name of the process instead of the handle

Tickerplant client events/functions¶

Any process which is a client of the Tickerplant (loads the tpClient library) has the following events:

Event name	Provided arguments	Description
`rollover.start`	Timestamp (EoD Time)	When the end-of-day is triggered by an upstream feed handler or on the TP timer
`tp.connection.lost`	Dictionary	When an upstream Tickerplant connection is lost. The dictionary contains the following keys: * `tpName`: The name of the Tickerplant lost * `handle`: The handle that was just closed * `subTables`: The list of tables that were subscribed to that TP * `subResult`: The original subscription result from the TP

In addition to the events above that bind into the event hook framework, there is one additional hook outside of the event handlers that can be defined that enables custom upd functions.

By adding a definition of this function, it will overwrite the default upsert

Function name	Provided arguments	Description/Objective
`.tpClient.upsertFunc`	Table & Data ([t;d])	When looking to change the default upsertFunc (`upsert`) to manage further changes to the data/table

Note

.tpClient.upsertFunc does not use the event framework for custom update functions in an effort to minimise any code execution overhead in high frequency updates.

The process types that subscribe to the TP by default:

rdb
ipdb/epdb
ctp
rte

HDB events¶

Event name	Provided arguments	Description
`hdb.reload.pre`	Symbol (HDB Path)	Just before the HDB is reloaded from disk
`hdb.reload.failed`	Dictionary	If the HDB reload fails. The dictionary contains the following keys: * `path`: The root of the HDB * `error`: The exception thrown `system "l"`
`hdb.reload.complete`	Symbol (HDB Path)	Once the HDB has been reloaded successfully

The HDB also expects the following event to be triggered from an external source (generally the EPDB):

Event name	Provided arguments	Description
`rollover.complete`	Timestamp (EoD Time)	Sent from the EPDB to notify that the on-disk HDB data has changed

RDB events¶

Event name	Provided arguments	Description
`rdb.eod.clear.pre`	Timestamp (EoD Time)	Before yesterday's data is cleared from the RDB
`rdb.eod.clear.post`	Timestamp (EoD Time)	After yesterday's data has been cleared from the RDB
`rdb.intraday.clear.post`	Dictionary	Called before the RDB performs an intraday clear (only when running alongside an IDB). The dictionary contains the following keys: * `lastTimes`: A dictionary of table names and the time to delete up to * `readFrom`: The latest on-disk path for the IDB to use

The RDB also expects the following event to be triggered from an external source (generally the IPDB):

Event name	Provided arguments	Description
`rollover.complete`	Timestamp (EoD Time)	Sent from the EPDB to notify that the RDB can clear yesterday's data
`intraday.complete`	Dictionary	Sent from the IPDB to notify the RDB to clear some intraday data (only when running alongside an IDB). The dictionary contains the following keys: * `lastTimes`: A dictionary of table names and the time to delete up to * `readFrom`: The latest on-disk path for the IDB to use

IDB events¶

The IDB shares all the same events as the HDB above. Additionally, the IDB also expects the following event to be triggered from an external source (generally the IPDB):

Event name	Provided arguments	Description
`intraday.complete`	Dictionary	Sent from the IPDB to notify the IDB to switch the partition it should load data from. The dictionary contains the following keys: * `lastTimes`: A dictionary of table names and the time to delete up to (only used for the RDB) * `readFrom`: The latest on-disk path for the IDB to use

IPDB/EPDB events¶

Event name	Provided arguments	Description
`pdb.rollover.status`	Dictionary	Called through the rollover process for status information. The dictionary contains the following keys: * `state`: One of `started`, `reloading` or `complete` * `eodDate`: The date being processed * `timeTaken`: The total rollover time, only when `state` is `complete`

The IPDB/EPDB triggers the following events:

rollover.complete on all HDBs, IDBs and RDBs in the same pipeline once persistence is complete
intraday.complete on all IDBs and RDBs in the same pipeline once intraday write is complete and only if there are IDBs in the pipeline

UDF processor events¶

Event name	Provided arguments	Description
`udf.reload`	Symbol (Triggering Process)	Causes the UDF processor to reload all UDFs from disk

The UDF processors will notify other UDF processors when they process any API call that affects the on disk structure (e.g. addUDF or deleteUDF)

GW events¶

Event name	Provided arguments	Description
`gw.pre.processing`	Dictionary	Argument pre-processing before GW query dispatch. The dictionary contains the following keys: * `func`: The executing function * `args`: The supplied arguments. Any function added into this event must return a dictionary of the same format as the input. The final `args` value will be used.
`gw.routing`	Dictionary	GW routing logic. The dictionary contains the following keys: * `route`: The route determined so far * `func`: The executing function * `args`: The supplied arguments. Any function added into this event must return a dictionary of the same format as the input. The final `route` value will be used.
`gw.post.processing`	Dictionary	Downstream response post-processing before return to the client. The dictionary contains the following keys: * `func`: The executing function * `args`: The supplied arguments * `responses`: Dictionary of process names and returned value * `result`: Result so far. Any function added to this event must return a dictionary in the same format as the input.
`gw.query.response`	Dictionary	Status of a completed query

Triggering remote events¶

As there is no event bus in Refinery core, the recommended communication method is now direct point-to-point handles firing events on the target processes. The low-level function that can be used for this is .proc.asyncOneShotWithTimeout:

/ Connect, wait and fire 'rdb.event' with args '1 2 3' on all local pipeline RDBs
q) pipelineRdbs:.proc.pipeline.self.getProcesses`rdb;
q) .proc.asyncOneShotWithTimeout[; 0D; (`.event.fire; `rdb.event; 1 2 3)] each pipelineRdbs;

End-of-day event flow¶

The end-of-day rollover is managed by a number of the events described above. The event flow is as follows:

End-of-day occurs
- Triggered by an upstream feed handler or Tickerplant timer
- Tickerplant calls .u.eod function on all subscribers
TP subscribers
- .tpClient.i.rollover is bound to .u.eod
- Fires rollover.start event in the subscriber process
On rollover.start
- RDB, RTE, CTP do nothing on receipt of this event by default
- IPDB starts its rollover
EPDB
- Fires rollover.complete on all process types defined by .pdb.cfg.sendRolloverTo
On rollover.complete
- HDB: Reloads the on disk data
- RDB: Deletes all data prior to the EoD time

Custom event handler¶

The 'event' library can be used for any custom code to separate concerns between the event generation and event processing. The following functions are available for use:

.event.fire[event; args]
- The default event trigger, firing the event with the args
- Any exceptions that occur within the handler functions are suppressed and do not affect other handler functions
.event.fireWithException[event; args]
- Any exception within the handler functions will be thrown
- All handlers are executed before the exception is thrown
.event.fireWithResults[event; dArg]
- Allows results to be passed between the handler functions bound to the event
- The result from the final handler will be returned to the caller
- All exceptions will be thrown
- args must be a single dictionary argument