FAQs about upgrading to Refinery 5¶

Q1. How do you add an instance to an already running pipeline?¶

For the new instances to be picked up by the gateway (DefaultEntrypoint Gateway), the pipeline must be bounced against the gateway. The gateway doesn't receive live updates from the Process Manager as it gets system info on startup. Generally you wouldn't update settings during the workday. If you have two gateways, you can restart one and then the other to allow the gateways to register to newly added instances without interrupting any queries.

Gateways send queries asynchronously. It uses deferred sync which can appear synchronous to the client application, but in the back-end is completely asynchronous. The gateway is never blocked waiting on queries nor are any downstream processes blocked while waiting on another process.

There are no issues with having multiple gateways, but there is little need for them as they are very lightweight in the system. Alternatively, there are gateway clients, which are q processes that sit above the gateway; these are used for dashboard processes that dispatch queries to the system similar to a sandbox.
Gateway clients are usually run in large clusters to allow dashboards to connect to them in round-robin sequence such that if one gw_client is being used for a query the next available gw_client gets used instead. 10+ gw_clients would happily sit above a single entrypoint-gw.

RealtimeUDF pipeline is the data pipeline for the results of REALTIME_UDF_A/B. The REALTIME_UDF_A/B will start up worker nodes, etc; the output is published to the RealtimeUDF pipeline Tickerplant. The RealtimeUDF pipeline is just like any other data pipeline, but the schema is already defined for RealTimeUDF output.

Currently there are only two pipeline types, entrypoint and realtime. The entrypoint pipeline type is for gateway processes and the realtime pipeline type is for data ingestion. As long as the data loader belongs to a realtime pipeline, then this is the best option for loading in data.

Ideally every file should be in q libraries. Having everything in q makes everything cleaner, easier to make and see changes. It can also improve the performance of the system. Making calls to KxControl to pull config can cause long startup times in large systems.
Pipeline YAMLs need to be created manually, but are straightforward to modify.
Table YAMLs are also created manually, but a script can be written to convert the platform schemas.
For analytics, a simple script can be written where the function definitions are pulled and saved into a q script; otherwise, they can be modified manually.
Config and overrides are best migrated to either q scripts in a table format or saved as a CSV file in the state-config directory of Refinery.
However, you don't have to migrate the analytics and config files to q scripts. All the legacy .al.load[] or .uc.get[] functions still work, but the benefits of migrating to a better structured system are not going to be available.

When talking about the order of execution of init functions, etc., a mixture of event hooks and .require.libNoInit can be used to control order of definition/execution of functions on start-up.
When converting platform schemas to q scripts, make sure to not map the analytics 1-to-1; this will create a mass of far too many q scripts. It is suggested to group any shared namespace/analytic group function definitions together into different libraries; e.g. .tp.get[] and .tp.set[] are in different analytics but are written in the same library.

Event buses are not in Refinery 5 core due to the objective of decentalizing Refinery 5, reducing single points of failure and making information transfer from point-to-point. However, they do exist in the finance domain to support legacy functionality. It is suggested that any enhancements made that require information exchange use the .proc library to manage connections and all communication is asynchronous. -25! should be used for any broadcast.

Function names can (and should) be named with dot notation; e.g. .myLibrary.component.doFunction.