Time zone allocation per pipeline¶
As of Refinery 5.6.3, pipelines can be set to a specific individual time zone, rather than inheriting this configuration from the system.yaml. This enables pipelines to rollover with a local time zone, as well as providing underlying configuration for API time zone adjustments to function relative to different ingestion time zones.
Note
These pipelines will run their EOD rollover events off the pipeline's internal clock not the local machine's internal clock.
Time zone configuration¶
Within the pipeline's YAML parameters add:
timezone: EST
demoPipeline:
pipeline:
name: DemoPipeline
type: realtime
expose-to-gw: true
timezone: EST
taxonomy:
region: global
data-source: demo
proc-layout:
-
all: all-processes
processes:
tp:
port: 32121
pub-mode: timer
pub-freq-ms: 100
log-to-journal: true
rollover-mode: daily-at-time
rollover-time: "00:00:00.001"
rdb:
port: 32122
timeout: 30
hdb:
timeout: 30
ipdb:
write-freq: 60000
write-row-limit: 0
epdb:
timeout: 30
idb:
timeout: 300
Time zone query explanation¶
As the system.yaml is in default setup (UTC), we can upload data to our EST pipeline using standard ingestion methods. In the Gateway client, run the following examples to see the difference between EST and UTC times. EST will be time of ingestion vs UTC being time of ingestion +05:00:00.
// Default (pipeline timezone)
getTicks `assetClass`dataType`startDate`endDate`symList!(`demo; `demoTable;.z.D;.z.D;`);
sym time val
-------------------------------------
f 2023.08.30D10:50:00.003340236 1
j 2023.08.30D10:50:00.003340236 2
k 2023.08.30D10:50:00.003340236 3
k 2023.08.30D10:50:00.003340236 4
k 2023.08.30D10:50:00.003340236 5
// EST
getTicks `assetClass`dataType`startDate`endDate`symList`timeZone!(`demo; `demoTable;.z.D;.z.D;`; `EST);
sym time val
-------------------------------------
f 2023.08.30D10:50:00.003340236 1
j 2023.08.30D10:50:00.003340236 2
k 2023.08.30D10:50:00.003340236 3
k 2023.08.30D10:50:00.003340236 4
k 2023.08.30D10:50:00.003340236 5
// UTC (EST + 5hrs)
getTicks `assetClass`dataType`startDate`endDate`symList`timeZone!(`demo; `demoTable;.z.D;.z.D;`; `UTC);
sym time val
-------------------------------------
f 2023.08.30D15:50:00.003340236 1
j 2023.08.30D15:50:00.003340236 2
k 2023.08.30D15:50:00.003340236 3
k 2023.08.30D15:50:00.003340236 4
k 2023.08.30D15:50:00.003340236 5
As this was an example of what happens when the pipeline is updated to be an EST pipeline, let's go through why this is necessary, as if we are working with a pipeline without the time zone defined.
When a pipeline is configured and time zone is not defined, the default time zone for the pipeline will be set to the system.yaml time zone. This means when we do the same ingestion as above, the data when queried through an API will display as such:
// Default (pipeline timezone)
getTicks `assetClass`dataType`startDate`endDate`symList!(`demo; `demoTable;.z.D;.z.D;`);
sym time val
-------------------------------------
f 2023.08.30D10:50:00.003340236 1
j 2023.08.30D10:50:00.003340236 2
k 2023.08.30D10:50:00.003340236 3
k 2023.08.30D10:50:00.003340236 4
k 2023.08.30D10:50:00.003340236 5
// EST (UTC - 5hrs)
getTicks `assetClass`dataType`startDate`endDate`symList`timeZone!(`demo; `demoTable;.z.D;.z.D;`; `EST);
sym time val
-------------------------------------
f 2023.08.30D05:50:00.003340236 1
j 2023.08.30D05:50:00.003340236 2
k 2023.08.30D05:50:00.003340236 3
k 2023.08.30D05:50:00.003340236 4
k 2023.08.30D05:50:00.003340236 5
// UTC
getTicks `assetClass`dataType`startDate`endDate`symList`timeZone!(`demo; `demoTable;.z.D;.z.D;`; `UTC);
sym time val
-------------------------------------
f 2023.08.30D10:50:00.003340236 1
j 2023.08.30D10:50:00.003340236 2
k 2023.08.30D10:50:00.003340236 3
k 2023.08.30D10:50:00.003340236 4
k 2023.08.30D10:50:00.003340236 5
Without the inclusion of a defined time zone for the pipeline, you may think that you're inputting the current EST timestamp, but in fact you're not. Incorporating this will allow your data to be accurately timestamped.