Non-time series data

First introduced in the Refinery 5.6.0, this feature allows you to set up a pipeline which can ingest non-time series data. You achieve this by adding a parameter to the pipeline configuration to enable non-time series data to be used within the system.

allow-non-timeseries-data: true

The next step is to have a table configuration that DOESN'T contain a designated time-col. This is because a non-time series table doesn't have a time column to designate. Within the Process Manager configuration, the PM sees that this table belongs to the non-time series pipeline and waives the necessity of this parameter.

Warning

Standard Refinery tables aren't supported in non-time series pipelines, only use non-time series tables.

Example non-time series data pipeline

This is an example of a historical only pipeline.

Note

The addition parameter has been added to the pipeline configuration.

pipeline:
  name: nonTSdataPipeline
  type: realtime

  expose-to-gw: true
  allow-non-timeseries-data: true

  taxonomy:
    region: global
    data-source: nonts

  proc-layout:
    -
      all: all-processes

  processes:
    tp:
      pub-mode: timer
      pub-freq-ms: 100
      log-to-journal: true
      rollover-mode: daily-at-time
      rollover-time: "00:00:00.001"
      enable-analyst: true
    hdb:
      timeout: 30
      enable-analyst: true
    ipdb:
      write-freq: 60000
      write-row-limit: 0
      enable-analyst: true
    epdb:
      enable-analyst: true
      timeout: 0

Example non-time series data table

This is an example of a basic sym-price-volume table.

Note

The omission of the time-col parameter in the table configuration.

table:
  name: nonTSdataTable
  id-col: sym
  intra-persist-type: splay
  end-persist-type: date-partition

  taxonomy:
    -
      region: global
      data-source: nonts

  columns:
    -
      name: sym
      data-type: symbol
      attribute: grouped
    -
      name: price
      data-type: float
    -
      name: volume
      data-type: long

Note

Despite not having a time column, the data can still be date-partitioned; the system will handle this and persist it alongside temporal tables. For larger tables this is the recommended end-persist-type.

Loading non-time series data

The process of loading in data is identical to loading in static data with no time column.

Querying non-time series data

As there is no time column to query against, you don't require the startDate or endDate parameters in your API arguments. The requirement is waived in the same way that the time-col is waived in the table config above. You can query the non-time series data via both the getTicks and getStats calls. These API queries are made from within the gw-client.

Non-time series getTicks API call

.gwClient.query.sync[`getTicks; `dataSource`dataType`idList!(`nonts; `nonTSdataTable; `)]

Non-time series getStats API call

.gwClient.query.sync[`getStats; `idList`dataSource`dataType`analytics ! (`; `nonts ;`nonTSdataTable; (avg;`price));]

Note

getStats will only aggregate the entire table as there is no time column.