Data Sharding

Use data sharding to improve your data ingestion performance. Before you implement this type of sharding, use RDB sharding and HDB clustering within a single pipeline.

If writing and sorting data is the limiting factor for data volumes in a single pipeline, you can shard and query across multiple pipelines. If the speed of data persistence isn't an issue, you can use instances in the pipeline definition.

Implement data sharding

You implement data sharding by creating a table schema that contains two or more taxonomy elements that expose the table to multiple pipelines. The taxonomies must share the same region and data-source, but must differ in the data-class and sub-class.

Note

The pipelines that share the same table schema must each maintain a distinct feed handler, tickerplant, and so on. They must also store distinct id-col values for the table. For example, pipeline 1 handles id-col values `A -> `M, and pipeline 2 handles id-col values `N -> `Z.

You can query the data in the table schema on each pipeline simultaneously from a gateway client when you enable the multi-pipeline-route parameter in the entrypoint schema for the system.

Sample entrypoint pipeline

pipeline:
  name: "DefaultEntrypoint"
  type: "entrypoint"

  enable-legacy-routing: false
  expose-to-gw: true                   
  expose-to-qr: true
  enable-monitoring: true

  proc-layout:
    -
      all: all-processes

  taxonomy:
    region: all
    data-source: all
    data-class: all
    sub-class: all

  processes:
    gw:
      instances: 1
      enable-analyst: true
      additional-q-libraries:
       - gwClient
      multi-pipeline-route: true
    udfp:
      instances: 1
      enable-analyst: true

In the following example, the streamTab table schema details the taxonomy of the two pipelines that it exposes the table to.

Sample table schema

table:
  name: streamTab
  id-col: sym
  time-col:  time
  intra-persist-type: splay
  end-persist-type: date-partition

  taxonomy:
    -
      region: global
      data-source: stream
      data-class: stream
      sub-class: stream
    -
      region: global
      data-source: stream
      data-class: stream2
      sub-class: stream2

  columns:
    -
      name: time
      data-type: timestamp
      attribute: sorted
    -
      name: sym
      data-type: symbol
      attribute: grouped
    -
      name: price
      data-type: float
    -
      name: volume
      data-type: long

First sample pipeline

pipeline:
  name: stream
  type: realtime

  expose-to-gw: true

  taxonomy:
    region: global
    data-source: stream
    data-class: stream
    sub-class: stream

  proc-layout:
    -
      all: all-processes

  processes:
    tp:
      pub-mode: timer
      pub-freq-ms: 100
      log-to-journal: true
      rollover-mode: daily-at-time
      rollover-time: "00:00:00.001"
      subscribe-from-delta-messaging: true
      enable-analyst: true
      port: 41221
    rdb:
      timeout: 0
      enable-analyst: true
      instances: 2
    hdb:
      timeout: 0
      enable-analyst: true
      instances: 2
    ipdb:
      write-freq: 3600000
      write-row-limit: 0
      enable-analyst: true
      port: 41224
    epdb:
      timeout: 0

Second sample pipeline

pipeline:
  name: streamSecond
  type: realtime

  expose-to-gw: true

  taxonomy:
    region: global
    data-source: stream
    data-class: stream2
    sub-class: stream2

  proc-layout:
    -
      all: all-processes

  processes:
    tp:
      pub-mode: timer
      pub-freq-ms: 100
      log-to-journal: true
      rollover-mode: daily-at-time
      rollover-time: "00:00:00.001"
      subscribe-from-delta-messaging: true
      enable-analyst: true
      port: 45499
    rdb:
      timeout: 0
      enable-analyst: true
      instances: 2
    hdb:
      timeout: 0
      enable-analyst: true
      instances: 2
    ipdb:
      write-freq: 3600000
      write-row-limit: 0
      enable-analyst: true
      timeout: 0
    epdb:
      timeout: 0

You can query data from both pipelines simultaneously by specifying only the dataType parameter in the server API calls. Refinery returns the data as a single table.

Sample queries

// Target stream pipeline
.gwClient.query.sync[`getTicks;(`dataType`dataSource`dataClass`startDate`endDate`idList!(`streamTab;`stream;`stream;.z.d;.z.d;`))]

// Target streamSecond pipeline
.gwClient.query.sync[`getTicks;(`dataType`dataSource`dataClass`startDate`endDate`idList!(`streamTab;`stream;`stream2;.z.d;.z.d;`))]

//Target both
.gwClient.query.sync[`getTicks;(`dataType`startDate`endDate`idList!(`streamTab;.z.d;.z.d;`))]

Note

If the multi-pipeline-route parameter in the entrypoint pipeline schema is false or excluded, the query targeting both pipelines fails with a GWNoRouteException.