Creating environment-based process settings¶

The following is an extension to the process settings as defined in Creating pipelines. Its purpose is to reduce the overhead of maintaining multiple pipeline YAML files, where only select process settings differ per.

Environment¶

Refer to Environment for setup. Once defined, add the environment key to your pipeline.yaml

pipeline:
  name: envexample
  type: realtime
  environment: KX_REFINERY_ENV

In this instance, we've defined our environment variable as KX_REFINERY_ENV, which has a value of dev. You can then utilise this value to parse processes from the dev block as outlined in the examples below.

Process Layout¶

environment defines which processes settings to parse based on an environment variable and determines how the process settings are applied to the pipeline. Each element of the array is an environment object containing an array of *process-types* that defines process specific settings.

The supported keys of the object are: * all-environments: Applies defaults per *process-type*

Example¶

In the following example, the ipdb, write-freq value of 20000 will overwrite the default 30000 for the dev environment.

pipeline:
  name: envexample
  type: realtime

  expose-to-gw: true

  environment: KX_REFINERY_ENV
  proc-layout:
   -
    all-environments:
      all: primary-server
    dev:
      rdb.0: primary-server
      rdb.1: secondary-server
      rdb:
        -
          host: primary-server
          instances: 2

  taxonomy:
    region: test
    data-source: example

  processes:
    all-environments:
      tp:
        pub-mode: timer
        pub-freq-ms: 100
        log-to-journal: true
        rollover-mode: daily-at-time
        rollover-time: "00:00:00.001"
        enable-analyst: true
        subscribe-from-delta-messaging: true
      rdb:
        timeout: 30
        enable-analyst: true
        instances: 2
        port:
          - 5001
          - 5002
      hdb:
        timeout: 30
        enable-analyst: true
      ipdb:
        write-freq: 30000
        write-row-limit: 0
      epdb:
        timeout: 0
    dev:
      ipdb:
        write-freq: 20000
    prod:
      ipdb:
        write-freq: 60000

In this example, there will be a total of 4 rdb instances for dev, 3 on primary-server and 1 on secondary-server. On all other environments there will be 2 rdb instances. This example assigns all of the processes to the same cluster-number of 0.

Refer to Creating pipelines for the standard process configuration settings, all of which are supported within environment key.

Limitations¶

Process settings per environment apply to all processes of the same *process-type*
All *process-type* instances share the same process settings within an environment block
All *process-type* instances across multiple pipelines share the same process settings within an environment block
Process settings are defined using the all-environments block first, then environment specific settings augment/override defaults
instances defined in processes are only applied to *process-type* within the *proc-layout*. The number of instances for *host/instances* and *process-type*.*process-instance* definitions will be parsed from the *proc-layout* section
When defining port ranges in the process settings, the number of ports much match the total number of expected instances across all pipelines