Skip to content

Setting up Manual Bar Generation

You must download and install the IceFixedIncome package before setting up bar generation. For more information about general bar generation, refer to getBars documentation. Ensure the Quote table contains data for the date you plan to run bargeneration. To backfill Quote data, follow the steps in the Backfilling Historical Data documentation.

The following environment variables must exist before proceeding:

  • INSIGHTS_HOSTNAME - Hostname of Insights deployment
  • CLIENT_NAME - Client ID used to request an access token from Insights
  • CLIENT_SECRET - Client secret used to request an access token

1. Create the bargen directory

Create a new directory to store all the files required to set up manual bar generation. In this example, create a directory named ice-fi-bargen.

mkdir ice-fi-bargen
Move the bargeneration pipeline spec to your ice-fi-bargen directory.

cp IceFixedIncome/pipeline-spec/bargeneration.q ice-fi-bargen/bargeneration.q

2. Update the pipeline code to execute

The IceFixedIncome package contains the pipeline code that executes when the bargen pipeline starts.

Note

If you have already set up the daily bar generation cronjob, you may have configured bargeneration.q. In this case, copy the modified bargeneration.q file to the ice-fi-bargen directory.

Open the bargeneration.q script in a code or text editor, for example: 

vi ice-fi-bargen/bargeneration.q
Edit the assemblyName variable on line 6 of bargeneration.q file to be the name of your already running fsi-app-ice-fi assembly.

// TODO Update the name of your assembly below within the quotation marks instead of the placeholder `$"ENTER_YOUR_ASSEMBLY_NAME_HERE"
assemblyName:`$"ENTER_YOUR_ASSEMBLY_NAME_HERE"

3. Create the script to deploy the bargen pipeline

Create a bash script to run the pipeline. Save this script in the same directory as the bargeneration.q file, within ice-fi-bargen. You can use any code or text editor to create this script, for example:

vi runBargen.sh

You can add the below code to your bash script.

# Usage:
# Flags which take a user input:
#   --date | -d                       : Select which date to run bargen for. Must be in YYYY.MM.DD format. Example `./runBargen.sh   -d 2000.12.31`

logMsg(){
  if [ -z "$*" ]; then
    echo "No message provided"
    return 1
  fi
  echo "$(date -u +'%Y-%m-%dT%H:%M:%S.%3NZ') ## $*"
}

## Default Arguments
## ASM_NAME be updated if using a different assembly name
ASM_NAME="fsi-app-ice-fi"
DATE=""

## Grab flags
while [[ $# -gt 0 ]]; do
  case $1 in
    -date|--date|-d)
      DATE=${2}
      shift ## past argument
      shift ## past value
      ;;
    -*|--*)
      logMsg "Unknown option $1"
      exit 1
      ;;
  esac
done

if [[ -z "$DATE" ]]; then
          logMsg "Date parameter must be supplied with -d or --date. Example: -d 2000.12.31"
          exit 1
fi

if [[ ! "$DATE" =~ ^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$ ]]; then
          logMsg "Date parameter must be of the format YYYY.MM.DD. Example: -d 2000.12.31"
          exit 1
fi

## Use date provided
sed -i "s|^trargs\[\`dt\]:.*|trargs\[\`dt\]:$DATE;|" bargeneration.q

################ FUNCTIONS ######################
renewToken(){
    logMsg "Renewing keycloak token"
    curl -s --header "Content-Type: application/x-www-form-urlencoded" \
         -d "grant_type=client_credentials&client_id=$CLIENT_NAME&client_secret=$CLIENT_SECRET" \
         "$INSIGHTS_HOSTNAME/auth/realms/insights/protocol/openid-connect/token" \
         | jq -r .access_token > token
}

teardown(){
    ## arg1- pipeline name
    PIPELINE_NAME=$1
    renewToken
    logMsg "Tearing down pipeline: $PIPELINE_NAME"
    curl -s -S -X POST -H "Authorization: Bearer $(cat token)" $INSIGHTS_HOSTNAME/streamprocessor/pipeline/teardown/insights-$PIPELINE_NAME?clearCheckpoints=true
}

runPipeline(){
    while getopts "p:s:" opt; do
      case $opt in
        p) PIPELINE_NAME="$OPTARG" ;;
        s) SPEC_FILE="$OPTARG" ;;
      esac
    done
    logMsg "Deploying pipeline: $PIPELINE_NAME with spec file: $SPEC_FILE"

    ## Token needs renewed before running pipeline
    renewToken

    ## Teardown pipeline if it already exists
    teardown $PIPELINE_NAME
    logMsg "Waiting for pipeline to teardown"
    sleep 10

    logMsg "Pipeline will write to assembly: $ASM_NAME"
    ## run request
    curl -s -S -X POST $INSIGHTS_HOSTNAME/streamprocessor/pipeline/create \
        -H "Authorization: Bearer $(cat token)" \
        -d "$(jq -n  --arg spec "$(cat $SPEC_FILE)" \
        --arg pipeline_name $PIPELINE_NAME \
        --arg asm_name $ASM_NAME \
        --arg configmap_name $ASM_NAME-assembly-configmap \
        '{
            name     : $pipeline_name,
            type     : "spec",
            config   : { content: $spec },
            settings : {
                minWorkers: "1",
                maxWorkers: "1"
            },
            env      : {
                ASM_NAME: $asm_name,
                KX_KURL_DEBUG_LOG: "1",
                KXI_SP_DIRECT_WRITE_ASSEMBLY: $asm_name
            },
            kubeConfig  : {
                configMaps: $configmap_name
            }
        }' | jq -asR .)"

}

runPipeline -p bargen -s bargeneration.q

4. Running the runBargen.sh script

Use the runBargen.sh script to manually run the bargen pipeline. The script accepts a single flag: - --date | -d : Select the date for running bargen. The date must be in YYYY.MM.DD format. For example, ./runBargen.sh -d 2000.12.31

To ensure the script is executable, run the following: 

chmod +x runBargen.sh

Now the script is ready to be run. Example use: 

./runBargen.sh -d 2000.12.31

The script terminates any existing bargen pipelines before launching a new one. Ensure any running pipelines have finished before starting a new one. The pipeline name is prefixed with insights, for example: insights-bargen.