Setting up Bar Generation
You must have downloaded and installed the IceFixedIncome package before setting up bargeneration. The general bar generation documentation is here.
1. Create the daily bargen directory
Create a new directory to contain all of the files needed to set up daily bargeneration. In this example, we'll call ours ice-fi-daily-bargen
mkdir ice-fi-daily-bargen
ice-fi-daily-bargen
directory. The files are:
- bargeneration.q
- bargen-inspect-pod.yaml
- bargen-pvc.yaml
- bargeneration-cronjob.yaml
cp IceFixedIncome/pipeline-spec/bargeneration.q IceFixedIncome/config/bargen-inspect-pod.yaml IceFixedIncome/config/bargen-pvc.yaml IceFixedIncome/config/bargeneration-cronjob.yaml ice-fi-daily-bargen
Edit bargeneration.q
Edit the assemblyName
variable on line 6 of bargeneration.q
file to be the name of your already running fsi-app-ice-fi assembly.
// TODO Update the name of your assembly below within the quotation marks instead of the placeholder `$"ENTER_YOUR_ASSEMBLY_NAME_HERE"
assemblyName:`$"ENTER_YOUR_ASSEMBLY_NAME_HERE"
Note, you will need the bargeneration.q
again if you are setting up manual bargeneration.
2. Create run-bargeneration-cronjob.sh
We need to create a bash script called run-bargeneration-cronjob.sh
that will run within the cronjob process. The script will submit a POST REST request to bring up a bargeneration pipeline using the bargeneration.q
pipeline spec.
vi run-bargeneration-cronjob.sh
And insert the below code
#!/bin/bash
## Env variables needed before running script with
## TODO: Review, and if necessary change ASM_NAME variable to the name of your ice fixed income assembly
ASM_NAME="fsi-app-ice-fi"
## Install the libraries needed to run the POST request
apt-get update -o=dir::cache=/mnt/bargen/packages/
apt-get install -f -y -o=dir::cache=/mnt/bargen/packages/ git vim jq curl
echo "alias aws='/mnt/bargen/packages/bin/aws'" >> ~/.bashrc
cd /mnt/bargen/
################ FUNCTIONS ######################
logMsg(){
if [ -z "$*" ]; then
echo "No message provided"
return 1
fi
echo "$(date -u +'%Y-%m-%dT%H:%M:%S.%3NZ') ## $*"
}
renewToken(){
logMsg "Renewing keycloak token"
curl -s --header "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=$CLIENT_NAME&client_secret=$CLIENT_SECRET" \
"$INSIGHTS_HOSTNAME/auth/realms/insights/protocol/openid-connect/token" \
| jq -r .access_token > token
}
getPipelineStatus(){
## arg1 pipeline name
renewToken
curl -s -S -X GET -H "Authorization: Bearer $(cat token)" $INSIGHTS_HOSTNAME/streamprocessor/pipeline/status/insights-bargeneration
}
teardown(){
# arg1- pipeline name
renewToken
curl -s -S -X POST -H "Authorization: Bearer $(cat token)" $INSIGHTS_HOSTNAME/streamprocessor/pipeline/teardown/insights-bargeneration?clearCheckpoints=true
}
runPipeline(){
export PIPELINE_NAME="bargeneration"
export SPEC_FILE="/mnt/bargen/bargeneration.q"
echo $SPEC_FILE $PIPELINE_NAME
# Token needs renewed before running pipeline
renewToken
## Teardown pipeline if it already exists
teardown $PIPELINE_NAME
logMsg "Waiting for pipeline to teardown"
sleep 10
# run request
curl -s -S -X POST $INSIGHTS_HOSTNAME/streamprocessor/pipeline/create\
-H "Authorization: Bearer $(cat token)" \
-d "$(jq -n --arg spec "$(cat $SPEC_FILE)" --arg aws_access_key_id $AWS_ACCESS_KEY_ID \
--arg aws_secret_access_key $AWS_SECRET_ACCESS_KEY \
--arg pipeline_name $PIPELINE_NAME \
--arg asm_name $ASM_NAME \
--arg configmap_name $ASM_NAME-assembly-configmap \
'{
name : $pipeline_name,
type : "spec",
config : { content: $spec },
settings : {
minWorkers: "1",
maxWorkers: "1"
},
env : {
KXI_SP_BETA_FEATURES: "true",
ASM_NAME: $asm_name,
AWS_REGION: "eu-west-1",
AWS_ACCESS_KEY_ID: $aws_access_key_id,
AWS_SECRET_ACCESS_KEY: $aws_secret_access_key,
KX_KURL_DEBUG_LOG: "1",
KXI_SP_DIRECT_WRITE_ASSEMBLY: $asm_name,
KX_TRACE_S3: "1"
},
kubeConfig : {
configMaps: $configmap_name
}
}' | jq -asR .)"
}
runPipeline
Edit the variable ASM_NAME
on line 5 of the run-bargeneration-cronjob.sh
to be the name of your already running ice-fixed-income assembly. (This should be the same name that you set assemblyName
to within bargeneration.q
)
## TODO: Review, and if necessary change ASM_NAME variable to the name of your ice fixed income assembly
ASM_NAME="fsi-app-ice-fi"
3. Create PVC to mount your volume and move files to the volume
Create a PVC that you will mount to your bargeneration process.
kubectl create -f bargen-pvc.yaml
Create the pod that mounts to the volume so that you can move files into the mount file path
kubectl apply -f bargen-inspect-pod.yaml
Then copy the run-bargeneration-cronjob.sh
script, and the bargeneration.q
pipeline spec file to your volume which should be at the path /mnt/bargen
. Replace {your-namespace} with the name of your namespace within your kubernetes cluster.
kubectl cp run-bargeneration-cronjob.sh {your-namespace}/pod-inspect-bargen-pvc:/mnt/bargen
kubectl cp bargeneration.q {your-namespace}/pod-inspect-bargen-pvc:/mnt/bargen
Note, you may have to update the permissions of the run-bargeneration-cronjob.sh
script once it is mounted on the volume. You can do this by running a chmod command on the pvc pod using kubectl exec
. For example:
kubectl exec -it pod-inspect-bargen-pvc -- chmod 777 /mnt/bargen/run-bargeneration-cronjob.sh
Once you have copied your files over, you can delete the pod which you used to mount your pvc (Your cronjob does not run if the volume is mounted by another process).
kubectl delete pod pod-inspect-bargen-pvc
4. Set up cronjob to have bargeneration run daily
Review the cronjob file bargeneration-cronjob.yaml
. The time it is due to run can be found within the schedule section of the spec section of the file.
By default it is defined as schedule: "00 02 * * *"
which means it runs at 2am UTC every day. See https://crontab.guru/ to see how you can configure the time to run at different times of the day.
Review the secrets that are used within the bargeneration-cronjob.yaml
file. The fields that you will need are
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- CLIENT_ID
- INSIGHTS_HOSTNAME
- CLIENT_SECRET
- CLIENT_NAME
To set the cronjob up to run, you need to run the command
kubectl apply -f bargeneration-cronjob.yaml
This runs the run-bargeneration-cronjob.sh
script within a pod. The script installs the necessary libraries to run a POST command to bring up the bar generation pipeline.
The pipeline will be prefixed with insights-
eg. insights-bargeneration