Skip to content

Setting up Manual Pipeline Execution

You must have downloaded and installed the relevant package containing the pipeline you want to use before setting up manual pipeline execution.

1. Create a new package

Create a new package that has the same name as the package with the pipeline you want to run but with 'z' prepended eg z-ice-fixed-income. The below commands will set this up, replace <your-package> with the relevant package name, eg ice-fixed-income, and <your-pipeline> with the pipeline name you want to use, eg bargeneration:

## Creates a new package with a manifest and init.q file.
kxi package init z-<your-package>

## Creates a new pipeline and src file in directories pipelines and src. Adds the new pipeline to the manifest file also.
kxi package add --to z-<your-package> pipeline --name <your-pipeline>

2. Edit pipeline code file

You may have already configured the pipeline code file at this stage, for example, bargeneration.q, eqeagentca.q. If you have set up the pipeline to run using another method, then editing the file again is unnecessary. In this case, simply copy the edited pipeline code file to your package, z-<your-package>.

Otherwise, if you do not have your pipeline code file configured, copy the pipeline code file, for example, bargeneration.q, eqeagentca.q from the downloaded package to your working directory. This command is for the bargeneration.q pipeline, which is in the ice-fixed-income package, and should be adjusted for the pipeline you are using.

scp IceFixedIncome/pipeline-spec/bargeneration.q .

Edit the assemblyName variable in the pipeline code file to be the name of your running assembly. In the case of the bargeneration pipeline, this is the fsi-app-ice-fi assembly.

// TODO Update the name of your assembly below within the quotation marks instead of the placeholder `$"ENTER_YOUR_ASSEMBLY_NAME_HERE"
assemblyName:`$"ENTER_YOUR_ASSEMBLY_NAME_HERE"

Then copy your updated pipeline code file to your z-<your-package> package, giving the file the same name as the pipeline you created in step 1. This command is for the bargeneration.q pipeline and should be adjusted for the pipeline you are using.

scp bargeneration.q z-<your-package>/src/<your-pipeline>.q

3. Edit the pipeline yaml file

Add the following code to the bottom of your <your-pipeline>.yaml pipeline file created in step 1. This defines the config map, which allows you to pick up schemas between packages. Update <your-package> and <your-pipeline> in the below command to be the values you are using. Also, update <assembly-name> to be the running assembly, matching the assembly name used in step 2.

sed -i -e '$a\configMaps:' -e '$a\- <assembly-name>-assembly-configmap' z-<your-package>/pipelines/<your-pipeline>.yaml

4. Create the packaging script to deploy the manual pipeline execution package

Create the packaging bash script. This should be created outside the z-<your-package> package.

vi genPackage.sh

Add the below code to your bash script, replacing <your-package> with the package name you have used above and replacing <your-pipeline>.q with the pipeline code file name in your case.

# Usage:
# Flags which take a user input:
#   --date | -d                       : Select which date to run the pipeline for. Must be in YYYY.MM.DD format. Example `./genPackage.sh   --d 2024.05.30   `
#   --package-name | -p               : Optional argument to set a custom name for your package. If not specified default package name is z-<your-package>

ASM_NAME="z-<your-package>"
DATE="yesterday"

# Grab flags
while [[ $# -gt 0 ]]; do
  case $1 in
    -date|--date|-d)
      DATE=${2}
      shift # past argument
      shift # past value
      ;;
    -package-name|--package-name|-p)
      ASM_NAME=("$2")
      shift # past argument
      shift # past value
      ;;
    -*|--*)
      echo "Unknown option $1"
      exit 1
      ;;
  esac
done

if [[ ${DATE} != "yesterday" ]] ; then
    sed 's/.z.d-1/'"$DATE"'/' ${ASM_NAME}/src/<your-pipeline>.q > changed.q && mv changed.q ${ASM_NAME}/src/<your-pipeline>.q
fi

sleep 5

echo Running generation pipeline within package ${ASM_NAME} for the date: ${DATE}

kxi assembly teardown --force --name ${ASM_NAME}

kxi package remote-remove ${ASM_NAME}/0.0.1

# Make directory where all packaging actions is to be performed
FSI_PKG_DIR=/tmp/fsiPackage/
mkdir -p ${FSI_PKG_DIR}
mkdir -p /tmp/packages
mkdir -p /tmp/artifacts
export KX_PACKAGE_PATH=/tmp/packages
export KX_ARTIFACT_PATH=/tmp/artifacts
export KX_PACKAGE_FORMAT=.yaml

cp -r ${ASM_NAME} ${FSI_PKG_DIR}

kxi package packit ${FSI_PKG_DIR}/${ASM_NAME} --tag --package-name ${ASM_NAME}

# Clean up the package dir
echo "Cleaning up FSI packaging directory "${FSI_PKG_DIR}
rm -rf ${FSI_PKG_DIR}/${ASM_NAME}

kxi package push --force ${ASM_NAME}/0.0.1

kxi package deploy ${ASM_NAME}/0.0.1

if [[ ${DATE} != "yesterday" ]] ; then
    sed 's/'"$DATE"'/.z.d-1/' ${ASM_NAME}/src/<your-pipeline>.q > changed.q && mv changed.q ${ASM_NAME}/src/<your-pipeline>.q
fi

Note, you may have to update the permissions of the genPackage.sh script. You can do this by running a chmod command. For example:

chmod 777 genPackage.sh

5. Review the layout

After setting all of the above up, the layout of your work should look something similar to the below:

.
├── genPackage.sh
└── z-<your-package>
    ├── init.q
    ├── manifest.yaml
    ├── pipelines
    │   └── <your-pipeline>.yaml
    └── src
        └── <your-pipeline>.q

6. Running the script

The genPackage.sh script allows you to run the pipeline manually. It takes 2 flags: - -package/ -p -- This is the name of the package. The default is z-<your-package>. It is recommended to use the same name as the directory that contains the package you created in step 1. - -date/-d -- This is the date that you want to run the pipeline for. By default, it looks at yesterday's date. The date must be written in YYYY.MM.DD format. There must be source data present for the date you want to run the pipeline for in order to work. This source data differs depending on the specifics of the pipeline you are running.

Example use

./genPackage.sh -p z-<your-package> -d 2024.06.01

The script tears down any <your-pipeline> pipelines currently running.

The pipeline name is prefixed with z-<your-package>, for example, z-ice-fixed-income-bargeneration.