Bloomberg Market Data Ingest

Pipelines are provided to ingest Bloomberg market data from .CSV or parquet file formats.

The pipelines that ingest .CSV format are:

• bbgquoteingest
• bbgtradeingest

The pipelines that ingest parquet format are:

• bbgquotepq
• bbgtradepq

Customizing the Pipelines

Pipelines can be adapted to meet different requirements. By editing the corresponding spec.q file for each pipeline, various aspects of the pipeline operation can be adjusted, such as:

• Each pipeline reads a file from S3 by default. This can be changed to a different reader or to a different location by editing the .fsi.reader variable. Refer to readers for the available readers.
• The file targeted by the pipeline for ingest is configured by the .fsi.filePath variable. This should be updated to point toward the appropriate file for ingestion.
• The region the file is located in is configured by the .fsi.region variable.

In order to edit the spec.q files, the Accelerator must be unpacked. You can then edit them with a text editor. Then, the package should be pushed and deployed.

Unpack

Refer to the following documentation for general commands to unpack and re-package a package.

Edit

Now you can make the changes required to the spec.q files.

Once changes have been made and packaged, the updated package must be pushed before being deployed.

Push and deploy

Refer to the following documentation on pushing a package.

Read the instructions to deploy an assembly.

Parquet Ingest Pipelines

The parquet format ingest pipelines copy the ingested data to the tmp directory in the sp worker pod as part of the ingest process. This must be configured to be large enough to accommodate the size of the data being ingested, that is, all the data from the current file being ingested. The default size for the tmp directory is 5MB. To modify this, the k8sPolicy value should be edited in the pipeline YAML file.

    worker:
      image: {}
      k8sPolicy:
        resources:
          tmpDirSize: 5Mi

The same commands referred to in the previous section can be used to unpack and re-package the package in order to modify pipeline yaml files.