Skip to content

Ingesting Data

This section details some common use-cases for ingesting data into the KX Insights Platform. Each of the links in the navigation pane on the left deploys a different type of assembly pipeline with examples for:

  • Import from the UI
  • Create a Pipeline in the UI
  • Publishing from the SDKs
  • Reading from Kafka
  • Batch ingest from S3 buckets
  • Mounting an existing S3 bucket as a database
  • Writing a new database to S3

Downloading assemblies

Each of the examples above (not being generated by the UI) deploys a unique assembly CR. The files used by these samples are hosted on Nexus and should be downloaded to run locally.

Firstly, get the correct version of the assemblies by running the command below.

export INSIGHTS_OPVERSION=$(kubectl get deployment/insights-kxi-operator --namespace kxi-operator -o jsonpath='{.metadata.labels.app\.kubernetes\.io/version}')

The assembly files can be downloaded and applied. The code block below does this for the RT client assembly.

Replace <USER> and <PASSWORD> in the example command for apply the basic-assembly below with your Nexus credentials.

wget --user=<USER> --password=<PASSWORD> https://nexus.dl.kx.com/repository/kx-insights-packages/assembly-samples/${INSIGHTS_OPVERSION}/sdk_sample_assembly.yaml

kubectl apply -f sdk_sample_assembly.yaml

Querying data

After data is ingested by one of the sample assemblies, it can be queried using the Service Gateway.

Full details on the available functionality for querying can be found here.