OpenAPI

Note

The Coordinator service is only used within Kubernetes.

To interact with the Stream Processor Coordinator, either use its Kubernetes service name within the cluster, or port-forward to access it locally.

kubectl port-forward sp-kxi-sp-0 5000

Create a pipeline

First, create a spec file using the Pipeline API.

cat spec.q

.qsp.run
    .qsp.read.fromCallback[`upd]
    .qsp.write.toConsle[]

Then, submit the job to the Coordinator.

curl -X POST http://localhost:5000/pipeline/create -d \
    "$(jq -n  --arg spec "$(cat spec.q)" \
    '{
        name        : "file",
        type        : "spec",
        persistence : {controller:{}, worker:{}},
        config      : { content: $spec },
        settings    : {
            minWorkers: "1",
            maxWorkers: "10"
        }
    }' | jq -asR .)"

See the available configuration options here.

The REST call returns the ID of the pipeline, which can be used in other REST calls for obtaining status information as the pipeline is brought online.

After submitting, you can watch the Controller and Worker StatefulSets spin up within the cluster and connect themselves together.

Get pipelines

Return basic information about all pipelines.

curl localhost:5000/pipelines | jq

[
  {
    "id": "file-gmhxu97kpo",
    "name": "file",
    "created": "2021-08-14T21:04:35.631193507",
    "state": "RUNNING",
    "error": ""
  }
]

Get details of pipelines

Return more detailed information about all pipelines, including aggregated performance metrics.

curl localhost:5000/details | jq

[
  {
    "id": "file-gmhxu97kpo",
    "name": "file",
    "created": "2021-08-14T21:04:35.631193507",
    "lastSeen": "2021-08-14T21:13:46.342579775",
    "state": "RUNNING",
    "error": "",
    "metrics": {
      "eventRate": 119400.708,
      "bytesRate": 311912.823,
      "latency": 18.0915
    }
  }
]

Get pipeline status

curl localhost:5000/pipeline/status/file-gmhxu97kpo

{ "state": "Running", "error": "" }

Tear down a pipeline

Tear down a single pipeline. This will remove all associated workloads and Kubernetes resources related to the pipeline.

curl -X POST localhost:5000/pipeline/teardown/<id>

Trigger an operator Read

For a full discussion on pull reader triggering and when to use it see: Pipeline Read Triggering.

Trigger all pull reader operators configured for triggering:

curl -X POST https://insights.kx.com/streamprocessor/pipeline/<id>/admin/triggerRead \
    --header "Authorization: Bearer $INSIGHTS_TOKEN" \

Trigger readers with operator IDs expr1 and expr2 only. Use a comma separated list for opIDs, assuming expr1 and expr2 are configured for triggering. (See pull reader options for details of trigger configuration options.)

curl -X POST https://insights.kx.com/streamprocessor/pipeline/<id>/admin/triggerRead?opIDs=expr1,expr2 \
        --header "Authorization: Bearer $INSIGHTS_TOKEN" \

Trigger a read from expr1 only (assuming expr1 is configured for reading):

curl -X POST https://insights.kx.com/streamprocessor/pipeline/<id>/admin/triggerRead?opIDs=expr1 \
        --header "Authorization: Bearer $INSIGHTS_TOKEN" \