General
.qsp. configPath return the path of mounted configuration files getPartitions return the current assigned partitions getPartitionCount return the count of all partitions push publish data to all downstream operators run install and run a pipeline teardown tear down a pipeline setTrace enables trace logging clearTrace clears trace logging and resets logging level
Stream Processor metadata is exposed across many APIs, such as state and pushing data further in a pipeline within asynchronous nodes. A pipeline operator's metadata is a dictionary containing the following keys.
Metadata Keys
These keys will be extended as new capabilities are added to the Stream Processor engine.
- key
-
An optional partition key for the events in this message. This is used to group related collections of data based on a common attribute. Events that do not share the same key are considered independent and may be processed independently of other keys. (any type)
- window
-
Indicates the start time or index of a window of data (temporal)
- offset
-
An event offset is a number or time that represents the current message offset to the stream reader. (int or temporal)
This value will be used to recover data if messages are lost downstream. If messages need to be replayed, the message offset will be used to indicate the last successfully processed message.
.qsp.push
Publish data to all downstream operators in the pipeline
.qsp.push[op;md;data]
Parameters:
name | q type | description |
---|---|---|
op | .qsp.op | The current operator configuration. |
md | .qsp.message.metadata | Metadata for the message being emitted from the current asynchronous operator. |
data | .qsp.message.event or .qsp.message.event[] | The data to be published from this async operator. |
publishes data to all downstream operators in the pipeline, and returns data
.
Should be used from asynchronous operators in a pipeline to continue the flow of data
in the pipeline. It is only required for .qsp.apply
. Other operators are synchronous and any
data returned will flow to the next operator.
.qsp.run
Install and run a pipeline
.qsp.run[pipe]
Parameters:
name | q type | description |
---|---|---|
pipe | #.qsp.pipe | The pipeline to install and run in the current stream processor. |
Returns generic null.
Depending on the deployment configuration, this function can run a pipeline in the current worker process or distribute work across several workers.
.qsp.teardown
Tear down a pipeline
.qsp.teardown[]
Tears down a pipeline, removing all state and timers; returns generic null.
Any onTeardown
(using .qsp.onTeardown
) handlers
or subscribers to teardown
will be called.
.qsp.getPartitions
Get assigned partitions
.qsp.getPartitions[]
Return a list of the partitions assigned to this worker. This can be used for subsetting a stream.
.qsp.getPartitionCount
Get count of all partitions
.qsp.getPartitionCount[]
Return the count of all partitions in the current stream. This can be combined with getPartitions[]
to subset a stream.
.qsp.configPath
Returns the location of user mounted configurations
.qsp.configPath[]
.qsp.configPath[object]
Parameters:
name | q type | description | default |
---|---|---|---|
object | string | The configuration object to get the path of. | The KXI_SP_CONFIG_PATH environment variable |
This is mostly useful in Kubernetes deployments to access the path of ConfigMaps and Secrets. Passing the name of the ConfigMap or Secret to this function will return the path where it was mounted.
The mount path can be changed by setting KXI_SP_CONFIG_PATH
to a directory on the
target deployment.
.qsp.setTrace
Enables program tracing logs
.qsp.setTrace[level]
Parameters:
name | q type | description | default |
---|---|---|---|
level | long | Level of trace logging to display. | 0 |
Level of verbosity of trace logging. Setting a level of 0 will clear trace logging. The following levels are available:
- Log data that is passed through readers and writers.
- Log data pushed through buffers.
- Log operator inputs.
- Log state operations.
.qsp.clearTrace
Disable program tracing logs
.qsp.clearTrace[]
Clears trace logging and sets the log level to its previous level