Skip to content

General

Stream Processor Metadata

Stream Processor metadata is exposed across many APIs, such as state and pushing data further in a pipeline within asynchronous nodes. A pipeline operator's metadata is a dictionary containing the following keys. These keys will be extended as new capabilities are added to the Stream Processor engine.

Name Type Description
key any An optional partition key for the events in this message. This is used to group related collections of data based on a common attribute. Events that do not share the same key are considered independent and may be processed independent of other keys.
window temporal Indicates the start time or index of a window of data.
offset int or temporal An event offset is a number or time that represents the current message offset to the stream reader. This value will be used to recover data if messages are lost down stream. In the case messages need to be replayed, the message offset will be used to indicate the last successfully processed message.

.qsp.push

Publishes data to all downstream operators in the pipeline. .qsp.push should be used from asynchronous operators in a pipeline to continue the flow of data in the pipeline. It is only required for .qsp.apply. Other operators are synchronous and any data returned will flow to the next operator.

Parameters:

Name Type Description
op #.sp.op The current operator configuration. To access the operator configuration from the body of an operator function, use .qsp.use and set the params field to include context.
md #.qsp.metadata Metadata for message that is being emitted from the current asynchronous operator.
data any Data that is to be published from this async operator.

Returns:

Type Description
any Returns the data that was passed to this function.

See Also: .qsp.use

.qsp.run

Installs and runs a pipeline in the current stream processor. Depending on the deployment configuration, this function can run a pipeline in the current worker process or distribute work across several workers.

Parameter:

Name Type Description
pipe #.qsp.pipe Streaming pipeline.

Returns:

Type Description
null

.qsp.teardown

Tear down a pipeline, removing all state and timers.

Any 'onTeardown' (using .qsp.onTeardown) handlers will be called.

Returns:

Type Description
null

Example:

  .qsp.teardown[]