Skip to content

Lifecycle

.qsp.finish

Calls finish on an operator to indicate that the processing has completed

Parameter:

Name Type Description
op #.sp.op | symbol Pipeline operator.

Returns:

Type Description
null

.qsp.finishTask

Marks a task as finished. If all tasks for the operator are finished, and the operator has been requested to finish, the operator is finished.

Parameters:

Name Type Description
op #.sp.op Pipeline operator.
taskID int Task identifier.

Returns:

Type Description
null

Example: Register an asynchronous task for true async using apply

 .qsp.run
     .qsp.read.fromCallback[`publish]
     .qsp.apply[{[op; md; data]
         tid: .qsp.registerTask[op];  // Register a task which represents the unfinished async kurl GET request
         .kurl.async ("https://www.google.ca";"GET";``callback!(::;{[op;md;data;tid;r] // GET request with a callback
             .qsp.finishTask[op;tid]; // Mark the task as finished
              data[`response]: r 1;   // Add GET request response to data
             .qsp.push[op;md;data];   // Push enriched data to the next operator
             }[op;md;data;tid]));
         }]
     .qsp.write.toConsole[];

.qsp.onCheckpoint

Sets the onCheckpoint handler for the pipeline. The onCheckpoint handler for the pipeline is called whenever a checkpoint is created. The return value of the onCheckpoint handler is saved to the checkpoint, and later passed into the pipeline onRecover handler.

Parameter:

Name Type Description
handler fn () → any Function to execute during checkpointing.

Returns:

Type Description
null

.qsp.onFinish

Sets the onFinish event handler of the operator with the given id. The onFinish event handler is called once an operator has been marked as finished, but before any successive operators are requested to finish.

Parameters:

Name Type Description
id symbol Operator Identifier.
handler fn (#.sp.op; #.qsp.metadata) → null Function to execute on finish event.

Returns:

Type Description
null

.qsp.onOperatorCheckpoint

Sets the onCheckpoint event handler of the operator with the given id. The onCheckpoint event handler is called when a checkpoint is created, before operator state is saved. This provides an opportunity for operators to make updates to their state before it is checkpointed.

Parameters:

Name Type Description
id symbol Operator ID.
handler fn (#.sp.op; #.qsp.metadata) → null Function to execute before an operator performs a checkpoint.

Returns:

Type Description
null

.qsp.onOperatorPostCheckpoint

Sets the onPostCheckpoint event handler of the operator with the given id. The onPostCheckpoint event handler is called after a checkpoint has either been saved to disk or written to the Controller.

Parameters:

Name Type Description
id symbol Operator ID.
handler fn ( #.sp.op; dict (...any: dict ( metadata: dict; state: any )) ) Function to execute after a checkpoint has been saved.

Returns:

Type Description
null

.qsp.onOperatorRecover

Sets the onRecover event handler of the operator with the given id. The onRecover handler is called when a pipeline recovers from a checkpoint. Operator state has already been restored once this handler has been called. This provides an opportunity for operators to make updates to their state before the pipeline starts running.

Parameters:

Name Type Description
id symbol Operator ID.
handler fn Function to execute after an operator recovers.

Returns:

Type Description
null

.qsp.onPostCheckpoint

Sets the onPostCheckpoint handler for the pipeline. The onPostCheckpoint handler for the pipeline is called after a checkpoint has been written either to disk or to the controller, and has been acknowledged by the controller.

Parameter:

Name Type Description
handler fn (#.qsp.dict (...symbol: dict (...any: dict (metadata: dict; state: any))))

.qsp.onRecover

Sets the onRecover handler for the pipeline. The onRecover handler for the pipeline is called when a pipeline recovers from a checkpoint. The handler is given the result of the last onCheckpoint pipeline handler as its only argument.

Parameter:

Name Type Description
handler fn (any) → null Function to execute after recovery.

Returns:

Type Description
null

.qsp.onSetup

Sets the onSetup event handler for the pipeline. The onSetup event handler is called before the pipeline is finished initializing.

Parameter:

Name Type Description
handler fn () → null Function to execute after all operators in a pipeline are setup.

Returns:

Type Description
null

.qsp.onStart

Sets the onStart event handler for the pipeline. The onStart event handler is called after the pipeline has invoked start on its readers.

Parameter:

Name Type Description
handler fn () → null Function to execute after all operators in a pipeline are started.

Returns:

Type Description
null

.qsp.onTeardown

Sets the onTeardown event handler for the pipeline. The onTeardown event handler is called after the pipeline is torn down.

Parameter:

Name Type Description
handler fn () → null Function to execute before a pipeline is torn down.

Returns:

Type Description
null

.qsp.registerTask

Registers a task for the given operator and returns an identifier for that task. A task represents an asynchronous request that an operator made, and once that request is finished, the task should be marked as finished using .qsp.finishTask

Parameter:

Name Type Description
op #.sp.op Pipeline operator.

Returns:

Type Description
int Task identifier.

Example: Register an asynchronous task for true async using apply

 .qsp.run
     .qsp.read.fromCallback[`publish]
     .qsp.apply[{[op; md; data]
         tid: .qsp.registerTask[op];  // Register a task which represents the unfinished async kurl GET request
         .kurl.async ("https://www.google.ca";"GET";``callback!(::;{[op;md;data;tid;r] // GET request with a callback
             .qsp.finishTask[op;tid]; // Mark the task as finished
              data[`response]: r 1;   // Add GET request response to data
             .qsp.push[op;md;data];   // Push enriched data to the next operator
             }[op;md;data;tid]));
         }]
     .qsp.write.toConsole[];