How to Implement Asynchronous Callbacks

This page shows you how to implement asynchronous callbacks in KDB-X for non-blocking remote calls. For more information on interprocess communication (IPC) in q, see the I/O chapter in Q for Mortals.

Overview

• Coordinate the three components
• Basic callback example
• Handling multiple parameters
• Generic function wrapper
• Remote execution with anonymous functions
• Preventing deadlock

Coordinate the three components

Asynchronous callbacks require coordinating three components: the client's call, the server function's signature, and the server's callback mechanism.

1. Client-side call: The client initiates an asynchronous request to a named function on the server. The last argument in the call is a symbol representing the name of the callback function that exists on the client.

   / Syntax: (neg h) (`remoteFunc; arg1; ..; argN; `clientCallback)
   (neg h) (`proc; 42; `clientFunc)
   

2. Server-side function signature: The remote function on the server must be defined to accept the callback function's name as its final argument.

   / Syntax: remoteFunc:{[arg1; ..; argN; callbackName] ... }
   proc:{[dataArg; callbackName] ... }
   

3. Server-side callback execution: Inside the remote function, use .z.w to get the connection handle of the calling process. Use this handle to make an asynchronous callback to the client's specified callback function, passing the result.

   proc:{[dataArg; callbackName]
     result: dataArg * 2;               / Process the data
     h: .z.w;                           / Get caller's handle
     (neg h) (callbackName; result);    / Execute the callback
   }
   

Notice in the above implementation:

• .z.w must be called inside the server function to get the connection handle of the calling process
• The callback must be asynchronous (neg h) to ensure non-blocking communication

Basic callback example

The simplest case is where a client calls a remote function with a single argument and provides a callback to receive the result.

The following examples use 0N! to print output to the console in the q session.

On the server process: Define a remote function proc that accepts a data argument x and a callback name y. It performs a simple operation and then uses the caller's handle (.z.w) to invoke the callback.

\p 5000                                           / Listen on port 5000
serverFunc:{0N!x;}                                / Represents the core server logic
proc:{serverFunc x; h:.z.w; (neg h) (y; 43)}      / Function for client to call

On the client process: Define the callback function clientFunc, open a handle to the server, and make the asynchronous call to execute proc.

clientFunc:{0N!x;}                                / Callback handler for result
h:hopen `::5000
(neg h) (`proc; 42; `clientFunc)

Result: The server console displays 42, and the client console then displays 43.

Handling multiple parameters

To call a remote function with multiple data arguments, pass them as a list.

On the server process: Define proc3 to accept a list of arguments x and a callback name y. It uses Apply (.) to pass the list of arguments to the target function add3.

\p 5000
add3:{x+y+z}                                      / A function that takes 3 arguments
proc3:{r:add3 . x; 0N!r; (neg .z.w) (y; r)}       / Wrapper to handle arg list and callback

On the client process: The client sends the arguments as a single list.

clientFunc:{0N!x;}
h:hopen `::5000
(neg h) (`proc3; 1 2 3; `clientFunc)

Result: The server console displays 6, and the client console then displays 6.

Generic function wrapper

To avoid writing a custom wrapper for every server-side function, you can implement a single generic function wrapper. This allows a client to execute any server-side function without the need to write additional code. Let's call this generic function wrapper the dispatcher function for the following example.

On the server process:

The dispatcher function takes three arguments:

• The target function's name
• The target function's arguments as a list
• The client callback name.

It dynamically executes the target function and returns the result.

\p 5000
add3:{x+y+z}                                  / An arbitrary server function
dispatcher:{(neg .z.w) (z; (value x) . y)}    / Wrapper to execute non-unary functions

On the client process: The client calls dispatcher, specifying the target function (add3), its arguments (1 2 3), and the callback function (clientFunc).

clientFunc:{0N!x;}
h:hopen `::5000
(neg h) (`dispatcher; `add3; 1 2 3; `clientFunc)

Result: The client console displays 6. The calculation is performed on the server, but no output appears there.

Remote execution with anonymous functions

Send a function literal (an anonymous function) from the client to be executed directly on the server. This powerful technique requires no pre-defined functions on the server.

On the server process: Simply start a q process listening on a port. No function definitions are needed.

$ q -p 5000

On the client process: The client sends an anonymous function that encapsulates both the logic to execute and the callback invocation.

q)clientFunc:{0N!x;}
q)h:hopen `::5000
q)(neg h) ({(neg .z.w) (z; x*y)}; 6; 7; `clientFunc)
42

The client's asynchronous message sends:

• An anonymous function: {(neg .z.w) (z; x*y)}
• Its arguments: 6 and 7
• The name of the callback function: `clientFunc

Result: The multiplication 6*7 occurs on the server, and the result 42 is displayed on the client console.

Security warning

Use this IPC pattern with extreme caution in a production environment. An unprotected server is vulnerable to arbitrary code execution from any client. Protect your server by authorizing which functions are permitted to run.

Preventing deadlock

When implementing callbacks, you must use asynchronous calls for both the initial request from the client and the callback from the server. Failure to do so will result in deadlock.

If a client makes a synchronous call to the server, it will block and wait for a response. If the server function then attempts to make a callback (even an asynchronous one) to the client, the client cannot process it because it is still blocked by its original synchronous call. The system will hang indefinitely.

Always use the negative handle form (neg h) for both the outgoing request and the callback invocation to ensure non-blocking communication in both directions.

Summary

In this guide, you:

• Learned how asynchronous callbacks work in q
• Learned the core implementation pattern with three key components
• Learned how to implement a basic callback with a single argument
• Learned how to handle multiple parameters in callbacks
• Learned how to create a generic function wrapper for flexible callback handling
• Learned how to execute anonymous functions remotely with callbacks
• Learned how to prevent deadlock by using asynchronous calls in both directions

You now have the essential skills to implement asynchronous callbacks in q for non-blocking remote calls.