Skip to content

Use WebSockets with q

This page explains how to implement WebSocket servers and clients in q for real-time, bidirectional communication, and how to publish live data to a browser.

q has supported the WebSocket protocol since v3.0. A WebSocket is a persistent, bidirectional, full-duplex channel over a single TCP connection, so the server can push data to the client the moment it becomes available. That makes WebSockets a good fit for interactive applications such as trading dashboards, live feeds, and chat systems.

For more on WebSockets in q, see the I/O chapter in Q for Mortals.

Create a WebSocket server

To turn a q process into a WebSocket server, listen on a port and define the .z.ws message handler. The server runs that callback for every message a client sends.

An echo server

This server returns every message it receives to the sender.

  1. Start a q session listening on port 5000:

    q)\p 5000

  2. Define the .z.ws callback. Sending on the negative handle neg[.z.w] returns the incoming message x to the client on the current connection handle, .z.w:

    q).z.ws: {neg[.z.w] x}

WebSocket messages are always asynchronous, which is why the callback sends on a negative handle.

Test the server with any WebSocket client. For example, open the ws.htm sample client in a browser and connect to ws://localhost:5000. Whatever you send comes back in the output area.

Free the port between examples

Every example on this page uses port 5000. Before you start another one, close the previous q process or pick a different port, or the new process can't bind.

A remote evaluation server

Redefine .z.ws to evaluate each message as q code, and .Q.s to format the result for display:

q).z.ws: {neg[.z.w] .Q.s value x}

This server runs arbitrary code from any client

Never expose raw value to untrusted clients. A client that can connect can run anything the q process can.

Use a command pattern instead: parse the input, check the command against an allowlist, and dispatch to a specific handler. See authentication and authorization.

A trap (@) returns an error in the submitted code to the client instead of leaving the callback to fail:

/ if 'value x' fails, the handler formats the error message for the client
q).z.ws: {neg[.z.w] @[.Q.s value@; x; {"'", x, "\n"}]}

Handle WebSocket events

q manages the WebSocket lifecycle through callbacks in the .z namespace:

Callback Runs when
.z.ws A message arrives from a client
.z.wo A client opens a connection. Receives the connection handle
.z.wc A client closes the connection. Receives the connection handle

Within any of these, .z.w holds the integer connection handle of the client being served.

Define .z.wo and .z.wc to track the connections a server holds:

q)activeWSConnections: ([handle: ()] connectTime: ())   / keyed by handle

q).z.wo: {`activeWSConnections upsert (x; .z.t)}        / x is the new handle
q).z.wc: {delete from `activeWSConnections where handle = x}

Inspect every socket, not just the WebSockets

-38! reports all current socket connections, WebSockets included.

Send data and manage connections

Send to a WebSocket client on the negative connection handle, neg[h], which is always asynchronous:

q)neg[h] "Hello, client!"                            / one client

q)handles: exec handle from activeWSConnections      / or every client
q)neg[handles]@\:"Market is now open"

Close connections

Close a connection with hclose. Flush it first, so that any queued outgoing message is sent before the handle goes away:

q)neg[h][]   / flush pending data, blocking until it is sent
q)hclose h   / close the handle

To close every active WebSocket connection:

q){neg[x][]; hclose x} each exec handle from activeWSConnections

Create a WebSocket client

A q process can act as a WebSocket client and connect to other servers. Available since v3.2t.

Define .z.ws before you connect

Define .z.ws first. If you don't, connecting signals a '.z.ws undefined error.

.z.ws serves both roles

The same callback handles incoming messages whether the process is acting as a server or as a client. As a client, .z.ws receives whatever the remote server sends.

Open a connection by sending an HTTP Upgrade request to the target server:

q).z.ws: {0N!x}   / handle incoming messages first

/ `$":ws://host:port" builds a connection handle for the request
q)h: (`$":ws://127.0.0.1:5000")"GET / HTTP/1.1\r\nHost: 127.0.0.1:5000\r\n\r\n"

On success the result is a two-item list: the integer connection handle, and the server's HTTP 101 ("Switching Protocols") response.

q)h
(6i;"HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=\r\n\r\n")

If the server refuses the upgrade, the handle is a null integer, 0Ni:

q)h
(0Ni;"HTTP/1.1 400 Bad Request\r\n...")

Other problems, such as a host that does not exist, signal an ordinary q error.

A q client and a q server

  1. Server. Start a q process on port 5000, and define .z.ws to reply with a formatted string:

    q)\p 5000
q).z.ws: {neg[.z.w] "server replied with ", $[10 = type x; x; raze string x];}

  2. Client. Define .z.ws to print what arrives, then connect:

    q).z.ws: {0N!"Client received msg:"; 0N!x;}
q)r: (`$":ws://127.0.0.1:5000")"GET / HTTP/1.1\r\nHost: 127.0.0.1:5000\r\n\r\n"

  3. Client. Send messages on the handle from the connection response, r 0:

    q)neg[r 0]"test"      / a string
q)neg[r 0]0x010203    / a byte vector

    The client console shows the server's replies:

    "Client received msg:"
"server replied with test"
"Client received msg:"
"server replied with 010203"

  4. Client. Close the connection, again using the first item of the connection result:

    q)hclose r 0

Authentication and security

On the server

A WebSocket connection starts as an HTTP Upgrade request, so secure the handshake with the .z.ac HTTP authentication callback. It integrates with standard mechanisms such as LDAP, OAuth 2.0, and OpenID Connect.

On the client

To authenticate a client, add credentials to the HTTP Upgrade request header, such as a bearer token or a session cookie.

For basic access authentication, put the username and password in the connection string:

q)`:ws://username:password@127.0.0.1:5001 "GET / HTTP/1.1\r\nHost: 127.0.0.1:5001\r\n\r\n"

Secure connections (WSS)

For an encrypted WebSocket connection (WSS), first configure the q process for SSL/TLS, then use the wss:// scheme in place of ws://.

Or handle TLS outside q

A tool such as stunnel can wrap a plain WebSocket server, which keeps TLS termination out of the q process altogether.

Serialize data

JSON

JSON is the standard format for web communication, and q supports it natively. .j.k parses an incoming JSON string into a q dictionary, and .j.j serializes q data into a JSON string.

/ parse JSON, read "cmd", and reply with a JSON confirmation
.z.ws: {
  d: .j.k x;                                / JSON string to dictionary
  res: `cmd`status!(d`cmd; "received");      / the response dictionary
  neg[.z.w] .j.j res                        / serialize to JSON and send
 }

The q IPC format (c.js)

For better performance with q-specific data types, send binary serialized data instead. The client then has to deserialize that format — in a browser, with the c.js library.

In q, -8! encodes data to bytes and -9! decodes bytes back to a q object.

The wslogin.htm example sends a JavaScript dictionary to q using c.js:

  1. Start the q server on port 5000. Its .z.ws decodes the incoming byte vector, takes the values from the resulting dictionary, and sends them back serialized:

    q)\p 5000
q).z.ws: {neg[.z.w] -8!value -9!x;}

  2. Open wslogin.htm in a browser, with c.js in the same directory.

  3. Click login. The browser serializes the JavaScript dictionary {u:"user", p:"At0mbang."} and sends it.

  4. The server deserializes it into the q dictionary `u`p!("user";"At0mbang.") and returns the serialized values, ("user";"At0mbang."), to the browser.

Performance and encoding

Compression

q supports permessage-deflate WebSocket compression, which reduces network traffic. Client and server negotiate it during the handshake. In Chrome DevTools, a successful negotiation shows the Sec-WebSocket-Extensions: permessage-deflate header in the HTTP response.

To measure the effect, use \ts. Compressed data takes longer to queue, for more CPU, and uses less bandwidth:

q)v: 10000#.Q.a                        / compressible data

/ queue 1000 messages uncompressed: quicker to queue, more memory and bandwidth
q)\ts do[1000; (-5)v]; show sum each .z.W
5| 10004000
6| 0
14 20610976

/ queue 1000 messages compressed: slower to queue, far smaller payload
q)\ts do[1000; (-6)v]; show sum each .z.W
5| 0
6| 47022
94 4354944

Turn compression off per connection

To disable compression for one client connection, for example to favor lower CPU use over bandwidth, send the sec-websocket-protocol HTTP header with the value kxnodeflate:

// a JavaScript client disabling compression
let ws = new WebSocket(url, "kxnodeflate");

UTF-8 encoding

Text messages must be UTF-8

The WebSocket protocol requires every text message to be UTF-8 encoded. Sending text in another encoding signals a 'utf8 error.

A real-time data publisher

This example builds a publish-and-subscribe service: a q process pushes live trade and quote updates to a web browser.

Set up

  1. Download the project files from kxcontrib/websocket.
  2. Run q pubsub.q to start the publisher.
  3. Run q fh.q to start a feed handler that generates sample data.
  4. Open websockets.html in a browser to watch the updates arrive.

The server: subscription management

The web client subscribes to server-side functions by name. The server's .z.ws evaluates each incoming message, which lets the client call functions such as sub:

/ pubsub.q: let the client call functions directly
.z.ws: {value x}

As the security warning above explains, evaluating client input like this is safe only on a trusted network. In production, restrict the client to an allowlist of functions.

A subs table records the active subscriptions: the client handle, the function subscribed to, and its arguments.

q)subs: 2!flip `handle`func`params!"is*"$\:()

When a client connects, it calls sub, which records its subscription:

/ x: the function to subscribe to, y: the parameters for it
sub: {`subs upsert (.z.w; x; enlist y)}

The server: publishing

A .z.ts timer calls pub for each subscription. pub runs the subscribed function and pushes the result to that client, serialized to JSON by .j.j:

/ pubsub.q
pub: {
  row: (0!subs) x;
  / run the subscribed function, such as getTrades`AAPL`GOOG
  newData: (value row`func) row`params;
  / send the update to the client as a JSON string
  (neg row`handle) .j.j `func`result!(row`func; newData);
  }

Set the timer to publish every subscription once a second:

.z.ts: {pub each til count subs}
\t 1000

The client: handling updates

In the browser, the JavaScript onmessage handler parses the incoming JSON. The server includes an identifier so the client can route each update to the right part of the page — here, the name of the subscribed function:

// websockets.html
ws.onmessage = function(e) {
    // parse the JSON string from the server
    var d = JSON.parse(e.data);

    // route on 'func' to the matching UI update handler
    switch(d.func){
        case 'getSyms'   : setSyms(d.result);   break;
        case 'getQuotes' : setQuotes(d.result); break;
        case 'getTrades' : setTrades(d.result); break;
    }
};

Next steps