Skip to content

Client logging and troubleshooting

This page explains how to capture SAPI client log output, set log levels, and trace a client request through the server logs.

SAPI generates its own log output, but it doesn't write it anywhere. Except in Java, you register a callback and route the output into your application's logging framework. A client that registers no handler produces no SAPI logs, which makes connection problems hard to diagnose — register one before you call connect.

Register a log handler

The callback receives a level, a category, and a message.

void logMessage(int level, const char *category, const char *message)
{
    printf("%d [%s] %s\n", level, category, message);
}

// Register before connecting.
sapiRegisterLogHandler(logMessage);

internal class Program
{
    private static void Main()
    {
        // Register before connecting.
        Kxs.RegisterLogHandler(LogMessage);

        Kxs kxs = new();

        // ...
    }

    public static void LogMessage(int logLevel = 1, string logCategory = "",
                                  string logMessage = "")
    {
        Console.WriteLine($"{logLevel} [{logCategory}] {logMessage}");
    }
}

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class Execute {
    private static final Logger LOG = LoggerFactory.getLogger(Execute.class);

    // ...
}

The Java library uses slf4j and registers the log handler automatically. Configure the destination and levels through your slf4j binding.

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
    handlers=[
        logging.FileHandler("sapi.log", "a", "utf-8"),
        logging.StreamHandler(),
    ],
)

LEVEL_MAP = {
    1: logging.DEBUG,     # VERBOSE
    2: logging.DEBUG,     # DEBUG
    3: logging.INFO,      # INFORMATIONAL
    4: logging.WARNING,   # WARNING
    5: logging.ERROR,     # ERROR
    6: logging.CRITICAL,  # FATAL
}

def log_message(level: int, cat: str, msg: str) -> None:
    logging.getLogger(cat or "CLIENT").log(
        LEVEL_MAP.get(level, logging.INFO), f"[{cat}] {msg}")

sapi = SAPI()
sapi.register_log_handler(log_message)

In C and C#, the log handler is registered globally rather than per connection.

Log levels

The level passed to the callback is a number:

Level Meaning
1 Verbose
2 Debug
3 Informational
4 Warning
5 Error
6 Fatal

Map these to your own framework's levels, as the Python example above does. setLogLevels changes the numeric value SAPI reports for each of its six levels, which is how you align SAPI's numbering with a framework that numbers levels differently.

Where client-side logs are written

Your log handler decides where SAPI's own output goes. Two other sets of files are written on the client host regardless:

  • RT replicator logs. In non-bridge mode, RT starts replicator subprocesses that write their own logs. Their level is set by the consoleLogLevel, logLevel, replFileLogLevel, and replConsoleLogLevel fields of the RT configuration. As with the SAPI log handler, SAPI doesn't dictate where these logs go, whether they rotate, or whether they count against the disk sizing for rtDir — that's determined by the client's own configuration and environment. See RT configuration fields.
  • RT data logs. Also in non-bridge mode, published messages are written to the directory set by rtDir before transmission. For sizing and the errors you see when this directory is wrong, see Client-side RT log files.

Trace a request through the server

Two values connect a client to the server logs:

  • instanceName, which SAPI puts in the header of every request. Set it to something identifiable rather than accepting the generated default, so you can find your client in server-side logs.
  • reqCorr, the request correlator returned in the response object. It appears in the log entries of every process that handled the request.

Server-side log entries for a client request look like this:

2023.06.28 13:34:57.924 DEBUG [maint0] SAPI {maint0-1} Sending .kxsi.trace to kxsGW_A, corr=1 args=()

For how to read these entries and follow a correlator across processes, see Understand log entries and correlators and Look up log files.

Common problems

The connection events that SAPI raises are the quickest way to see how far a client got. Register handlers for them, or wait on them, before you call connect. See Connection event callbacks.

No event fires when a connection is lost. The client reconnects on its own and raises the Gateway connected event again, so the only signals you get are a failed query in the meantime and the reconnect event afterwards.

Discovery is unreachable

The client can't connect to any endpoint in discoverySvcHosts, so it never reaches the Gateways.

  • Check that qetcd is running on the target node.
  • Check that the port is the base port + 3 (or + 4 for TLS) rather than the base port.
  • Check that the firewall allows the connection.
  • Check that your TLS settings match the port in use.
  • If Discovery works but the client still can't query, check whether the all Gateways connected event fires. If it doesn't, at least one Gateway has no connection.

RT is unreachable

The client connected and registered a publisher, but messages aren't reaching the server.

  • Check whether the stream is configured with external: true, and whether the feed maps to the stream.
  • In non-bridge mode, check whether local RT log files are accumulating rather than being truncated, which indicates that transmission is blocked rather than publishing. SAPI creates a log directory per publisher by appending the instance name and topic to rtDir.
  • Unless you set configUrl, SAPI writes an RT configuration file named <instance>_<topic> under configPath, filled with the feed endpoints from Discovery. Open it to see which endpoints the client is using.
  • Closing the client flushes local RT logs to the server. By default the flush has no timeout, so a close can wait indefinitely while the server is unreachable. Set rtFlushTimeoutSecs to bound it.

For confirming that data reached RT, see About RT.

Failure to bind the listening port

SAPI can't bind a port from listenPorts for inbound Response Process connections.

  • Check that the range is wide enough for the number of clients on the host.
  • Check whether another process holds the ports.

Query failures and Response Process connection failures

The client calls an API and the call times out even though the Gateway accepted the request.

This is the failure mode to check first when queries time out but publishing works, because Response Process connections run from the server to the client, the opposite direction to everything else.

  • Check whether the response connection accepted event fires. It fires with the remote IP address when the server connects back. In C, C# and Python the event fires before connect returns, so you can check straight after the call. In Java, listeners miss the events raised during the initial connect, so you can't use this check at startup.
  • If the event never fires, the server can't reach the client. Check that advertiseHosts resolves to an address the server can reach, that the client's listenPorts range is open inbound on the client's firewall, and whether the client host is behind NAT.
  • If the event fires with an unexpected address, suspect NAT or the wrong network interface.

See Network requirements.

Next steps