Troubleshooting¶

This guide is meant to help users diagnose system and configuration issues while running the kdb Insights Database.

General tips¶

Work with proper versions of the images. DA, RC, GW, Agg, and SM must be the same version. Using mismatched versions may cause unexpected problems.
Use the log correlator to trace the query through the logs. By default, the GW generates a random GUID as the correlator, but you can supply your own in the request to make scanning the logs easier.
```
(`myAPI;`my`args!1 2;`myCallback;``logCorr!(`;"myLogCorrelator"))
```

DAP configuration issues¶

For reference, the daps section of the shard file in the package should look similar to this:

daps:                # Start of DAP configuration section
  instances:
    hdb:             # Start of config for DAPs with KXI_SC env set to "hdb"
    mountName:hdb  # Mount this DAP to provide read access to (must match name in `mounts` section)
    rdb:             # Start of config for DAPs with KXI_SC env set to "rdb"
    mountName:rdb

A DAP generates fatal errors if the package is misconfigured. The following sections describe these errors and provide solutions.

Missing `dap` from elements section of assembly¶

This issue occurs when the package does not have a daps section in the shard file. This can be due to the section being missing, or a simple typo as in the below example.

dapp:                 # Typo in "daps"
  instances:
    hdb:
      mountName:hdb
    rdb:
      mountName:rdb

Missing service class config under `dap` instances of assembly¶

If this startup error occurs, it is because the KXI_SC environment variable set for the DAP does not match any name under daps.instances of the package. To resolve this, compare the KXI_SC set for the DAP with those in its package.

Mount does not exist within assembly `mounts`¶

DAPs fail to start if the mount they are configured to provide access to does not exist within the package. This mount is set by daps.instances.*.mountName. For example, in the above snippet the DAP hdb has a mount called hdb, and the DAP rdb DAP has a mount called rdb.

To resolve this error ensure that a mounts section exists for the package, and that the mount names there match those defined in the DAP config. An example mounts section for the above might look like this:

mounts:
  rdb: # This line defines the name of the mount that `mountName` references
    type: stream
    baseURI: none
    partition: none

  hdb:
    type: local
    baseURI: file:///data/hdb
    partition: date

Error mounting database: `.DS_Store`¶

.DS_Store is a hidden file automatically created on macOS, and used to store custom attributes of a given folder. The presence of these files results in a FATAL error when re-starting the database. To resolve this issue, use the following command to delete all these files inside the database folder.

find $DB_PATH -name .DS_Store -type f -exec rm {} \;

No mount dir defined - set `baseURI` of associated mount¶

The mounts configuration needs to have the baseURI set for the mount so that the DAP can find where to mount the on-disk data from.

Invalid table configuration, exiting¶

DAPs fail to start if one or more schemas are misconfigured. Before failing with the above error the DAP outputs any detected errors with the schema definition. Possible errors are summarized below:

Error Message	Remediation
`assembly.table.%s has invalid name, consider using`	Table name is invalid and must be changed
`assembly table.%s missing required key(s):`	Add required key to table
`assembly.table.%s columns missing required key(s):`	Columns must have a name and type. Add the missing key
`assembly.table.%s has no columns`	Remove table, or add columns
`assembly.table.%s column name: '%s' is invalid, please consider using '%s'`	Change noted column name to suggestion or something else
`assembly.table.%s.%s column type: %s is invalid`	Column type is invalid, change to supported type
assembly.table.%s missing required key(s): `prtnCol	Partitioned tables require a `prtnCol` key
`assembly.table.%s.%s prtnCol is not timestamp type`	The `prtnCol` key must be of a timestamp type
`assembly.table.%s %s: %s conflict with %s: %s. Can NOT apply %s# to %s column of %s table.`	Add the referenced column with attribute to referenced `sortCols` key
`assembly.table.%s: %s conflict with parted attribute: Can NOT apply parted attribute to vector columns %d`	A vector column type cannot have a parted attribute applied to it

No assembly labels defined - set labels in assembly¶

The RC's routing of DAPs relies on there being labels defined for the package, so DAP/RC registration expects that the DAP has some labels attached to the package. If none are defined, then the DAP startup fails. To fix this, add at least one label to the labels section of the package shard file. For example, to add a region label of eu, add this to the package shard file:

labels:
  region: eu

Queries¶

My queries are not returning data¶

When no data is returned, either the query is succeeding but finding no data given the parameters of the request, or the request itself is failing. The way to tell the difference is to look at the rc, ac, and ai in the header of the response. If the rc and ac are 0, then from the DAP's perspective the queries have succeeded and there was no data to return. If either the rc or ac are non-zero the ai should give more information about the specific error encountered.

My queries are refused¶

If the Service Gateway returns immediately with an error of the form:

'Resource Coordinator connection not established

Then this indicates that the Service Gateway has not connected to the RC. Alternatively, you can check the Service Gateway logs for the following messages:

// Error
INFO  com.kx.sg.IPCService - Coordinator connection establish error: java.net.ConnectException: Connection refused (Connection refused). Will try again in 5 seconds...

// Success
INFO  com.kx.sg.IPCService - Connected to coordinator <rc_address>

If the Error message appears multiple times without the Success message, then the Service Gateway is unable to connect to the RC. Check the connection details in the Service Gateway configuration.

My queries always fail¶

This can happen for multiple reasons. The first element in the response is the response header (see "header" page) and the rc, ac, and ai fields (see "codes" page) contain details on the error. The following table summarizes the possible routing errors from the RC.

`rc`	`ac`	`ai`	cause	solution
`NOT_READY (58)`	`NOT_READY (20)`	`"No resources connected"`	No DAPs (or other RCs) are connected to the RC.	See RC-DA and RC-RC.
`NOT_SUPPORTED (12)`	`NOT_SUPPORTED (17)`	`"SQL library not loaded"`	Attempting to make a SQL query in unsupported q version.	Upgrade q.
`APP (5)`	`*`	`"SQL parse error"`	Error parsing SQL statement.	Fix query.
`APP (5)`	`ARG (23)`	`"bad args purview dimensions"`	Invalid routing arguments.	Check query, labels must be symbols, `startTS` and `endTS` are timestamps.
`APP (5)`	`ARG (23)`	`"startTS >= endTS"`	Invalid request timestamps.	Start time must be less than end time.
`NOT_SUPPORTED (12)`	`DOMAIN (27)`	`"outside all RC taxonomies"`	Request for unknown label values.	Check that labels in query match labels of desired DAPs (package shard file). If this looks correct, then the desired DAPs or RCs may not be connected properly. See RC-DAP and RC-RC.
`NO_DEST (56)`	`NO_DEST (17)`	`"no Agg"`	Can't find an aggregator.	See RC-Agg.
`TIMEOUT (45)`	`ERR (10)`		Request timed out before completing	See Timeouts.
`DISCONNECT (42)`	`ERR (10)`	`"AGG restarted. You may not have enough memory."`	Aggregator process has disconnected from RC. Checking process logs should point to a more specific reason (e.g `wsfull` in a OOM event).	Increase assigned resources. See Aggregator crash.
`DISCONNECT (42)`	`ERR (10)`	`"DAP restarted. You may not have enough memory."`	A DAP has disconnected from RC. Checking process logs should point to a more specific reason (e.g `wsfull` in a OOM event).	Increase assigned resources. See DAP crash.
`DISCONNECT (42)`	`ERR (10)`	`"Secondary RC died"`	A peer RC helping to service a multi-RC request has disconnected.	See Resource coordinator crash.

Timeouts¶

Constant timeouts are usually due to the RC being unable to find a DAP to cover a temporal region of the request, which would point to a DAP not correctly connecting with the RC. When the RC encounters a timeout it augments the log and ai code of the request with metadata about portions of the request that did not complete.

The metadata includes the RC responsible for the request, the current status of the request, and details of items in the queue. The queue details include the labels and temporal range for the request being serviced, and the reason the portion was not sent to a DAP to be serviced.

No DAP covers labels/time range¶

The RC reports this in cases where no DAP has registered with the appropriate time ranges or labels. If you can attach or open an IPC handle to the RC process and there are no active requests, run the following command on the process:

.sgrc.i.summarizeDAPs[]

This returns a table with the labels, startTS and endTS for each contiguous set of label values. If any label combination does not cover from startTS=-0Wp to endTS=0Wp, one or more of the DAPs with those labels is missing.

Busy executing another API¶

Occurs when one or more DAPs needed to service the request were too busy servicing another API. Depending on the request parameters, there might not be anything wrong other than the system was not sufficiently resourced to handle the query load. If this issue occurs often, some possible remediation steps are to look into the query code to see if it can be optimized (in the case of a user defined analytics), or to add more DAPs to the system to service queries.

DAP reference vintage %n does not match %s reference vintage %n¶

DAPs keep track of where they are in the RT stream any time they ingest reference data, and they report this position (or vintage) to the RC. When sending a request to DAPs, the RC checks that the vintages of the portions of the request match, so as to minimize disparities in the data reported. This timeout error can occur when one or more DAPs are falling behind in ingestion, and so the reference vintage they report is lagging behind the global value. When this occurs, it is best to check the logs and determine the cause of any ingestion issue that might be affecting particular DAPs.

Unavailable for unspecified reasons¶

When this reason is specified, the DAP has marked itself unavailable to the RC. This usually occurs when the DAP is acting on a reload signal and is thus performing in-memory data purges, or garbage collect calls. The DAP logs contain details of what it was doing during the time of the request.

This can occur in cases where the DAP has started and registered with the RC, but has remained marked as "unavailable". In this case, the issue could be that it has not ingested its end-of-replay marker. In a healthy system you should be able to see the DAP log both of these:

INFO DA Injecting end of log replay message, marker
INFO DA Finished RT log replay

In cases where there is an ingestion issue, you see only the first log message. If you have access to the DAP console, you can confirm it has not received the end-of-replay message by querying .da.EorReceived in the DAP that is unavailable and confirming the value is 0b.

Unknown reason for not assigning request portion to DAP(s)¶

Occurs when the RC does not know why the request portion was unable to be served. This requires deeper investigation into the logs and state of the system.

Process crashes¶

Any process in the query path can exit and restart while a query is in flight. The most common cause is memory exhaustion: the process runs out of the memory allocated to it and is terminated with an out-of-memory (OOM) error. Where you have access to the cluster, confirm an OOM from the pod state with kubectl describe pods/<POD NAME>, which reports the following for the terminated container:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Refer to Sizing kdb Insights deployments for the memory each process requires.

What a client sees depends on which process crashed, because the failure is reported by the nearest surviving process upstream of the crash:

If a DAP or an Aggregator crashes, the RC that owns it fails the affected requests and reports the reason in the response header (rc, ac, and ai).
If an RC crashes, the Service Gateway fails the requests it had dispatched there. The Service Gateway composes these responses itself, so they carry only rc, ac, and ai for qIPC clients, and a plain-text message with no header at all for REST clients. Expect less detail than in an RC-reported failure.
If the Service Gateway crashes, no process is left to answer at all, so what the client sees is decided by whatever sits in front of the Service Gateway rather than by the database.

Three rules explain the HTTP status codes below:

A failure reported by the RC in the response header is returned to REST clients as 400 Bad Request, regardless of the value of rc. The failure detail is in the ai field of the returned header, not in the status code. A 400 here does not mean the request was malformed.
A failure generated by the Service Gateway itself, because it never received a response from a Resource Coordinator, is returned to REST clients as 500 Internal Server Error with a plain-text body and no header.
A request the Service Gateway refuses before dispatching it, because no Resource Coordinator is available to send it to, is returned as 503 Service Unavailable.

Only the Service Gateway's own crash falls outside these rules, because no process is left to apply them; refer to Gateway crash for more details.

qIPC clients always receive the response header, so they see the same rc, ac, and ai values as REST clients but no status code. Refer to REST vs qIPC for more details.

Summary¶

Process that crashed	REST status	`rc`	`ac`	`ai` / body
DAP, while executing the request	`400`	`DISCONNECT (42)`	`ERR (10)`	`"DAP restarted. You may not have enough memory. Please add parameters or filters or reach out to your admin to resize the DAP."`
DAP, before the portion was assigned	`400`	`TIMEOUT (45)`	`ERR (10)`	Timeout metadata; see Timeouts
Aggregator	`400`	`DISCONNECT (42)`	`ERR (10)`	`"AGG restarted. You may not have enough memory. Please add parameters or filters or reach out to your admin to resize the AGG."`
RC, in-flight request	`500`	`ERR (10)`	`ERR (10)`	`"Coordinator connection has closed"`
RC, in-flight request that times out first	`500`	`TIMEOUT (45)`	`ERR (10)`	`"Request was timed out by the service gateway"`
RC, new request while no RC is ready	`503`	n/a	n/a	`"No Coordinator connections are available and ready for service"`
Peer RC in a multi-RC request	`400`	`DISCONNECT (42)`	`ERR (10)`	`"Secondary RC died"`
Scratchpad, in-flight request	`500`	`ERR (10)`	`ERR (10)`	`"Coordinator connection has closed"`
Scratchpad, not yet restarted	`503`	n/a	n/a	`"No Coordinator connections are available and ready for service"`
GW	Set by the proxy in front of the GW, commonly `502`	n/a	n/a	Not generated by the database

DAP crash¶

When the RC detects that the handle to a DAP has closed, it removes that DAP from its caches and immediately fails any request the DAP was executing, rather than letting the request run to its timeout.

The RC logs the following:

WARN  Removing DAP=<dap>
DEBUG Handle to DAP closed, failing request, dap=<dap>

The affected requests are returned with rc=DISCONNECT (42), ac=ERR (10), and:

DAP restarted. You may not have enough memory. Please add parameters or filters or reach out to your admin to resize the DAP.

For REST clients this is a 400 response whose body carries the header, as shown in Aggregator crash. The RC also tells the Aggregator to forget the request, so no partial aggregation is left behind.

A DAP crash does not always fail the request

The RC only fails a request when the crashed DAP was the one executing it. If the DAP crashed before the RC assigned it a portion of the request, the portion stays in the RC queue and is served by another DAP covering the same purview. Where no other DAP covers that purview, the request instead fails with rc=TIMEOUT (45), and the ai field reports No DAP covers labels/time range. Refer to Timeouts.

Warning

A query sent directly to a database, rather than routed through an RC, does not go through the path above. The Service Gateway holds its own connection to the DAP, so when that connection closes the Service Gateway fails the request itself with rc=ERR (10), ac=ERR (10), and "Database connection has closed", returned to REST clients as HTTP 500. The Service Gateway logs:

DEBUG com.kx.sg.kxi.IPCService - Responding to request=<logCorr> that database assembly=<assembly>, target=<target> has closed connection

Remediation: confirm the process restarted because of an OOM, then size the DAP for both its in-memory data and its query workload. Refer to Data Access Process (DAP) in the sizing guide. Allocating multiple DAPs per tier and label set keeps queries serviceable while one restarts; refer to Query resilience.

Aggregator crash¶

When the RC detects that the handle to an Aggregator has closed, it fails every request that was destined for that Aggregator. The RC logs the following:

WARN  Closing Agg, addr=<agg_host:agg_port>
DEBUG Handle to aggregator closed, failing request, agg=<agg>

The affected requests are returned with rc=DISCONNECT (42), ac=ERR (10), and:

AGG restarted. You may not have enough memory. Please add parameters or filters or reach out to your admin to resize the AGG.

A REST client receives HTTP 400 with the header in the response body:

{
  "header": {
    "auditID": "d45567a4-27d6-4935-b4c4-e5fda0e6a189",
    "corr": "3a447d76-0533-4576-9214-c18227828e4d",
    "logCorr": "d45567a4-27d6-4935-b4c4-e5fda0e6a189",
    "rcvTS": "2026-02-23T16:49:19.442000000",
    "http": "json",
    "api": ".uda.weatherTimed",
    "rc": 42,
    "ac": 10,
    "ai": "AGG restarted. You may not have enough memory. Please add parameters or filters or reach out to your admin to resize the AGG."
  },
  "payload": []
}

Because aggregation happens after all DAPs have returned their partial results, an Aggregator is most likely to crash on queries whose combined partial results are large. Use the logCorr value to trace the query through the Aggregator logs.

Remediation: reduce the volume the aggregation has to hold, by narrowing the time range of the query or adding filters, or increase the memory allocated to the Agg. Allocating multiple Aggs to each RC lets the RC route around a restarting Agg; refer to Query resilience.

Resource Coordinator crash¶

An RC crash is detected by the Service Gateway rather than reported by the RC.

When the Service Gateway sees the connection to an RC close, it allows a grace period for any straggling responses to arrive, then fails every request that had been dispatched to that RC. The grace period is the Service Gateway's default timeout, set by KXI_SG_TIMEOUT (default 60000 ms). The Service Gateway logs the following:

INFO  com.kx.sg.kxi.IPCService - Allowing <timeout> (ms) for straggling requests to complete before timing out requests that were dispatched to <rc>.
DEBUG com.kx.sg.kxi.IPCService - Responding to request=<logCorr> that coordinator has closed connection

The failed requests are returned with rc=ERR (10), ac=ERR (10), and the message:

Coordinator connection has closed

Because the Service Gateway generates this response itself, REST clients receive HTTP 500 with a plain-text body containing the message, not a JSON header and payload.

Two variations are worth noting:

If the request's own timeout expires before the grace period ends, the Service Gateway returns its timeout response instead: HTTP 500 with rc=TIMEOUT (45), ac=ERR (10), and the message Request was timed out by the service gateway.
A streaming request is not failed. The Service Gateway cannot append to a response buffer that is already being written, so it lets the database finish streaming and logs:
```
WARN  com.kx.sg.kxi.IPCService - Coordinator associated with streaming request=<corr> has closed connection. Allowing database to continue streaming response...
```

Requests that arrive after the RC has gone and before it has re-registered find no coordinator to dispatch to, and are refused with HTTP 503:

No Coordinator connections are available and ready for service

Where entitlements are enabled with KXI_ENTITLEMENTS_ENABLED=true, the same response reads No Coordinator connections are available and ready for service or the user is not entitled. A 503 here does not on its own mean the user lacks an entitlement.

In a multi-RC deployment, a request can span several RCs. If the primary RC survives but a peer RC servicing part of it crashes, the receiving RC fails the request itself with rc=DISCONNECT (42), ac=ERR (10), and ai of "Secondary RC died". The RC logs:

WARN  Failing RC, handle=<handle>
DEBUG Handle to RC closed, failing request, handle=<handle>

Remediation: allocate multiple RCs so the Service Gateways can round-robin to a surviving RC while one restarts, and confirm that the DAPs and Aggregators re-register with the restarted RC. Refer to Query resilience and RC-DAP.

Service Gateway crash¶

The Service Gateway is the entry point for queries and the process that writes every response described above back to the client, whether the header came from an RC or the Service Gateway composed the message itself. When the Service Gateway crashes there is nothing left to do either: in-flight requests are never answered and the connection simply closes.

This is the one crash in this section whose status code the database does not choose. What the client sees is produced by whatever sits in front of the Service Gateway:

Connecting to the Service Gateway directly, the client sees a dropped connection or a connection refused, with no HTTP status, no rc, ac, or ai, and no correlator to trace.
Behind an ingress or API gateway, as in kdb Insights Enterprise, the proxy answers on the Service Gateway's behalf once it has no healthy endpoint to route to. 502 Bad Gateway is the usual response, but the exact status and body are the proxy's, so treat them as an indication to check the Service Gateway rather than as a database error code.

Because a proxy-generated response carries no header, diagnose this crash from the pod state rather than the response body:

kubectl describe pods/<GW NAME>

Two windows during a Service Gateway restart look similar to a client but are distinct:

While the Service Gateway is down or its readiness probe is failing, the pod is not in the ingress endpoints and requests never reach a Service Gateway, giving the proxy response above.
Once the Service Gateway is ready but before it has connected to an RC, requests do reach a Service Gateway and are refused by it, with the HTTP 503 described in Resource coordinator crash. To confirm you are in this window rather than the previous one, look for the Service Gateway's coordinator connection messages described in My queries are refused.

Clients may report the same crash differently

Clients that wrap the HTTP call, such as the kdb Insights Enterprise web interface and the VS Code extension, surface their own wording rather than the status code, and two clients can word the same underlying crash differently. Confirm against the Service Gateway pod state and logs before treating them as separate faults.

Remediation: run multiple Service Gateway replicas so that the ingress always has a healthy endpoint to route to. Refer to Query resilience.

RC-DAP¶

At startup, a DA reaches out to its configured RC and initiates connection. The DA logs should have a message of the form:

[rdb] INFO  DA Initializing RC connection, rc=<KXI_SG_RC_ADDR>
[rdb] INFO  KXDSCONN Attempting to connect to rc at <KXI_SG_RC_ADDR>
[rdb] INFO  KXDSCONN Connected to proc=rc addr=<KXI_SG_RC_ADDR> handle=<handle>
[rdb] INFO  DA Registering with RC

If this message does not appear, the DA did not connect to the RC. Verify connection details.

If connection is established, the RC logs should contain a message of the form:

[KXI-SG-RC-sg-rc] INFO  SGRC Received DAP registration request, handle=<handle>
[KXI-SG-RC-sg-rc] INFO  SGRC Registering valid DAP, handle=<handle> asm=<dap_package_name> instance=<KXI_SC> addr=<KXI_NAME>:<KXI_PORT>

Ensure that the purview matches the expected values. The RC uses these values (except ver) to route requests; they must the labels in your request.

[KXI-SG-RC-sg-rc] INFO  SGRC Adding new purview ID, id=1 labels=[labelKey=`labelValue]

If there is no registration message, look for one of the following error messages in the RC log:

// Incorrect args to the registration call.
ERROR [KXI-SG-RC-sg-rc] SGRC Incorrect DAP registration param types

// Bad purview.
ERROR [KXI-SG-RC-sg-rc] Invalid DAP purview registration

// Invalid metadata.
ERROR [KXI-SG-RC-sg-rc] Invalid DAP metadata

// Invalid schema definitions.
ERROR [KXI-SG-RC-sg-rc] Invalid DAP schemas

// Unexpected update.
ERROR [KXI-SG-RC-sg-rc] Unknown DAP update

// Invalid purview on update.
ERROR [KXI-SG-RC-sg-rc] Invalid purview update, rejecting DAP

Any of these messages indicates a bug. Failing any of the above, look for any error messages in the DAP or RC that would indicate that initialization failed before the registration was attempted, which would also indicate a bug. Contact technical support.

RC-RC¶

At startup, if using multiple RCs, for every RC pair, one RC should initiate a connection to the other. To ensure the RCs are configured to find each other, look for the following message:

INFO  [KXI-SG-RC-sg-rc] SGMRC Setting multi-RC mode to <mode>

If the message is not present, one of the following messages should appear:

INFO  [KXI-SG-RC-sg-rc] SGRC Setting mono-RC mode

// OR

FATAL  [KXI-SG-RC-sg-rc] SGRC Unrecognized 'KXI_DISC_MODE' mode: <mode>

These indicate the KXI_DISC_MODE environment variable is either unset (former) or set to an unrecognized value (latter). See "SG configuration page" for details.

Assuming discovery mode is correctly set, for every pair of RCs, one RC should attempt to connect to the other (which one is not guaranteed). Say, RC_0 connects to RC_1. Ensure the following messages appear in the logs:

// RC_0
INFO  [KXI-SG-RC-sg-rc-0] SGRC Attempting connection to RC, name=<rc_names> hostport=<rc_host:ports>

// RC_1
INFO  [KXI-SG-RC-sg-rc-1] SGRC Registering RC, name=<rc_0_name> host=<rc_0_host> port=<rc_0_port>
INFO  [KXI-SG-RC-sg-rc-1] SGRC Reciprocating registration with RC

// RC_0
INFO  [KXI-SG-RC-sg-rc-south] SGRC Registering RC, name=<rc_1_name> host=<rc_1_host> port=<rc_1_port>

In the event of communication failure, there may be a log message of the form "Error sending: <error>", but the RCs should retry in a few seconds, so these can be ignored, unless they repeat constantly.

If these messages do not appear in either RC, then one or both are incorrectly configured.

If using "kubernetes" discovery mode, ensure that the annotations are correctly set in the pod metadata:

kind: Pod
metadata:
  annotations:
    kxi-kdisc: "${Name of the RC container}: sc=KXI-SG-RC port=${Container port number}" # Check these!

The name of the RC container should match the name spec.containers.name of the RC container:

spec:
   containers:
   - name: ${Name of the RC container}

RC-Agg¶

At startup, the Aggregator should open a connection to its RC. The following message should appear in the Agg's logs:

INFO [KXI-SG-AGG-sg-agg] SGAGG Attempting to register with RC, hp=:<rc_host>:<rc_port>
INFO [KXI-SG-AGG-sg-agg] SGAGG Connected to RC, handle=<handle_number>

If instead, the following messages appears:

FATAL [KXI-SG-AGG-sg-agg] Must define KXI_SG_RC_ADDR env variable

then the KXI_SG_RC_ADDR environment variable is not set (should be KXI_SG_RC_ADDR="<rc_host>:<rc_port>"). If the following message continually pops up:

INFO [KXI-SG-AGG-sg-agg] SGAGG Attempting to register with RC, hp=:<rc_host>:<rc_port>
WARN [KXI-SG-AGG-sg-agg] SGAGG Unable to connect to RC

check to make sure the RC's host and port are set.

Once connection is established, a confirmation message should appear in the RC's logs:

INFO [KXI-SG-AGG-sg-agg] Registering Agg, host=<agg_host> port=<agg_port> handle=<handle_number>"

DAP¶

Ingestion events¶

Hard reset and skip forward¶

Sometimes an issue can occur during ingestion when an RT hard reset is triggered or if a DAP attempts to subscribe to a position that has been deleted from the RT logs (possibly due to disk storage issues).

In the case of a hard-reset the DAP will output a log similar to the below (where <POS> is replaced with the stream position of the event):

[rdb] INFO  Encountered hard-reset event at pos=<POS>

and in skip-forward a log similar to this:

[rdb] INFO  Encountered skip-forward event at pos=<POS>

In these situations the DAP can no longer be certain about the state of the log or the database and so must rely on the Storage Manager to writedown whatever data it has so that the system can have a consistent database state. When these events occur DAPs will be unavailable for query and the Storage Manager will trigger an immediate writedown. At the end of the writedown the DAPs will receive the reload signal and the system will have recovered. When this happens the DAP will output a log like this:

[rdb] INFO  Recovered from RT event at pos=<POS>

where <POS> will match the position of the originating incident. Between the points where the incident is triggered and the process has recovered, the DAP may log that it is ignoring table ingestion updates, this is expected behavior.

Early reload¶

A similar event called and "early-reload" can occur in the DAP. This occurs when a DAP receives a reload signal via qipc before it ingests the triggering prtnEnd signal. In this case recovery behavior is similar to the other ingest events except that the process can remain available for query while it waits to fully recover. During this period the DAP will also ignore table updates until it eventually ingests the prtnEnd which triggered the reload.