Transfer data between nodes¶

This page explains how to use DBC (Database Copy) to copy or move the data of a database, or a subset of it, from one node to one or more other nodes.

DBC is an administrative utility that transfers the data of an entire database, or a subset of it, from a source node to one or more destination nodes. You can use it to add nodes to a live production environment without shutting down active processes or interfering with normal operations.

DBC supports two types of transfer:

Copy — the data stays on the source node.
Move — the data is deleted from the source node after the transfer completes successfully.

You can restrict a transfer by feed and by date range. DBC transfers only full partitions; you can't transfer part of a day's data.

DBC is fault tolerant. You can pause and resume a transfer at any time without losing data or repeating work already done.

Note

The database writer (DBW) merges only tables that are defined on both nodes, and only tables that it writes and replicates. It ignores tables that exist on only one node and logs a warning. If a table that exists on both nodes has a different schema on each, DBW rejects the whole PIT, and the transfer can't merge. Don't make schema changes during a transfer.

How a transfer works¶

The following diagram shows the events that occur when you copy or move a database. The diagram shows the events in a line for clarity, but DBC processes each partition separately, so partition 1 can be transferring while partition 2 is merging.

flowchart TD
    subgraph SRC["Source node"]
        S1["Create PIT (point-in-time snapshot)"] --> S2["Assign transfer ID"]
        S2 --> S3["Send PIT in chunks"]
    end
    subgraph DST["Destination node(s)"]
        D1["Write chunks to local PIT"] --> D2["Map IDs, if required"]
        D2 --> D3["DBW merges PIT, one partition at a time"]
        D3 --> D4["Delete local PIT"]
    end
    S3 --> D1
    D4 --> Q{"Copy or move?"}
    Q -->|Copy| E1["Transfer complete"]
    Q -->|Move| S4["DBW deletes source data at EOD"]
    S4 --> E2["Transfer complete"]

The source node creates a point-in-time snapshot (PIT) of the database, and assigns the transfer an ID made of the source node name followed by the date and time, for example A_2020.02.05.170151.
The source node sends the PIT to each destination node in chunks. It sends the configuration data first, then the IDB (intraday database) as a single partition, then the HDB (historical database) partitions from newest to oldest.
Each destination node writes the chunks it receives to a PIT under its own pitDir.
If the source node belongs to a different cluster, DBC maps the IDs in the transferred data to newly reserved IDs on the destination. See ID mapping.
DBW on the destination node merges the PIT into the database one partition at a time, starting with the IDB. If the destination already has the partition, DBW merges the data into it.
After DBW merges a partition, it records the merged feeds and date range in the loci registry, and DBC deletes the partition from the destination PIT. After DBW merges the last partition, DBC deletes the rest of the destination PIT, including the IDB.
For a move, DBW on the source node deletes the transferred data at the next end of day.

Warning

You can cancel a transfer at any time, but you can't undo a merge. Partitions that DBW has already merged into a destination database stay there.

Note

DBC can't create a PIT while DBW is running an EOX subtask (end-of-interval or end-of-day processing). If you start a transfer then, it fails with PIT creation failure, and you can run it again when DBW is free. Merges into the destination database also wait until DBW is idle.

Point-in-time snapshots¶

A PIT contains the sym file and the tables that DBW writes. On the source node, a PIT consists of hard links to the data rather than copies of it, so it uses little extra space. Hard links can't cross volumes, so if pitDir is on a different volume from the database, DBC creates the links next to the data and links to them from pitDir.

A PIT on a destination node is a full copy of the transferred data and takes the same space as the equivalent database. Make sure the volume that holds pitDir on each destination node has enough free space for the whole transfer. Before the first chunk is written, the destination node checks the free space against the size of the PIT. If there isn't enough, it logs an error and doesn't accept the data.

ID mapping¶

If the source node isn't in the same cluster as the destination node, IDs in the transferred data can clash with IDs that already exist on the destination. Before it merges the IDB, DBC reads the IDs the transferred tables refer to, reserves new IDs on the destination through the master data service, and records the mapping from remote to local IDs. DBW applies this mapping when it merges the data. If the source and destination nodes are in the same cluster, DBC doesn't map IDs.

Configure DBC¶

DBC uses the following parameters in systemParams.yaml. Make any changes in a systemParams.yaml override object, not in your core systemParams.yaml.

Parameter	Description	Default
`pitDir`	The root directory in which DBC stores PITs. The default is set using the `KXS_ROOT` environment variable, which you define in your environment. DBC fails to start if `pitDir` is null. Static.	`${KXS_ROOT}/db/pit`
`dbcBufSize`	The maximum number of bytes read from the PIT at one time. DBC packages the bytes it reads into a transfer chunk, serializes it and sends it to the destination nodes over IPC. Larger chunks can speed up a transfer but can affect other processes. Static.	`1000000`
`dbcCacheChunks`	The maximum number of chunks DBC keeps in memory. DBC keeps the most recently read chunks, and uses them if a destination node asks for a chunk that it has already read. A larger number can speed up a transfer but reduces the memory available to other processes. Dynamic.	`5`
`feedMapFiles`	The files that define your feed-map functions. DBC and DBW load them. See Transfer a subset of feeds. Dynamic.	Empty

Moves also use the dbwPruneChunk parameter. See Move a database.

Before you begin¶

You run every DBC command with the .maint.dbc function in the maintenance console. The function is node neutral: you don't need to run it on one of the nodes in the transfer. The maintenance console communicates with the DBC service that already runs on each node.

q).maint.dbc[`copy;`A;`B]

Get help¶

Run .maint.dbc`help to list each DBC command and its arguments. An asterisk (*) at the end of a command's description marks a command that requires arming, and a plus sign (+) marks a command that requires confirmation.

q).maint.dbc`help
Usage:  .maint.dbc`command  or  .maint.dbc[`command[;param1[;param2[;param3[;param4]]]]]

Commands:
  `help
        Displays this command summary
  `state;`<node>;(verbose)
        Displays the state of copy or move operations to or from the specified node
  `move;`<srcnode>;`<destnodes>;(feeds);(daterange)
        Moves some or all of a node to other nodes*+
  `copy;`<srcnode>;`<destnodes>;(feeds);(daterange)
        Copies some or all of a node to other nodes*+
  `createpit;`<node>;(daterange)
        Creates a point-in-time snapshot of a database or a temporal subset of it*
  `deletepit;`<node>;`<pid>
        Deletes a point-in-time snapshot on a node*+
  `merge;`<nodepair>;`<id>;<feeds>;<datelist>
        Merges a copied database root or PIT into the specified node*+
  `pause;`<node>
        Pauses database copy or move operations to or from the specified node*
  `resume;`<node>
        Resumes database copy or move operations to or from the specified node*
  `cancel;`<node>
        Cancels database copy or move operations to or from the specified node*+
  `throttle;`<node>;(ms)
        Throttles database copy or move operations to or from the specified node*

  Arguments in parentheses (...) are optional

* These commands require arming via <.maint.arm>
+ These commands require user confirmation

Arm DBC commands¶

Every DBC command except help and state is armed. An armed command is critical, and you must arm it in a separate step before you can run it. Some armed commands also ask you to confirm.

q).maint.arm`dbc
Now armed: `dbc

The arming timeout is five minutes by default. You can change it in .maint.ARMPER.

Impact on other processes¶

You can run DBC commands at any time without locking processes or interfering with normal operations. After a transfer starts, the maintenance console is free for other tasks. You can use throttling and the pause and resume commands to control the system resources that you give to a transfer.

Transfer limits¶

DBC supports one transfer at a time for each node, whether the node is the source or a destination. For example, A to B, and A to B and C, are allowed, but A to B and B to C at the same time isn't. If a node is already in a transfer, DBC reports Transfer in progress and doesn't start the new transfer.

Choose feeds and date ranges¶

The feeds and date range arguments are both optional. With no feeds and no date range, DBC transfers the whole database. The date range comes after the feeds, so to specify a date range for all feeds, pass the null symbol (`) as the feeds argument.

Specify feeds as a list of feed IDs, for example 100 101h, or as a list of feed names. DBC transfers only feeds that are configured on the source node. If none of the feeds you specify is configured on the source node, the command does nothing.

Specify a date range as a two-element list of dates. Both ends of the range are inclusive. DBC includes each HDB partition whose date is in the range.

Date range	Value
No date range	`(-0Wd;0Wd)`
Today only	`(.z.d;.z.d)`
Up to and including today	`(-0Wd;.z.d)`
Yesterday only	`(.z.d-1;.z.d-1)`
The last 15 days	`(.z.d-14;.z.d)`

If you use the negative infinity date -0Wd, the range extends indefinitely into the past. If you use the positive infinity date 0Wd, the range extends up to the time DBC creates the PIT. The date range filters HDB partitions only. DBC always includes the IDB and any tables that aren't partitioned, whatever the date range.

Before it starts a transfer, .maint.dbc checks that DBC is armed, that the source and destination nodes are valid and different from each other, and that the start date isn't later than the end date.

Note

DBC doesn't currently activate or deactivate feed subscriptions on the source or destination nodes, whatever the date range. After a transfer with an open-ended range (0Wd), update your feed.yaml configuration yourself so that the destination nodes ingest the transferred feeds, and, for a move, so that the source node stops ingesting them.

Transfer a subset of feeds¶

DBC filters a table by feed only if the transfer covers some, but not all, of the feeds on the source node. To filter a table, DBC needs a feed-map function that identifies the feed of each record. DBC sends any table that doesn't define a feed-map function without filtering.

To use a feed-map function:

Reference the function by name in the feedMapFn attribute of the table's schema file. See Tables and schemas.
Make the function available in memory in both DBC and DBW. Put the functions in a file, and list the file in the feedMapFiles parameter in systemParams.yaml.

A feed-map function takes two parameters (DBW rejects any other rank):

The first parameter is a symbol containing the table name.
The second parameter is the deserialized table.
It returns a list of shorts containing the feed for each record in the table. Every feed must exist in the feed configuration, or the transfer fails with feedMapFn produced nonexistent feed.
It doesn't support unenumerated symbols.

If filtering doesn't remove any records from a file, DBC sends the file as it is.

Warning

If a feed-map file doesn't load, or a function that a table refers to is missing, DBW disables source deletion for moves and logs Pruning disabled due to feed function load failure.

Warning

DBC loads the functions into the DBC process, so they can call any function available to DBC. Test your functions carefully.

Copy a database¶

Syntax	`copy;`<srcnode>;`<destnodes>;(feeds);(daterange)
Optional	`feeds`, `daterange`
Arming required	Yes
Confirmation required	Yes

The command returns as soon as the source node starts the transfer. The message includes the transfer ID.

Example 1: copy an entire database. Copy all the data on node A to nodes B and C.

q).maint.arm`dbc
Now armed: `dbc
q).maint.dbc[`copy;`A;`B`C]
Continue? (y/n)
Transfer of A_2020.02.05.170151 has begun

Example 2: copy individual feeds. Copy the data of feeds 100 and 101 on node A to nodes B and C.

q).maint.dbc[`copy;`A;`B`C;100 101h]
Continue? (y/n)
Transfer of A_2020.02.05.170151 has begun

Move a database¶

A move is a copy followed by the deletion of the transferred data from the source node.

Syntax	`move;`<srcnode>;`<destnodes>;(feeds);(daterange)
Optional	`feeds`, `daterange`
Arming required	Yes
Confirmation required	Yes

Example 3: move data for the last seven days. Move feeds 100 and 101 on node A to nodes B and C, for the last seven days inclusive of today.

q).maint.dbc[`move;`A;`B`C;100 101h;-6 0+.z.d]
Continue? (y/n)
Transfer of A_2020.02.05.170151 has begun

The source node deletes data only after every destination node has merged the transferred data. DBC then asks DBW on the source node to delete the data, and the transfer enters the DELETE stage. At each end of day, DBW deletes up to dbwPruneChunk partitions, starting with the oldest.

Parameter	Description	Default
`dbwPruneChunk`	The maximum number of partitions that DBW deletes in one end of day. Dynamic.	`4`

Look up the state of a move to follow the deletion. The curDel column shows the next partition DBW will delete. When DBW has deleted everything, the stage changes to COMPLETED and the transfer no longer appears in the state output.

Look up the state of a transfer¶

Syntax	`state;`<node>;(verbose)
Optional	`verbose`
Arming required	No
Confirmation required	No

Use state to check whether a node is part of an active transfer before you start a new one. If necessary, you can cancel the active transfer to give priority to a more important one.

A transfer has a state, a stage and a merge state. The state is one of:

State	Meaning
`UNKNOWN`	The transfer failed. DBC tries to recover it automatically. See Recover from a failed transfer.
`ACTIVE`	The transfer is proceeding normally.
`PAUSED`	The transfer is paused.
`CANCELED`	The transfer is cancelled.
`COMPLETED`	The destination node has received all the data. Merging, and for a move deletion, can still be in progress.

The stage tells you what the transfer is doing:

Stage	Meaning
`UNKNOWN`	The stage is not known.
`PIT`	DBC is creating the PIT.
`TRANSFER`	DBC is sending data to the destination nodes, or the destination nodes are still merging it.
`MAP`	DBC is preparing the IDB for merging, including mapping IDs when the source node is in a different cluster.
`DELETE`	A move is waiting for DBW to delete the source data.
`COMPLETED`	The copy or move is finished.

The merge state shows the progress of the merge on the destination node:

Merge state	Meaning
`NOT_APPLICABLE`	The merge hasn't started.
`RECEIVED`	DBW accepted a partition for merging.
`COMPLETED`	DBW merged a partition. The `curMerge` column shows which one.

A transfer with the state COMPLETED and the stage TRANSFER is still merging, and a move with the state COMPLETED and the stage DELETE is waiting for DBW to delete the source data. state doesn't show cancelled transfers or transfers in the COMPLETED stage. If a node has no other transfer, it reports There are no transfers to or from the node.

The summary view shows the source and destination nodes, the operation, the stage and state, the merge state, the PIT start and end dates, the feeds, the number of partitions and bytes, the current partition, the partition currently being merged, and, for a move, the next partition to delete. A blank current partition means the IDB or a null partition.

q).maint.dbc[`state;`A]
srcNode dstNode op   stage    state  mergeState     pitStart   pitEnd     feeds
--------------------------------------------------------------------------------------
A       C       COPY TRANSFER ACTIVE NOT_APPLICABLE 2020.09.05 2020.09.06 100 101 102 103

prtns bytes
---------------
0     27732634

curPrtn curMerge curDel
-----------------------

Set verbose to 1b to add the transfer ID and start time, the current table and file, the total partitions and bytes in the PIT (totPrtns and totBytes), and details of each process in the transfer: state, threads, memory used, heap memory, peak memory, cache size, buffer utilization, and read, send, write and turnaround times.

q).maint.dbc[`state;`A;1b]

Two of the verbose columns help you tune a transfer:

bufUtil is a number between 0 and 1 that shows how much of the buffer the process used, on average, in each monitoring interval. A value close to 1 means the transfer is making good use of dbcBufSize. If the value is low, consider lowering dbcBufSize.
turnTm is the average time between one request for a chunk from a destination node and the next. It includes any throttle delay, and the time the source takes to read and send a chunk.

Throttle a transfer¶

A throttle slows a transfer so that it doesn't affect higher-priority work on the system. You apply a throttle by specifying the time, in milliseconds, that a destination node waits before it asks for its next chunk. For example, 500 milliseconds is a relatively strong throttle.

Syntax	`throttle;`<node>;(ms)
Arming required	Yes
Confirmation required	No

Apply the throttle to the source node after you start the transfer. DBC applies it to every destination node of that transfer.

q).maint.dbc[`throttle;`A;1000]

A throttle stays in effect on the destination nodes until you remove it or restart DBC, and it applies to the next transfer that a destination node receives.

Remove the throttle before the transfer completes, because after that the command has no transfer to act on. To remove a throttle, run the command on the source node without the ms argument. The default is 0, which means no throttling.

q).maint.dbc[`throttle;`A]

Pause and resume a transfer¶

Pause a transfer to give system resources to a higher-priority task. A pause stays in effect until you resume the transfer, and the transfer then continues from where it stopped.

Syntax	`pause;`<node>
Arming required	Yes
Confirmation required	No

Syntax	`resume;`<node>
Arming required	Yes
Confirmation required	No

When a transfer has more than one destination, the node you name decides what is affected:

If you pause the source node, all destination nodes are paused.
If you pause one destination node, only that node is paused. The source node and other destination nodes continue.

You can't pause, resume or cancel a transfer that is already complete.

Cancel a transfer¶

You can cancel a transfer at any time until it completes. DBC removes data that it transferred to the destination but hasn't merged. Data that DBW has merged into the destination database is final and can't be removed or undone. A cancelled transfer no longer appears in the state output.

Syntax	`cancel;`<node>
Arming required	Yes
Confirmation required	Yes

Note

DBC doesn't delete the source PIT when you cancel. See Delete a PIT.

Transfer to an inactive node¶

DBC needs at least one destination node to be connected when you start the transfer. If none is connected, the command fails with No available recipients.

If a destination node is stopped while a transfer is running, or fails to receive data, the transfer stops for that node. When the node starts again, DBC resumes the transfer for it and populates the node with the outstanding data.

Monitor a transfer¶

DBC on each node writes its own log file, so you can follow a transfer from start to finish. To find the logs, see Look up log files. Some of the entries described here are written at DEBUG level, so set the DBC log level accordingly. See Set up log levels in logLevels.yaml.

The source log shows a timestamp for each event in the transfer, including:

The copy or move being invoked
The handshake with each destination node
The PIT being created, including its ID, size and number of partitions
The transfer being paused or resumed
The transfer being complete for each destination node

The destination log shows the following records for each partition:

Record	Description
`Rolling partition`	A separator for each partition, showing the previous partition that DBC transferred (`prev`) and the new partition (`cur`).
`Queueing merge`	The PIT start and end dates.
`Queue merge successful`	DBW accepted the partition for merging. `adt` is the last partition queued for merging, and `zdt` is the final partition in the PIT.
`Merge successful`	The most recent partition that DBW merged (`idt`).

At the end of the transfer, the destination log shows the total duration, bytes and number of partitions.

2020.07.28 12:01:37.276 INFO  [kxsDBC_B] DBC Merge successful, pid=A_2020.07.28.120104 idt=2020.07.20
2020.07.28 12:01:37.403 DEBUG [kxsDBC_B] DBC Rolling partition, pid=A_2020.07.28.120104 prev=2020.07.17 cur=2020.07.16
2020.07.28 12:01:37.416 DEBUG [kxsDBC_B] DBC Queueing merge, pid=A_2020.07.28.120104 prtn=2020.07.17 pitStart=2020.07.13 pitEnd=0Wd
2020.07.28 12:01:37.419 DEBUG [kxsDBC_B] DBC Queue merge successful, pid=A_2020.07.28.120104 adt=2020.07.17 zdt=2020.07.13
2020.07.28 12:01:37.953 INFO  [kxsDBC_B] DBC Merge successful, pid=A_2020.07.28.120104 idt=2020.07.17

Recover from a failed transfer¶

DBC recovers from common failures without your intervention:

If a destination node receives no data for more than 10 seconds, it logs a warning and asks the source node to resume from where the transfer stopped. It repeats this up to 10 times in a row.
If it still receives no data after that, the DBC process stops. When the process restarts, DBC recovers the transfer from the Transfer table and resumes from the point of failure.

If DBW rejects the PIT, for example because of a schema mismatch, DBC logs Queue unsuccessful and the transfer doesn't progress. Check the DBW log on the destination node for the reason.

If a transfer fails repeatedly, check the DBC log files on the source and destination nodes to find out why. After you fix the problem, DBC resumes the transfer automatically.

Transfer a database manually¶

If you transfer a database by other means, for example by removing and inserting drives, you must complete these steps:

Create a PIT of your source node. This step is strongly recommended.
Transfer the data.
Merge the transferred database into the destination node.
Delete the PIT if you no longer need it.

Create a PIT¶

Syntax	`createpit;`<node>;(daterange)
Optional	`daterange`
Arming required	Yes
Confirmation required	No

If you specify a date range, the PIT includes only the HDB partitions in that range, plus the IDB. DBC can't create a PIT while DBW is running an EOX subtask, and the command then reports PIT creation failure.

Delete a PIT¶

To delete a PIT on a source node, specify the node and the PIT ID.

Syntax	`deletepit;`<node>;`<pid>
Arming required	Yes
Confirmation required	Yes

You can't delete the PIT of a transfer that is active, paused or failed (state UNKNOWN). You can delete it after the transfer completes or is cancelled.

DBC deletes the destination PIT when the merge finishes, but it never deletes the source PIT. After a copy or move completes, or you cancel a transfer, delete the source PIT with deletepit. Because a PIT holds hard links to the source files, a move doesn't free the disk space of the moved data until you delete the PIT.

Merge a transferred database¶

DBW merges the database, starting with the most recent partition, which is the IDB. The merge command queues the merge with DBW on the destination node and reports only that it queued the request. Check the DBW and DBC logs on the destination node for the outcome.

Specify the node pair (the originating node, then the node to merge into), the PIT ID, the feeds and a list of dates. The PIT ID is the name of a directory in pitDir on the destination node. The directory must contain the PIT layout (config, idb and hdb), including config/nxDbTbls, which DBW uses to check that the schemas match. Specify the feeds as a list of feed IDs (shorts), for example 100 101h.

Syntax	`merge;`<nodepair>;`<id>;<feeds>;<datelist>
Arming required	Yes
Confirmation required	Yes

The datelist contains:

Element	Description
`datelist[0]`	PIT start date. Mandatory.
`datelist[1]`	PIT end date. Mandatory.
`datelist[2]`	The oldest partition to merge. Optional. The default is `0Nd`, which merges only the IDB.

To merge the whole PIT, set datelist[2] to the date of the oldest partition in the PIT. To merge only part of it, use a later date. For example, if the PIT start and end dates are (.z.d-5;.z.d) and datelist[2] is .z.d-2, DBW merges the IDB and the partitions from .z.d back to .z.d-2. You can run the command again later with an older date to merge more of the PIT, but not with a newer one.

Next steps¶

Using the maintenance console — how to start the console that you run .maint.dbc in.
System parameters systemParams.yaml — the reference for the file that holds the DBC parameters.
Tables and schemas — how to set feedMapFn for a table.
Add a node — add a node before you copy data to it.
Delete historical partitions — remove partitions from a source node outside a move.