Skip to content

Storage Manager initial import

How to use Storage Manager with an existing kdb+ database

Storage Manager (SM) guarantees atomicity during write-down; and at the same time ensures that a database is mountable by vanilla kdb+ process at any point in time. To achieve this, SM uses symbolic links to represent a standard kdb+ segmented database, while keeping the backing data in a proprietary structure. Data in object storage is excluded from this transformation, and kept in standard kdb+ format.

Thus, to work with an existing database, SM first needs to adjust the database to its own format.

Import scenarios

Two scenarios are supported.

Simple partitioned database

The database is in the standard format for a partitioned (non-segmented) database. Put the database under the data sub-directory of the directory pointed to by baseURI of the HDB-based mount, that is, the mount whose type=local, and partition=date. The database is converted in-place to SM format.

It is possible to have some partitions located in object storage: set the store property of the last HDB-based tier to point to it (e.g. s3://historical-data/db), and SM will add an entry for it in the generated par.txt.

Partitions only in object storage

This scenario resembles the Simple partitioned database scenario, except that the location pointed to by the first HDB-based tier contains only the sym file (if applicable): all the partitions exist in object storage. SM will add an entry for it in the generated par.txt.

The following conditions must be met for all the above scenarios:

  • tables match the schema specified in the assembly configuration
  • partition values are date
  • no overlap between partition values (across tiers)
  • a backup copy of the data exists

Backup policy

Note that the backup is not enforced, since it is likely originating in a different volume before being copied to the SM volume. It is up to the user to ensure that this data is backed up somewhere prior to starting SM.

Future support

In the future, SM will support importing a fully segmented database, whose segments map one-to-one with tiers specified in the assembly configuration.

Error recovery

SM has a recovery mechanism: if it gets interrupted during a long conversion, on restart it continues where it left off. If an error occurs during conversion, SM rolls back the database to its original state.