Skip to content

Segmented databases

You can distribute partitioned tables across multiple storage devices to:

  • Give them more space
  • Support parallelization

The root of a segmented database contains only the sym list and a file par.txt, which unifies the partitions of a database, presenting them as a single database for querying.

par.txt

File par.txt defines the top-level partitioning of the database into directories. Each row of par.txt is a directory path. Each such directory is itself partitioned in the usual way, typically by date. The directories must not be empty.

In the following layout, par.txt and sym sit on the first device, and each remaining device holds whole date partitions, each with the same table directories:

DISK 0             DISK 1                     DISK 2
db                 db                        db
├── par.txt        ├── 2020.10.03            ├── 2020.10.04
└── sym            │   ├── quotes            │   ├── quotes
                   │   │   ├── price         │   │   ├── price
                   │   │   ├── sym           │   │   ├── sym
                   │   │   └── time          │   │   └── time
                   │   └── trades            │   └── trades
                   │       ├── price         │       ├── price
                   │       ├── sym           │       ├── sym
                   │       ├── time          │       ├── time
                   │       └── vol           │       └── vol
                   ├── 2020.10.05            ├── 2020.10.06
                   │   ├── quotes            │   ├── quotes
                   ..                        ..

The matching par.txt:

/1/db
/2/db

Do not end the paths with a directory delimiter

Write /1/db, not /1/db/. Depending on the file system, the trailing slash can leave q unable to resolve the segment.

Working with segmented databases

You create and modify segmented databases in exactly the same way as partitioned tables. The only addition is the par.txt file, which is a plain text file you can maintain by hand.

Security-related options such as reval or the command-line option -u 1 restrict access to within the current directory. That restriction can prevent users from reaching a segmented database when par.txt references partitions outside the current directory.

To provide access, use symbolic links (symlinks). Create one per segment:

ln -s /db1 db1$
ln -s /db2 db2$

The database root then holds the links alongside par.txt and sym:

total 16
lrwxr-xr-x  1 user  kx   6 18 Sep 12:30 db1$ -> /db1
lrwxr-xr-x  1 user  kx   6 18 Sep 12:30 db2$ -> /db2
-rw-r--r--  1 user  kx  10 18 Sep 12:30 par.txt
-rw-r--r--  1 user  kx  48 18 Sep 12:30 sym

and par.txt names the links rather than the absolute paths:

db1$
db2$

A trailing $ in the directory name makes KDB-X skip the symbolic links themselves and use the directories they point to.

Next steps