Skip to content

About schema changes

This page explains how KX Sensors handles schema changes during partial upgrades, including automatic old-to-new and new-to-old data conversions.

When performing partial upgrades, the schema of your in-memory database for processes not targeted by the release may not match the schema of your system or on-disk database for a period. When this happens, KX Sensors performs automatic old-to-new or new-to-old conversions so that incoming and outgoing data always matches the expected schema, and no data in transit is lost.

Most KXS schema changes are supported by dynamic upgrades. Exceptions include:

  • Modification to a table category or the primary keys of a keyed table.
  • Deleting a shard from a sharded table.

Modifications that are possible through an offline upgrade but not using a dynamic upgrade include:

  • Modification of a database column where the type has not changed.
  • A database column type change that cannot be handled by the automatic conversion alone. The following is a non-exhaustive list of such changes:
    • A change from string to any other type, including symbol.
    • A change from symbol to any other type, excluding string.
    • A change from/to guid to/from any other type.
    • A change from any type to char.
    • A change from/to a complex type such as a nested vector/list, or a dict.

It is expected in the above cases that a custom upgrade script using .conv.applyColFn to return the new value of the column in its appropriate data type is used. Consult Perform an offline upgrade for further information.

Test your schema changes

Database upgrades should be thoroughly tested before being deployed in a production setting. To ensure database integrity when testing your upgrades, it is strongly recommended that you verify the schema of the data being ingested while testing.

The following dynamic flags in systemParams.yaml should be set to true:

Parameter Description
enforcePositivePK If set to true, KXS enforces integer type primary keys to have a positive value when publishing.
enforcePubSchema If set to true, KXS automatically verifies the schema of your data as it's published by SDL and replayed by DBW and throws an error if the message being published or table data being replayed does not match the expected schema defined in the relevant schema.yaml.

Note

These validations consume system resources and should be disabled in a production environment.

About on-disk conversions

An on-disk conversion refers to updating the schema and data of your HDB and IDB; that is, old data that was received before the upgrade. Depending on the volume of data that you capture every day and the nature of your changes, this process may take up to a few hours or even longer in certain cases.

During this time, any queries of your historical data retrieve data according to your old schema. If you added new columns in your upgrade with default values, the new columns are not available until the release is complete. Once the release is complete, any queries of your historical data retrieve data according to your new schema and the default values in the new columns become available.

If you target your schema change to a particular node, only the database in that node is upgraded. Databases in non-targeted nodes are not upgraded.

About table and column renames

Schema upgrades involving the rename of columns or the table name itself can be performed using the origNames property within the corresponding schema.yaml.

In the below example, the table Test is renamed to TestRenamed and the column col is renamed to renamedCol:

TestRenamed:
  m-meta: schema.yaml
  origNames: Test
  - name: renamedCol
    origNames: col
    type: integer

Next steps