Interpreting DBM Metrics

There are a number of kxi_sm_dbm_ metrics available and each report a single fact - a status code, a counter, a partition count. By combining, you can distinguish between the different states the DB Migration (DBM process) can be in, which helps to explain why a migration was paused, is taking longer than expected or alert that it has a backlog it cannot catch up with.

Scenario	Metric signature	Interpretation
Idle	kxi_sm_dbm_status=0, kxi_sm_dbm_pending_partitions=0	No migration is currently required or running. kxi_sm_dbm_duration_seconds reflects the last completed migration.
Migration in progress	kxi_sm_dbm_status=1, kxi_sm_dbm_pending_partitions decreasing across successive polling intervals	A migration is actively moving partitions and making progress. kxi_sm_dbm_duration_seconds won't update until this run completes.
Paused by EOD (expected)	kxi_sm_dbm_status=2, kxi_sm_dbm_discontinued_count increased since the previous polling interval, kxi_sm_dbm_restarts unchanged across the same interval	DBM suspends the migration for the duration of the EOD writedown (or when a database schema conversion needs to start). Self-resolving - the migration resumes once EOD completes.
Paused by a DBM crash (unexpected)	kxi_sm_dbm_status=2, kxi_sm_dbm_restarts increased since the previous polling interval, kxi_sm_dbm_discontinued_count unchanged across the same interval	The DBM process died while a migration was still in progress and then restarted; SM detects that the migration was never cleanly closed out and marks it as paused. Investigate the DBM process logs - this warrants alerting, unlike the EOD case above.
Stalled migration	kxi_sm_dbm_status=1 for longer than expected, kxi_sm_dbm_pending_partitions flat across polling intervals	The migration reports as active but isn't progressing, and kxi_sm_dbm_restarts is not incrementing, so it isn't crash-looping either. This can be alerted on as a likely hang.
Steady backlog (not falling behind)	kxi_sm_dbm_discontinued_count increments on (almost) every EOD cycle, but the per-EOD-cycle peak of kxi_sm_dbm_pending_partitions holds roughly flat across EOD cycles rather than trending up; kxi_sm_dbm_duration_seconds is similarly flat/stale from one EOD cycle to the next	DBM never fully clears the queue before EOD cuts it off, but it's migrating roughly as many partitions each EOD cycle as accumulate in the meantime, so the backlog is holding steady rather than growing. Lower urgency than the row below; worth tracking, but not an active incident on its own.
Large or growing backlog	kxi_sm_dbm_pending_partitions starts each run at a large value (sampled just after kxi_sm_dbm_status moves to 1), and that starting value grows from one EOD cycle to the next; kxi_sm_dbm_discontinued_count increments on (almost) every cycle instead of occasionally	DBM is structurally falling behind - more partitions need relocating than it can get through in the window between EOD completion and the next EOD start. See Detecting a growing migration backlog for how to confirm this from the metrics.
Migration slowing down (early warning)	kxi_sm_dbm_status completes back to 0 each cycle (not 2), but kxi_sm_dbm_duration_seconds trends upward across successive completed runs	Migrations are still finishing inside the available window, but each one takes measurably longer than the last. This is the leading indicator for the backlog scenario above - catching it here, while runs still complete, gives time to act before DBM starts missing the EOD window and kxi_sm_dbm_discontinued_count starts climbing.

Because kxi_sm_dbm_status=2 covers both the EOD-paused and crash-paused cases, the way to tell them apart is to check which counter increased over the polling interval where the status changed: an increase in kxi_sm_dbm_discontinued_count means a planned pause, while an increase in kxi_sm_dbm_restarts means the process crashed.

Detecting a growing migration backlog

kxi_sm_dbm_pending_partitions is reset to 0 the instant a migration run is discontinued or completes, so a backlog does not show up as the gauge "getting stuck" at a high number - instead it shows up as the value the gauge jumps to every time a new run starts. Since kxi_sm_dbm_status only turns 1 for the (usually short) window between EOD completion and the next EOD start, that starting value is easy to miss on a coarse scrape interval; alert on the peak of kxi_sm_dbm_pending_partitions per EOD cycle rather than its instantaneous value.

A growing backlog looks like:

• The per-cycle peak of kxi_sm_dbm_pending_partitions trending upward over successive EOD cycles, rather than settling back to a small, steady figure.
• kxi_sm_dbm_discontinued_count incrementing on nearly every cycle (the migration is cut off by EOD before it can finish) rather than occasionally.
• kxi_sm_dbm_duration_seconds staying flat/stale - a full, uninterrupted migration run is rarely, if ever, completed, so this gauge only reflects the last time the migration did finish, which may be far in the past.
• Correlated growth in kxi_sm_hdb_partitions and kxi_sm_hdb_size, indicating the underlying partition count/volume is outpacing DBM's throughput rather than DBM being unhealthy.

If this pattern holds, a solution may be increasing the number of cores used by the DBM process to improve performance. Also check the underlying storage (local disk or object storage) throughput.