Skip to content

KX Insights Core 2.0.8

Release Date

2021-12-01

Object Store

[FIX] Creation of an inventory file will help speed up hdb load times.

gsutil ls -lr gs://kxinsights-marketplace-data/db///* | awk '{printf "{ \"Key\": \"%s\" , \"Size\": %s }\n", $3, $1}' | head -n -1 | jq -s '.' | sed 's/gs:\/\/kxinsights-marketplace-data\///g' | gzip > all.json.gz

aws --output json s3api list-objects --bucket kxinsights-marketplace-data --prefix 'db' --query 'Contents[].{Key: Key, Size: Size}' | gzip > all.json.gz

az storage blob list --account-name kxinsightsmarketplace --container-name data | jq '.[] | {Key: .name , Size: .properties.contentLength }' | jq -s '.' | gzip > all.json.gz

Upload to the bucket itself via

=== GCP

gsutil cp all.json.gz gs://kxinsights-marketplace-data/_inventory/all.json.gz

aws s3 cp all.json.gz s3://kxinsights-marketplace-data/_inventory/all.json.gz

az storage blob upload --account-name kxinsightsmarketplace \ --container-name data --name _inventory/all.json.gz --file all.json.gz

User can control which file is used as inventory via env var

export KX_OBJSTR_INVENTORY_FILE=_inventory/all.json.gz

The reading of the inventory file bypasses the cache, and to avoid cache invalidation issues, is not readable explicitly.

The file must be gzipped json, as an array of {Key:string,Size:int} objects.

[FIX] Renamed environment variables to be vendor neutral(S3->OBJSTR).

KX_S3_CACHE_PATH->KX_OBJSTR_CACHE_PATH
KX_TRACE_S3->KX_TRACE_OBJSTR

[FIX] A memory corruption issue has been fixed in the kxreaper caching process. When the same cached file is accessed twice or more in a row, and then eventually when that file is squeezed out of the cache due to size constraints on the dir, it will double free/crash/abort. This scenario is more likely to be hit when the cache size is small(e.g. 10MB) and can only hold a few files.

REST Client

[FIX] Azure metadata registration can return non json messages if there are no service accounts. This would cause an error as shown below

q).j.k "getting assigned identities for pod default/insights-discovery-proxy-79f67f865-c9zn5"
'illegal char g at 0
  [0]  .j.k "getting assigned identities for pod default/insights-discovery-proxy-79f67f865-c9zn5"

The library has been updated to handle error conditions like this and log the errors as debug messages.

Known Issues

  • KX_S3_CACHE_SIZE is still active. This will also be renamed in an upcoming release.