5.4.0

Release Date: 2021.11.09

NOTE - This release contains a non-upwardly compatible change. Pipeline YAML changes required.

Refinery Core

• Added ability for tables to be saved as root-splay with intraday persist of none
• UDFs with a "." in the name are now supported
• PDB split into EPDB and IPDB to improve rollover performance !!! NOTE !!!: NUC - requires modification of pipeline YAMLs

Pipeline YAML changes

As part of 5.4.0, a split pdb into ipdb and epdb was introduced. This requires the following change in pipeline yamls.

Before

   pdb:
    write-row-limit: 10000000
    write-freq: 60000

After

    ipdb:
      write-row-limit: 10000000
      write-freq: 60000
    epdb:
      timeout: 0

The below script will automate this transition. Without this change the process manager will reject the pipeline yaml and fail to start.

//Script to update Refinery YAML configs to support new split pdb.
//Will output "*.old" with originals as a backup.
//Edit the updateFile function if no backups are wanted, and/or rm *.yaml.old after running the script



//Set path to pipeline directory
//If within Refinery, can use this environment variable
yamldir:getenv[`REFINERY_CONFIG_DIR],"/system-config/pipelines"



updateFile:{[yamldir;file]
    //Read in yaml file
    yaml:read0 fileName:` sv (yamldir;file);

    //Get the indent level and location of "pdb" tag
    indent:sum" "=first yaml pdbLocation:where yaml like "* pdb:";

    //Define replacement block
    replaceWith:("epdb:";
        "  timeout: 0";
        "ipdb:");
    indentedReplace:(indent#" "),/:replaceWith;

    //Ammend Yaml
    newYaml:((first pdbLocation)#yaml),indentedReplace,(1+first pdbLocation)_yaml;

    //Write .old for backup
    (`$string[fileName],".old") 0: yaml;

    //Write new
    fileName 0: newYaml;
    }

yamls:k where (k:key[yamldir]) like "*.yaml"
updateFile[yamldir] each yamls