New features in the 2.4 release of kdb+
Detailed change list / release notes
Commercially licensed users may obtain the detailed change list / release notes from http://kxdownloads.com
This release builds on the existing 2.3. The non upward compatible changes (NUC) are the change to .z.ph (which only impacts people who've written their own http handler), the additional .z.pc close event when closing stdin, and the removal of the hopen shortcut of just hopen`:port. All other code should run without change.
Although 2.3 used multithreading to farm out queries to multiple partitions using the number of threads set via the -s parameter the input queue was still single threaded. In 2.4 it is possible to multithread that queue as well. Starting a kdb+ task with a positive -p parameter like before preserves the single threaded behaviour.
q .. -p 5001 / single threaded input queue as before
if, however a negative port number is given the input queue is multithreaded
q .. -p -5001 / multithreaded input queue
The number of slaves set with -s are still dedicated to running queries over the partitions in parallel. However the number of threads used to handle the input queue is not user settable. For now each query gets an own thread - later it is probably going to change to a pool of threads, one per cpu.
The shortcut syntax of hopen`:port has been removed, with 2.4 the host section must also be provided. The hostname can be elided, but the colon must be there.
hopen`:localhost:5001 / ok hopen`::5001 /ok hopen`:5001 / NOT ok, it will have opened a file handle to file "5001"!
The alternate shortcut of supplying solely the (integer) portnumber is still valid.
hopen 5001 / open port 5001 on local machine
z.pw adds an extra callback when a connection is being opened to a kdb+ session.
Previously userid and password were checked if a userid:password file had been specified with the -u or -U parameter. The checking happened outside of the user's session in the q executable. Only if that check was passed did the connect get created and a handle passed into the "user space" as a parameter to .z.po.
Now after the -u/-U check has been done (if specified on he command line) the userid and password are passed to .z.pw allowing a function to be run to perform custom validation - for example to check aganst an LDAP server. If .z.pw returns a 1b the login can proceed and the next stop will be .z.po, if it returns a 0b the user attempting to open the connection will get an access error.
.z.pi has been extended to handle console input as well as remote client input (like with qcon). This allows validating console input, or replacing the default display (.Q.s value x) with a home-knitted version.
to get the old (2.3 and earlier) display reset .z.pi
to return to the default display execute \x .z.pi
in addition to the close events previously handled .z.pc is now also called when stdin is "closed" by redirecting it to /dev/null. In this case the handle provided to .z.pc will be 0 (the current console)
By default callbacks like .z.po are not defined in the session, which makes it awkward to revert to the default behaviour after modifying them - for example when debugging or tracing. \x allows deleting their definitions to force default behaviour.
.Q .q visibility
In 2.3 the display of the definitions of functions in .q and .Q was supressed. People complained .. so the display is back.
before 2.4 the timer event would next happen n milliseconds after completion of the execution of the timer's code. In 2.4 the event fires every n milliseconds.
Shorthand for `time$.z.z, similarly .z.d <-> `date$.z.z, .z.T <-> `time$.z.Z, .z.D <-> `date$.z.Z
1066 and all that
The valid year range has been expanded to 1710-2290
\b, \B and .z.b
Extended support for tracking dependencies.
\b lists all dependencies (views)
\B lists all dependencies that have been invalidated
.z.b lists dependencies and the values they depend on
q)a 11 q)b 12 13 14 q)a:99 q)b 100 101 102 q)\b ,`b q)\B `symbol$() q)a:22 q)\B ,`b q)b 23 24 25 q)\B `symbol$() q).z.b a| b q)
-11! (used to replay logfiles) has been made more robust when dealing with corrupt files.
Previously it would have crashed and exited on encountering corrupt records, now it will truncate the file to remove the invalid records and only process valid elements.
Rather than the individual cells being executed directly .z.ps is called on each allowing easier customisation of log loaders
Run a kdb+ server under inetd, see kdb+ under inetd/xinetd
Block client access to a kdb+ database. The ability to write directly to disk allows an incorrectly written query to create havoc with a database. If the kdb+ server is started with the -b flag all client writes are disabled.
note: this functionality was incorrectly assigned to the -z startup parameter for a while during the 2.4 test cycle
startup kdb+ in quiet mode - don't display the startup banner and license information.
makes it easier to process the output from a q session if redirecting it as part of a larger application
\1 filename & \2 filename
\1 and \2 allow redirecting stdin and stdout to files from within the q session. the filename and any intermediate directories are created if needed
clients \a \f \w \b
When the -u/-U client restrictions are in place the list of valid commands for a client has been reduced to \a \f \w and \b
maintain group attribute for append and update
The `g# attribute is maintained in places where in 2.3 it would have been cleared.
n:100000 / this has .5% change of sym x:+(n?100*n;n?`3;n?100) /id sym size t:([o:`u#!0]s:`g#0#`;z:0) \t .[`t;();,;]'x
Before 2.4 the data being "decorated" with the `s# flag was not validated - so it was possible to flag as sorted a list that actually wasn't - causing problems when using primitives that depend on order like bin. Now an error will be signalled if data is incorrectly flagged.
q)`s#1 2 3 `s#1 2 3 q)`s#1 2 3 3 `s#1 2 3 3 q)`s#1 2 3 3 2 'fail q)
In the case of tables it just checks the first column.
month, year and int partition columns all virtual
Before 2.4 the only the date column in a database partitioned by date was virtual. With 2.4 the year, month or int columns in databases partitioned by year, month or int respectively are treated the same way
With 2.4 it is possible to have the same named partition on multiple partitions (as defined in par.txt) allowing for example splitting symbols by A-M on drive0, N-Z on drive1. Aggregations are handled correctly.
skip leading #!
When loading a script file that starts with #!, the first line is skipped.
On OSs that support it the values passed to \c (console width and height) are taken from the system variables $LINES and $COLUMNS
no limit on stdin
before 2.4 there was a limit of 999 characters on stdin/console - with 2.4 there is no limit
mlim and glim increased to 1024
the maximum number of simultaneously mapped nested columns (mlim) and the maximum number of active `g# indices (glim) have been increased to 1024
mcount and friends null-aware
Windowed functions like mcount, mavg have been made consistently aware of null values
q)3 mcount 1 1 1 1 1 1 1 2 3 3 3 3 q)3 mcount 1 1 0N 1 1 1 1 2 2 2 2 3
.z.ph and .z.pp changed
Previously kdb+ stripped the header information from incoming http requests before passing it on to the user via .z.ph or .z.pp.
Unfortunately that meant useful information like the client's browser or preferred languages was being discarded.
With 2.4 the header information is passed back as a dictionary in addition to the body text as before.
This is a change in behaviour, but will only affect those who have customised .z.ph or .z.pp directly. The previous value is now the first element of a 2 element list, the new header dictionary is the second element.
General speedups to the code to handle scalar values
abs is primitive
\r old new
unix mv (rename)
With 2.4, the upsert function appends splayed tables on disk. Note that keyed tables cannot be splayed, so splayed upsert is the same as insert.
q)`:x/ upsert (a:til 3) `:x/ q)count get`:x 3 q)`:x/ upsert (a:til 3) `:x/ q)count get`:x 6