ML Graph¶
Graph .ml.createGraph Generate an empty graph
Configuration .ml.addCfg Add a configuration node to a graph .ml.delCfg Delete a configuration node from a graph .ml.updCfg Update the content of a configuration node
Nodes .ml.addNode Add a functional node to a graph .ml.delNode Delete a functional node from a graph .ml.updNode Update the content of a functional node
Edges .ml.connectEdge Connect the output of one node to the input of another .ml.disconnectEdge Disconnect an edge connected to a specified node
As outlined here the graph structure described below follows the basic structure of a directed mathematical graph consisting of nodes which contain the core functionality of a pipeline and edges which describe the connections between nodes. The image below shows a basic representation of a graph containing multiple nodes connected together by relevant edges within this structure.
Prior to the description of the functionality which allows a user to generate, update and remove components of a graph, a number of notes on the technical aspects on the structure of these components must be outlined.
Functional node¶
A functional node is defined in this library as a dictionary.
key | value |
---|---|
inputs | The expected input datatype: a char atom, for a node receiving one input; or a dictionary mapping named inputs to their datatypes. |
outputs | The expected output datatype: a char atom, for a node putting out one item; or a dictionary mapping named outputs to their datatypes. |
function | A function containing the logic to be executed by the node. Its rank must be the count of the inputs entry, and its result either a single value (for a node with a single output) or a dictionary with keys from the outputs dictionary. |
The following image shows a graphical representation of such a node followed by a number of examples showing the basic structure of these nodes.
// Node taking one input of float type and outputting one float type
q)nodeInputs:"f"
q)nodeOutputs:"f"
q)nodeFunction:{x wsum x}
q)node:`inputs`outputs`function!(nodeInputs;nodeOutputs;nodeFunction)
// Node taking multiple inputs and returning multiple values
q)nodeInputs:`xIn`yIn`zIn!"sF!"
q)nodeOutputs:`xOut`yOut`zOut!"sF!"
q)nodeFunction:{[x;y;z]`xOut`yOut`zOut!(x;y;z)}
q)node:`inputs`outputs`function!(nodeInputs;nodeOutputs;nodeFunction)
Configuration node¶
Configuration nodes in this library are a subset of the functional nodes described above. A configuration node takes as input a dictionary provided by a user and outputs this unmodified adding the named node to the overall graph.
Edges¶
An edge is a connection between the output of one functional or configuration node and the input of another node. In order to ensure that a graph is valid all input nodes must be connected to the output from another node within the graph. Output nodes do not need to be connected to anything for a graph to be valid. In the case an output node is not connected, the return of a fully executed graph will store the output data for retrieval by the user.
Ensure the type allocated to the input coincides with the type allocated to the output
This is currently not checked by pipeline execution.
.ml.addCfg
¶
Add a configuration node to a graph
.ml.addCfg[graph;nodeId;config]
Where
graph
is a graph originally generated using.ml.createGraph
.nodeId
is a symbol unique to the graph denoting the name to be associated with the configuration node.config
is a dictionary containing any configuration information to be supplied to other nodes in the graph.
returns a graph with the the new configuration added to the graph structure
// Add a configuration node to the graph
q)show graph:.ml.addCfg[graph;`config;`a`b!1 2]
nodes| (+(,`nodeId)!,``config)!+``function`inputs`outputs!((::;::);(::;![,`ou..
edges| (+`destNode`destName!(,`;,`))!+`sourceNode`sourceName`valid!(,`;,`;,0b)
// Display the nodes
q)graph.nodes
nodeId| function inputs outputs
------| ---------------------------------------------------------------
| :: :: :: ::
config| :: ![,`output]@[enlist]@[;`a`b!1 2] (`symbol$())!"" (,`out..
q)graph.edges
destNode destName| sourceNode sourceName valid
-----------------| ---------------------------
| 0
// Attempt to add a node with non unique name
show graph:.ml.addCfg[graph;`config;`a`b!1 2]
'invalid nodeId
.ml.addNode
¶
Add a functional node to a graph
.ml.addNode[graph;nodeId;config]
Where
graph
is a graph originally generated using.ml.createGraph
nodeId
is a symbol unique to the graph denoting the name to be associated with the functional nodeconfig
is a functional node
returns a graph with the the new node added to the graph structure.
// Define a node with one input and one output
q)newInput:"!"
q)newOutput:"F"
q)newFunction:{[dict]dict`floatVector}
// Generate relevant configuration and add to an empty graph
q)newNode:`inputs`outputs`function!(newInput;newOutput;newFunction)
q)show graph:.ml.addNode[graph;`newNode;newNode]
nodes| (+(,`nodeId)!,``newNode)!+``function`inputs`outputs!((::;::);(::..
edges| (+`destNode`destName!(``newNode;``input))!+`sourceNode`sourceNam..
q)graph.nodes
nodeId | function inputs ..
-------| --------------------------------------------------------------..
| :: :: :: ..
newNode| :: ![,`output]@[enlist]{[dict]dict`floatVector} (,`input)!,"!"..
q)graph.edges
destNode destName| sourceNode sourceName valid
-----------------| ----------------------------
| 0
newNode input | 0
// Define a node with multiple inputs and outputs
q)multiInput:`input1`input2!"!+"
q)multiOutput:`output1`output2!"!+"
q)multiFunction:{[input1;input2]`output1`output2!(input1;input2)}
// Generate the node with multiple inputs and outputs
q)multiNode:`inputs`outputs`function!(multiInput;multiOutput;multiFunct..
q)show graph:.ml.addNode[graph;`multiNode;multiNode]
nodes| (+(,`nodeId)!,``newNode`multiNode)!+``function`inputs`outputs!((..
edges| (+`destNode`destName!(``newNode`multiNode`multiNode;``input`inpu..
q)graph.nodes
nodeId | function inputs ..
---------| ------------------------------------------------------------..
| :: :: :: ..
newNode | :: ![,`output]@[enlist]{[dict]dict`floatVector} (,`inpu..
multiNode| :: {[input1;input2]`output1`output2!(input1;input2)} `input1..
q)graph.edges
destNode destName| sourceNode sourceName valid
-------------------| -----------------------------
| 0
newNode input | 0
multiNode input1 | 0
multiNode input2 | 0
.ml.connectEdge
¶
Connect the output of one node to the input to another
.ml.connectEdge[graph;sourceNode;sourceName;destNode;destName]
Where
graph
is a graph originally generated using.ml.createGraph
sourceNode
is a symbol denoting the name of a node in the graph which contains the relevant outputsourceName
is a symbol denoting the name of the output to be connected to an associated input nodedestNode
is a symbol denoting the name of a node in the graph which contains the relevant input to be connected todestName
is a symbol denoting the name of the input which is connected to the output defined bysourceNode
andsourceName
returns the graph with the relevant connection made between the inputs and outputs of two nodes.
// Graph edges prior to connection
q)graph.edges
destNode destName | sourceNode sourceName valid
-------------------| ----------------------------
| 0
sourceNode input | 0
destNode destName| 0
// Connect a suitable output from one node to suitable input of anoth..
q)graph:.ml.connectEdge[graph;`sourceNode;`sourceName;`destNode;`dest..
// Display edges following appropriate connection
q)graph.edges
destNode destName | sourceNode sourceName valid
-------------------| ---------------------------
| 0
sourceNode input | 0
destNode destName| sourceNode sourceName 1
.ml.createGraph
¶
Generate an empty graph
.ml.createGraph[]
returns a dictionary containing the structure required for the generation of a connected graph. This includes a key for information on the nodes present within the graph and edges outlining how the nodes within the graph are connected.
// Generate an empty graph
q)show graph:.ml.createGraph[]
nodes| (+(,`nodeId)!,,`)!+``function`inputs`outputs!(,::;,::;,::;,::)
edges| (+`destNode`destName!(,`;,`))!+`sourceNode`sourceName`valid!(..
// Retrieve the nodes schema
q)graph.nodes
nodeId| function inputs outputs
------| --------------------------
| :: :: :: ::
// Retrieve the edges schema
q)graph.edges
destNode destName| sourceNode sourceName valid
-----------------| -----------------------------
| 0
.ml.delCfg
¶
Delete a named configuration node
.ml.delCfg[graph;nodeId]
graph
is a graph originally generated using.ml.createGraph
nodeId
is a symbol denoting the name of a configuration node to be deleted
returns the graph with the named node removed.
// Display the graph nodes prior to deletion of a configuration node
q)show graph.nodes
nodeId | function ..
-------------| -------------------------------------------------------..
| :: :: ..
configuration| :: @[;`xdata`ydata!(+`x`x1!(0.06165008 0.285799 0.66847..
// Delete the configuration node
q)graph:.ml.delCfg[graph;`configuration]
// Display the graph nodes
q)show graph.nodes
nodeId| function inputs outputs
------| --------------------------
| :: :: :: ::
.ml.delNode
¶
Delete a named function node
.ml.delNode[graph;nodeId]
Where
graph
is a graph originally generated using.ml.createGraph
nodeId
is a symbol denoting the name of a functional node to be deleted
returns the graph with the named functional node removed.
// Display the graph nodes prior to deletion of a configuration node
q)show graph.nodes
nodeId | function inputs outputs
-------| -------------------------------------------
| :: :: :: ::
newNode| :: {x} (,`input)!,"!" (,`sourceName)!,"!"
// Delete the configuration node
q)graph:.ml.delNode[graph;`newNode]
// Display the graph nodes
q)show graph.nodes
nodeId| function inputs outputs
------| --------------------------
| :: :: :: ::
.ml.disconnectEdge
¶
Disconnect an edge from the input of a node
.ml.disconnectEdge[graph;destNode;destName]
Where
graph
is a graph originally generated using.ml.createGraph
destNode
is the name as a symbol of the node containing the edge to be deleteddestName
is the name as a symbol of the edge associated with a specific input to be disconnected
returns the graph with the edge connected to the destination input removed from the graph.
// Display the graph prior to disconnection of an edge
q)graph.edges
destNode destName | sourceNode sourceName valid
-------------------| ---------------------------
| 0
sourceNode input | 0
destNode destName| sourceNode sourceName 1
// Disconnect an edge
q)graph:.ml.disconnectEdge[graph;`destNode;`destName]
// Display the graph edges post disconnection
q)graph.edges
destNode destName | sourceNode sourceName valid
-------------------| ---------------------
| 0
sourceNode input | 0
destNode destName| 0
.ml.updCfg
¶
Update the contents of a configuration node
.ml.updCfg[graph;nodeId;config]
Where
graph
is a graph originally generated using.ml.createGraph
nodeId
is a symbol denoting the name of a configuration node to be updatedconfig
is a dictionary containing any configuration information to be supplied to other nodes in the graph
returns the graph with the named configuration node contents overwritten.
// Display the node prior to overwriting the configuration contents
q)graph.nodes
nodeId | function ..
-------------| -------------------------------------------------------..
| :: :: :: ..
configuration| :: ![,`output]@[enlist]@[;`xdata`ydata!(,0.3138309;`tes..
// Update the configuration node content
q)graph:.ml.updCfg[graph;`configuration;enlist[`xdata]!enlist 2?1f]
q)graph.nodes
nodeId | function ..
-------------| --------------------------------------------------------..
| :: :: ..
configuration| (,`xdata)!,:: (,`xdata)!,![,`output]@[enlist](,`xdata)!,..
.ml.updNode
¶
Update the contents of a functional node
.ml.updNode[graph;nodeId;config]
Where
graph
is a graph originally generated using.ml.createGraph
nodeId
is a symbol denoting the name of a functional node to be updatedconfig
is a functional node
returns the graph with the named functional node contents overwritten.
// Display the node prior to overwriting node content
q)graph.nodes
nodeId | function inputs outputs
----------| ------------------------------------------------------------
| :: :: :: ::
sourceNode| :: {x} (,`input)!,"!" (,`sourceName)!..
destNode | :: ![,`output]@[enlist]{x} (,`destName)!,"!" (,`output)!,"F"
// Generate node content to overwrite the function for sourceNode
q)updSrcInput:"!"
q)updSrcOutput:enlist[`sourceName]!enlist "!"
q)updSrcFunction:{x+1}
q)updSrc:`inputs`outputs`function!(updSrcInput;updSrcOutput;updSrcFunction)
// Overwrite the functional node
q)graph:.ml.updNode[graph;`sourceNode;updSrc]
q)show graph.nodes
nodeId | function inputs outputs
----------| ----------------------------------------------------------
| :: :: :: ::
sourceNode| :: {x+1} (,`input)!,"!" (,`sourceName)!..
destNode | :: ![,`output]@[enlist]{x} (,`destName)!,"!" (,`output)!,"F"