Kurl Examples

This page provides practical use cases across major cloud providers and APIs for the Kurl module in KDB-X.

Example 1 - Request data from Google BigQuery

Pre-requisite

Obtain credentials for your GCP account.

gcloud init
export GCP_TOKEN=$(gcloud auth print-access-token)

Request dataset

Use these details in your kdb-x session:

.kurl:use`kx.kurl
.kurl.register(`oauth2; "*.googleapis.com"; "";enlist[`access_token]!enlist getenv`GCP_TOKEN)
.kurl.sync("https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-public-data/datasets";`GET; ::)
200i
"{\n  \"kind\": \"bigquery#datasetList\",\n  \"etag\": \"IyFqxYKvltLbvNdsx9qlIQ==\",\n  \"nextPageT..

Example 2 - Push data to Azure

Pre-requisite

This example will not work against KX public buckets. Our public buckets are read only.

For a private bucket, use a Storage Account with the IAM role Storage Blob Data Owner

Upload non KDB-X data to Azure Storage

This example shows how to create non KDB-X files in cloud storage.

In the code snippet below replace <account> with the lowercase name of your storage account.

.kurl:use`kx.kurl
// Create a container
headers:enlist ["x-ms-version"]!enlist "2017-11-09";
opts:enlist[`headers]!enlist headers;
resp:.kurl.sync ("https://<account>.blob.core.windows.net/mycontainer?restype=container"; `PUT; opts);

// Now, add a text file to the bucket
if[201 <> first resp; 'last resp];
opts[`headers]:("x-ms-version";"x-ms-blob-type";"Content-Type")!("2017-11-09";"BlockBlob";"text/plain");
opts[`body]:"hello world";
resp:.kurl.sync ("https://<account>.blob.core.windows.net/mycontainer/hello.txt"; `PUT; opts);
if[201 <> first resp; 'last resp];

Uploading large files

For larger files, you may not be able to upload in one single request.

You may upload a file in sequential blocks by using the (Put Block and Append Block APIs)[https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs]

You may also upload a file in parallel using the same Put Block API with a BlockBlob.

azcopy for databases

Use azcopy to upload KDB-X databases. These examples demonstrate how to upload singular files easily.

Append Block (single threaded)Put Block (multi threaded)

filePath:`:/tmp/example.file

// Set fileSize and block sizes, produce ranges
fileSize:hcount filepath
blockSize:"j"$4e6 // 4Mb (Azure suppots 4Mib, 4Mb close enough)
ranges:"j"$fileSize&reverse each 1_,':[blockSize*til 1+ceiling fileSize%blockSize]

// First, create the empty blob
headers:("x-ms-version";"Content-Type";"x-ms-blob-type")!("2019-02-02";"text/plain";"AppendBlob");
opts:`body`headers!("";headers);
resp:.kurl.sync ("https://<account>.blob.core.windows.net/mycontainer/myblob";`PUT; opts);
if[201i <> first resp; 'last resp];

// Read into a file at an offset, given a length, and upload that block.
uploadBlock:{[filePath;range]
    opts:`body`headers!(read1(filePath;range 0; range[1] - range 0);headers);
    .kurl.sync ("https://<account>.blob.core.windows.net/mycontainer/myblob?comp=appendblock";`PUT; opts);
    if[201i <> first resp;'last resp];
    }

// Upload each block one after another
uploadBlock[filePath;] each ranges;

filePath:`:/tmp/example.file

// Set fileSize and block sizes, produce ranges
fileSize:hcount filePath
blockSize:"j"$4e6 // 4Mb (Azure suppots 4Mib, 4Mb close enough)
ranges:"j"$fileSize&reverse each 1_,':[blockSize*til 1+ceiling fileSize%blockSize]

// Create a block blob
headers:("x-ms-version";"Content-Type";"x-ms-blob-type")!("2019-02-02";"text/plain";"BlockBlob");
opts:`body`headers!("";headers);
resp:.kurl.sync ("https://<account>.blob.core.windows.net/mycontainer/myblob";`PUT; opts);
if[201i <> first resp; 'last resp];

// Generate block IDs, these need to be equal-length distinct strings
//  these strings must also be valid base64 + uri encoded, but are otherwise arbitrary
blockIDs:{raze string x} each 0x0 vs/: til count ranges;

// Upload all the blocks in parallel
uploadBlock:{[filePath; range; blockID]
    opts:`body`headers!(read1 (filePath;range 0; range[1] - range 0);headers);
    resp:.kurl.sync ("https://<account>.blob.core.windows.net/mycontainer/myblob?comp=block&blockid=",blockID;`PUT; opts);
    if[201i <> first resp;'last resp];
    }
.[uploadBlock[filePath;;];] peach flip (ranges;blockIDs);

// Now create the blob by putting all then block IDs together in order
blockListHead:("<?xml version=\"1.0\" encoding=\"utf-8\"?>";"<BlockList>")
blockListMid:"  <Latest>",/:blockIDs,\:"</Latest>";
blockListTail:enlist "</BlockList>";
headers:("x-ms-version";"Content-Type")#headers;
opts:`body`headers!("\n" sv blockListHead,blockListMid,blockListTail;headers);
resp:.kurl.sync ("https://<account>.blob.core.windows.net/mycontainer/myblob?comp=blocklist";`PUT; opts);
if[201i <> first resp;'last resp];

Example 3 - Upload a folder to Amazon S3

This example shows how to create non KDB-X files in object storage. KDB-X objects can be accessed using the objstor module instead of kurl.

Pre-requisite

Replace <bucket> and <region> with the lowercase names of your S3 bucket and AWS Region.

Example

Create a folder of example empty files.

mkdir -p upload/
touch upload/hello1.txt
touch upload/hello2.txt
touch upload/hello3.txt
touch upload/hello3.txt

bucket:"https://<bucket>.s3.<region>.amazonaws.com/";

// Recursively list directories, returning only files
getFiles:{$[11h=type d:key x;raze .z.s each` sv/:x,/:d;d]};
// Stringify and drop colons
files:reverse getFiles `:upload;

// Synchronously upload files one after another.
// @param bucket {string} HTTPS URL for bucket, must end in trailing slash
// @param hfile {symbol} Local file path
uploadFile:{[bucket;hfile]
    filename:1 _ string hfile;
    resp:.kurl.sync (bucket,filename; `PUT; ``file!(::;hfile));
    // if not HTTP 200 OK, or 201 Created, its failed
    if[not first[resp] in 200 201; 'last resp];
    }[bucket;];

uploadFile each files;

Example 4 - Get and put files into S3-Compatible storage

This example shows how to create and retrieve non KDB-X files from object storage. KDB-X objects should be accessed using the objstor module instead of kurl.

With s3-compatible storage set up, such as MinIO, kurl may be used to read files from storage.

Setting up MinIO would have you export an endpoint, similar to:

export KX_S3_ENDPOINT=http://127.0.0.1:9000"

See below an example statement showing how one would read a CSV from a bucket hosted on s3:

r: .kurl.sync ("http://127.0.0.1:9000/myBucket/hello.csv"; `GET; `service`region!("s3";"us-east-1"));
if[first resp <> 200; 'last resp];

// Print the CSV file
\c 200 200
show last r;

To upload a CSV file from in-memory variable:

// Upload in-memory text
myCSV:"\n" sv csv 0: ([] x:til 50; y: 50?`8);
r: .kurl.sync ("http://127.0.0.1:9000/myBucket/world.csv"; `PUT; `body`service`region!(myCSV; "s3";"us-east-1"));
if[not first[resp] in 200 201; 'last resp];

To upload a CSV file from local disk:

r: .kurl.sync ("http://127.0.0.1:9000/myBucket/world.csv"; `PUT; `file`service`region!(`:path/to/myCSV.csv; "s3";"us-east-1"));
if[not first[resp] in 200 201; 'last resp];

Example 5 - Create a namespace in AWS Cloud Map

Use AWS Cloud Map to create an HTTP namespace and wait for it to come online.

Create service.q.

// Create a new namespace, this returns an operation ID
body:.j.j enlist[`Name]!enlist "kxceNamespace"

headers:("Content-Type";"x-amz-target")!
  ("application/x-amz-json-1.1";"Route53AutoNaming_v20170314.CreateHttpNamespace")

URL:"https://servicediscovery.us-east-2.amazonaws.com"
resp:.kurl.sync (URL;`POST;`headers`body!(headers;body))
if[first resp <> 200; 'last resp]

checkOperation:{[resp]
  if[200 <> first resp; ' last resp];
  j:.j.k last resp;
  if["SUCCESS" ~ j . `Operation`Status; show "Namespace created"; .test.done:1b]; }

getOperation:{[url;id]
  show "Calling getOperation to see if namespace has been created...";
  headers:("Content-Type";"x-amz-target")!
    ("application/x-amz-json-1.1";"Route53AutoNaming_v20170314.GetOperation");
  opts:`headers`body`callback!(headers;id; checkOperation);
  .kurl.async (url;`POST;opts)
  }[URL;]

// The response of creating a namespace returns an input to GetOperation
.test.id:last resp
.test.done:0b
.z.ts:{ if[.test.done; system "t 0"; :(::)]; getOperation .test.id; }
\t 500

Start a kdb-x session and load service.q: This will check, ever 500ms, to see if the namespace has been created.

q)\l service.q

Next steps

• Check out the reference guide for information on available APIs.