Send Feedback
Skip to content

Kurl Reference

This page provides detailed technical reference for the Kurl module APIs.

Registration

Registering with Kurl caches authentication information for a given domain wildcard, which subsequent requests against that domain will use automatically. The module supports several registration methods. The .kurl.init function cycles through these options automatically.

To call the underlying APIs directly, refer to the following sections.

Registration credentials

Prerequisite

When registering with a cloud provider, first obtain the required access credentials. Credentials are discovered using a provider chain – a hierarchy of locations checked in order. The hierarchy varies between providers and is documented in the following sections.

Amazon Web Services

  1. Using credentials from environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY:

    export AWS_ACCESS_KEY_ID=".."
export AWS_SECRET_ACCESS_KEY=".."
export AWS_SESSION_TOKEN=".."

  2. Using credentials from $HOME/.aws/credentials:

    cat ~/.aws/credentials
[default]
aws_access_key_id=".."
aws_secret_access_key=".."
aws_session_token=".."

    You can define multiple profiles. Each profile section uses the profile name as the tenant, except for the profile named default, which uses an empty string as its tenant.

  3. Using credentials from the file location defined within the environment variable AWS_SHARED_CREDENTIALS_FILE.

  4. Using IAM roles for service accounts through the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables, which EKS injects automatically when IAM annotations are configured on the service account.

  5. Using credentials requested from the URI defined in the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, if running under ECS.

  6. Using credentials from instance metadata if on an EC2 instance. Check for ec2 in /sys/hypervisor/uuid as shown in this user guide.

  7. Using credentials from instance metadata if on an EC2 instance. Check for Amazon EC2 in /sys/devices/virtual/dmi/id/bios_vendor as shown in the documentation.

Microsoft Azure

  1. Using account name and key from environment variables AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_SHARED_KEY:

    export AZURE_STORAGE_ACCOUNT="..."
export AZURE_STORAGE_SHARED_KEY="..."

  2. Using credentials from instance metadata if on an Azure instance. Check for ^7783-7084-3265-9085-8269-3286-77$ in /sys/devices/virtual/dmi/id/chassis_asset_tag as explained in the documentation.

  3. Using credentials from the az CLI command: az account get-access-token (if az is installed).

Google Cloud Platform

  1. Using credentials from internal metadata if on an GCP instance. Check for ^Google Compute Engine$ in /sys/devices/virtual/dmi/id/product_name as shown in the documentation.

  2. Using credentials from the gcloud CLI command: gcloud auth print-access-token (if gcloud is installed):

    gcloud init
export GCP_TOKEN=$(gcloud auth print-access-token)

.kurl.init

This API accepts an optional list of vendors to register. It calls .kurl.register using the credential‑lookup hierarchy described in the Registration credentials section.

Input parameters

Name Type Description Required Default
vendors list/symbols The cloud vendor to connect to, options include aws, azr, gcp, none or leave empty to register with 3 cloud vendors. No all

Example

.kurl:use`kx.kurl
.kurl.init`aws // register with AWS
.kurl.init[]  // register with AWS, AZR, GCP

.kurl.register

This API accepts a 4-item list. The elements of this list are defined below:

Input parameters

Name Type Description Required Default
type symbol One of aws_cred, aws_sts, oauth2, oauth2_jwt, azure, basic Yes None
domain str Wildcard for domain Yes None
tenant str/symbol Session name, username, ID, etc., meaningful to your application. Refer to tenants Yes ""
authInfo list/dict Authentication info specific to the type Yes None

Tenants

When registering with Kurl, the registration call accepts an optional tenant key. When you provide a tenant key, only requests that include the same tenant key and matching domain use the associated authentication information.

Tenants function as session keys. Using tenants, applications can be built to support multiple sessions. An application can maintain a mapping of tenants to session metadata.

For applications with user-based sessions, .z.u is a suitable tenant value.

// If current connection handle is defined, use existing authenticated username
// else use the default service account, no user triggered this call
// .e.g the timer triggered
tenant:[.z.w = 0i; ""; .z.u]

For simple applications where sessions are not required, use an empty string as the tenant.

When registering automatically, Kurl uses an empty string as the tenant, representing the default service account (IAM).

Domains

When making calls to a RESTful API, you supply an HTTP/HTTPS endpoint. An endpoint is a URI with the scheme [protocol://]host[:port]/path?query.

Provide domain names as wildcards when registering authentication inputs. The library applies authentication to all requests matching that domain.

// Register OAUth2 login for googleapis.com domain
.kurl.register (`oauth2; "*.googleapis.com"; ...)

// When you make requests to the googleapis.com domain,
// the supplied info (...) will be used
// For example, use the same info for storage and logging
.kurl.sync ("storage.googleapis.com"; `POST; ...) // Will match registration
.kurl.sync ("logging.googleapis.com"; `POST; ...) // Will match registration

To restrict authentication to a specific endpoint, provide an exact protocol, host, and port.

// Register OAuth2 login for specifically https logging URI
.kurl.register (`oauth2; "https://logging.googleapis.com"; ...)

.kurl.sync ("storage.googleapis.com"; `POST; ...) // Will not match registration
.kurl.sync ("logging.googleapis.com"; `POST; ...) // Will match registration

The domain parameter accepts either a wildcard domain name or a full URL (excluding the path and query component).

Example

Registering using environment variables:

info:`AccessKeyId`SecretAccessKey`Token!(getenv`AWS_ACCESS_KEY_ID;getenv`AWS_SECRET_ACCESS_KEY;getenv`AWS_SESSION_TOKEN)
.kurl.register (`aws_cred;"*amazonaws.com";"";info)

AWS registration

.kurl.aws.registerByCredentialsFile

This API accepts a credentials file.

Input parameters

Name Type Description Required Default
file symbol Credentials file location Yes None

Example

.kurl.aws.registerByCredentialsFile hsym`$getenv[`HOME],"/.aws/credentials"

setenv[`AWS_SHARED_CREDENTIALS_FILE] "/tmp/credentials"
.kurl.aws.registerByCredentialsFile hsym`$getenv`AWS_SHARED_CREDENTIALS_FILE

.kurl.deregister

This API accepts a 2-item list. The elements of this list are defined below:

Input parameters

Name Type Description Required Default
domain str Wildcard for domain Yes None
tenant str/symbol Session name, username, ID, etc., meaningful to your application Yes ""

Example

.kurl.deregister("*amazonaws.com";"")

Requests

Request examples

The examples below assume that a CSV file is available in an S3 bucket that you can access.

.kurl.sync

Input parameters

Name Type Description Required Default
url str URL to target Yes None
method str GET or POST request Yes None
options dict The available keys are covered below Yes ""

Options

Possible entries in options and their datatypes:

Option Type Description Example
body 4 10h Request body .kurl.sync (url; method;``body!(::;.j.j `a`b`c!1 2 3))
callback -11 100 104 105h Async callback .kurl.async (url; method;``callback!(::;{show x}))
file 10h HTTP PUT file .kurl.sync (url; method; ``file!(::;"/tmp/file.txt"))
follow_redirects -1h Flag to enable following redirect Location .kurl.sync (url; method; ``follow_redirects!(::;1b))
no_upload -1h For HTTP PUT, avoid setting CURLOPT_UPLOAD. This leaves content undefined, instead of 0 length .kurl.sync (url; `PUT; ``no_upload!(::;1b))
headers 99h Request headers .kurl.sync(url;method;``headers!(::;enlist["Content-Type"]!enlist "application/json"))
binary -1h Flag to force the response to be type 4h, regardless of MIME type .kurl.sync (url; method; ``binary!(::;1b))
insecure -1h Flag to enable curl --insecure .kurl.sync (url; method; ``insecure!(::;1b))
response_headers -1h Flag to include response headers .kurl.sync (url; method; ``response_headers!(::;1b))
max_redirects -6 -7h Maximum number of redirects .kurl.sync (url; method; ``max_redirects!(::; 5))
max_retry_attempts -6 -7h Maximum attempts for exponential backoff. Kurl retries on 503 errors by default, up to 10 times. Set to 0 to disable, or any other value to override the default of 10. When Kurl retries requests, it does not reset signed headers or temporal headers – if the server refuses requests older than a given time, retried requests still fail. Your application can handle longer retries by re-issuing the request. .kurl.sync (url; method; ``max_retry_attempts!(::; 5))
sign_query -1h Azure-specific flag. Azure Header Signing V2 has exclusive options to handle inconsistencies between Azure Monitor and Azure Storage. For Azure authentication, query strings are not signed in the CanonicalizedResource portion of the signature if the method is POST. This option exists for edge cases and should rarely be needed. .kurl.sync (url; `POST; ``sign_query!(::;1b))
region 10 -11h Explicitly set AWS signing region. If unspecified, it is parsed from the request URI. Typically set together with service. .kurl.sync (url; method;``region`service!(::;"us-east-1";"s3"))
service 10 -11h Explicitly set AWS signing service. If unspecified, it is parsed from the request URI. Typically set together with region. .kurl.sync (url; method;``region`service!(::;"us-east-1";"s3"))
tenant 10 -11h Session name – refer to Tenants .kurl.sync (url; method;``tenant!(::;"bob"))
timeout -6 -7 -12 -16h Request timeout in milliseconds. 0 is immediate, 0W is infinity .kurl.async (url; method; `callback`timeout!(cb; 5000))

Warning

When using HTTP PUT, specify body or file, not both. Leaving both blank is acceptable if the REST server uses the query string.

Backoff time

Backoff time (ms) is calculated as follows, where n is the number of failed attempts minus 1:

100 * 2 xexp n

The first retry occurs after 100ms, then 200ms, 400ms, 800ms, 1600ms, and so on.

The timeout flag takes precedence over remaining retry attempts — a request times out after the specified duration regardless of attempts remaining.

Example

.kurl.sync ("https://kx-bucket.s3.us-east-2.amazonaws.com/test_data/data.csv";`GET;::)
200i
"sym,price,size\nFDP,1.2,100\n"

// example with no registration required
.kurl.sync ("https://api.census.gov/data/2020/dec/pl?get=NAME,P1_001N&for=state:*"; `GET; ::)

.kurl.async

Input parameters

.kurl.async accepts the same input parameters as .kurl.sync, including the options dictionary.

Example

.kurl.async ("https://kx-bucket.s3.us-east-2.amazonaws.com/test_data/data.csv";`GET;``callback!(`;{.debug.x:x}))
.debug.x
200i
"sym,price,size\nFDP,1.2,100\n"