# Kurl Reference

*This page provides detailed technical reference for the Kurl module APIs.*

## Registration

Registering with Kurl caches authentication information for a given domain wildcard, which subsequent requests against that domain will use automatically. The module supports several registration methods. The `.kurl.init` function cycles through these options automatically.

To call the underlying APIs directly, refer to the following sections.

## Registration credentials

> **Note — Prerequisite**
>
>
> When registering with a cloud provider, first obtain the required access credentials. Credentials are discovered using a provider chain – a hierarchy of locations checked in order. The hierarchy varies between providers and is documented in the following sections.
>
### Amazon Web Services

1. Using credentials from environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`:
    ```bash
    export AWS_ACCESS_KEY_ID=".."
    export AWS_SECRET_ACCESS_KEY=".."
    export AWS_SESSION_TOKEN=".."
    ```

1. Using credentials from `$HOME/.aws/credentials`:
    ```bash
    cat ~/.aws/credentials
    [default]
    aws_access_key_id=".."
    aws_secret_access_key=".."
    aws_session_token=".."
    ```

    You can define multiple profiles. Each profile section uses the profile name as the tenant, except for the profile named `default`, which uses an empty string as its tenant.

1. Using credentials from the file location defined within the environment variable `AWS_SHARED_CREDENTIALS_FILE`.

1. Using [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) through the `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` environment variables, which EKS injects automatically when IAM annotations are configured on the service account.

1. Using credentials requested from the URI defined in the environment variable `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`, if running under ECS.

1. Using credentials from instance metadata if on an EC2 instance. Check for `ec2` in `/sys/hypervisor/uuid` as shown in this [user guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html).

1. Using credentials from instance metadata if on an EC2 instance. Check for `Amazon EC2` in `/sys/devices/virtual/dmi/id/bios_vendor` as shown in [the documentation](https://github.com/openshift/openshift-ansible/pull/6792/files).

#### Amazon Web Services SDK

The [AWS SDK module](https://github.com/KxSystems/awssdk) provides a wrapper to the AWS C++ SDK. 

> **Note — AWS SDK environment variables**
>
>  1. KURL_USE_AWS_SDK — set to true to use the AWS SDK module
>  1. AWS_PROFILE — if using `$HOME/.aws/credentials` or `$HOME/.aws/config`, specify the profile to authenticate with
>  1. KX_KURL_AWS_SDK_LOG_PREFIX — optional parameter to set AWS C++ SDK log path 
>
>  ```bash
>  AWS_PROFILE=rolesanywhere 
>  KURL_USE_AWS_SDK=1
>  KX_KURL_AWS_SDK_LOG_PREFIX="/tmp/sdk_log/aws_"
>  ```
>
The AWS SDK module supports the following methods for authentication:

1. Using credentials from environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`:
    ```bash
    export AWS_ACCESS_KEY_ID=".."
    export AWS_SECRET_ACCESS_KEY=".."
    export AWS_SESSION_TOKEN=".."
    ```

1. Using credentials from `$HOME/.aws/credentials`:
    ```bash
    cat ~/.aws/credentials
    [myprofile]
    aws_access_key_id=".."
    aws_secret_access_key=".."
    aws_session_token=".."
    ```

    You can define multiple profiles. Each profile section uses the profile name as the tenant, except for the profile named `default`, which uses an empty string as its tenant.

1. Using credentials from `$HOME/.aws/config`:

    This can be used for [RolesAnywhere](https://aws.amazon.com/iam/roles-anywhere/) authentication
    ```bash
    $ cat ~/.aws/config
    [profile rolesanywhere]
    output = json
    region = eu-west-1
    credential_process = aws_signing_helper credential-process --certificate /home/username/certs/rolesanywhere-client-cert.pem --private-key /home/username/certs/rolesanywhere-client-key.pem --trust-anchor-arn arn:aws:rolesanywhere:eu-west-1:<redacted>:trust-anchor/<redacted> --profile-arn arn:aws:rolesanywhere:eu-west-1:<id>:profile/<redacted> --role-arn arn:aws:iam::<id>:role/rolesanywhere-personal-role
    ```


### Microsoft Azure

1. Using `account name` and `key` from environment variables `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_SHARED_KEY`:
    ```bash
    export AZURE_STORAGE_ACCOUNT="..."
    export AZURE_STORAGE_SHARED_KEY="..."
    ```

1. Using credentials from instance metadata if on an Azure instance. Check for `^7783-7084-3265-9085-8269-3286-77$` in `/sys/devices/virtual/dmi/id/chassis_asset_tag` as explained in the [documentation](https://github.com/Microsoft/PowerShell-DSC-for-Linux/blob/master/Providers/nxOMSAutomationWorker/automationworker/worker/linuxutil.py).

1. Using credentials from the `az` CLI command: `az account get-access-token` (if `az` is installed).

### Google Cloud Platform

1. Using credentials from internal metadata if on an GCP instance. Check for `^Google Compute Engine$` in `/sys/devices/virtual/dmi/id/product_name` as shown in the [documentation](https://cloud.google.com/compute/docs/instances/managing-instances#use_other_method).

1. Using credentials from the `gcloud` CLI command: `gcloud auth print-access-token` (if `gcloud` is installed):
    ```bash
    gcloud init
    export GCP_TOKEN=$(gcloud auth print-access-token)
    ```

## .kurl.init

This API accepts an optional list of vendors to register. It calls `.kurl.register` using the credential‑lookup hierarchy described in the [Registration credentials](#registration-credentials) section.

### Input parameters

| **Name**      | **Type**          | **Description** | **Required** | **Default** |
|-----------|---------------|-------------|----------|---------|
| vendors  | list/symbols | The cloud vendor to connect to, options include `aws`, `azr`, `gcp`, `none` or leave empty to register with 3 cloud vendors. | No | all |

### Example
```q
.kurl:use`kx.kurl
.kurl.init`aws // register with AWS
.kurl.init[]  // register with AWS, AZR, GCP
```

## .kurl.register

This API accepts a 4-item list. The elements of this list are defined below:

### Input parameters

| **Name**      | **Type**       | **Description** | **Required** | **Default** |
|-----------|------------|-------------|----------|---------|
| type      | symbol     | One of `aws_cred`, `aws_sts`, `oauth2`, `oauth2_jwt`, `azure`, `basic` | Yes | None |
| domain    | str        | Wildcard for domain | Yes | None |
| tenant    | str/symbol | Session name, username, ID, etc., meaningful to your application. Refer to [tenants](#tenants) | Yes | "" |
| authInfo  | list/dict  | Authentication info specific to the type | Yes | None |

#### Tenants

When registering with Kurl, the registration call accepts an optional tenant key. When you provide a tenant key, only requests that include the same tenant key and matching domain use the associated authentication information.

Tenants function as session keys. Using tenants, applications can be built to support multiple sessions. An application can maintain a mapping of tenants to session metadata.

For applications with user-based sessions, `.z.u` is a suitable tenant value.
```q
// If current connection handle is defined, use existing authenticated username
// else use the default service account, no user triggered this call
// .e.g the timer triggered
tenant:[.z.w = 0i; ""; .z.u]
```

For simple applications where sessions are not required, use an empty string as the tenant.

When registering automatically, Kurl uses an empty string as the tenant, representing the default service account (IAM).

#### Domains

When making calls to a RESTful API, you supply an HTTP/HTTPS endpoint. An endpoint is a URI with the scheme `[protocol://]host[:port]/path?query`.

Provide domain names as wildcards when registering authentication inputs. The library applies authentication to all requests matching that domain.
```q
// Register OAUth2 login for googleapis.com domain
.kurl.register (`oauth2; "*.googleapis.com"; ...)

// When you make requests to the googleapis.com domain,
// the supplied info (...) will be used
// For example, use the same info for storage and logging
.kurl.sync ("storage.googleapis.com"; `POST; ...) // Will match registration
.kurl.sync ("logging.googleapis.com"; `POST; ...) // Will match registration
```

To restrict authentication to a specific endpoint, provide an exact protocol, host, and port.
```q
// Register OAuth2 login for specifically https logging URI
.kurl.register (`oauth2; "https://logging.googleapis.com"; ...)

.kurl.sync ("storage.googleapis.com"; `POST; ...) // Will not match registration
.kurl.sync ("logging.googleapis.com"; `POST; ...) // Will match registration
```

The domain parameter accepts either a wildcard domain name or a full URL (excluding the path and query component).

### Example

Registering using environment variables:
```q
info:`AccessKeyId`SecretAccessKey`Token!(getenv`AWS_ACCESS_KEY_ID;getenv`AWS_SECRET_ACCESS_KEY;getenv`AWS_SESSION_TOKEN)
.kurl.register (`aws_cred;"*amazonaws.com";"";info)
```

## AWS registration

### .kurl.aws.registerByCredentialsFile

This API accepts a credentials file.

#### Input parameters

| **Name**      | **Type**       | **Description** | **Required** | **Default** |
|-----------|------------|-------------|----------|---------|
| file      | symbol     | Credentials file location | Yes | None |

#### Example
```q
.kurl.aws.registerByCredentialsFile hsym`$getenv[`HOME],"/.aws/credentials"
```
```q
setenv[`AWS_SHARED_CREDENTIALS_FILE] "/tmp/credentials"
.kurl.aws.registerByCredentialsFile hsym`$getenv`AWS_SHARED_CREDENTIALS_FILE
```

## .kurl.deregister

This API accepts a 2-item list. The elements of this list are defined below:

### Input parameters

| **Name**      | **Type**       | **Description** | **Required** | **Default** |
|-----------|------------|-------------|----------|---------|
| domain    | str        | Wildcard for domain | Yes | None |
| tenant    | str/symbol | Session name, username, ID, etc., meaningful to your application | Yes | "" |

### Example
```q
.kurl.deregister("*amazonaws.com";"")
```

## Requests

> **Note — Request examples**
>
> The examples below assume that a CSV file is available in an S3 bucket that you can access.
>
## .kurl.sync

### Input parameters

| **Name**      | **Type**   | **Description** | **Required** | **Default** |
|-----------|--------|-------------|----------|---------|
| url       | str    | URL to target | Yes | None |
| method    | str    | GET or POST request | Yes | None |
| options   | dict   | The available keys are covered below | Yes | "" |

#### Options

Possible entries in `options` and their datatypes:

| **Option** | **Type** | **Description** | **Example** |
|---|---|---|---|
| `body` | `4 10h` | Request body | ```.kurl.sync (url; method;``body!(::;.j.j `a`b`c!1 2 3))``` |
| `callback` | `-11 100 104 105h` | Async callback | ```.kurl.async (url; method;``callback!(::;{show x}))``` |
| `file` | `10h` | HTTP PUT file | ```.kurl.sync (url; method;  ``file!(::;"/tmp/file.txt"))``` |
| `follow_redirects` | `-1h` | Flag to enable following redirect `Location` | ```.kurl.sync (url; method; ``follow_redirects!(::;1b))``` |
| `no_upload` | `-1h` | For HTTP PUT, avoid setting `CURLOPT_UPLOAD`. This leaves content undefined, instead of 0 length | ```.kurl.sync (url; `PUT; ``no_upload!(::;1b))``` |
| `headers` | `99h` | Request headers | ```.kurl.sync(url;method;``headers!(::;enlist["Content-Type"]!enlist "application/json"))``` |
| `binary` | `-1h` | Flag to force the response to be type `4h`, regardless of MIME type | ```.kurl.sync (url; method; ``binary!(::;1b))``` |
| `insecure` | `-1h` | Flag to enable `curl --insecure` | ```.kurl.sync (url; method; ``insecure!(::;1b))``` |
| `response_headers` | `-1h` | Flag to include response headers | ```.kurl.sync (url; method; ``response_headers!(::;1b))``` |
| `max_redirects` | `-6 -7h` | Maximum number of redirects | ```.kurl.sync (url; method; ``max_redirects!(::; 5))``` |
| `max_retry_attempts` | `-6 -7h` | Maximum attempts for exponential backoff. Kurl retries on 503 errors by default, up to 10 times. Set to `0` to disable, or any other value to override the default of 10. When Kurl retries requests, it does not reset signed headers or temporal headers – if the server refuses requests older than a given time, retried requests still fail. Your application can handle longer retries by re-issuing the request. | ```.kurl.sync (url; method; ``max_retry_attempts!(::; 5))``` |
| `sign_query` | `-1h` | Azure-specific flag. Azure Header Signing V2 has exclusive options to handle inconsistencies between Azure Monitor and Azure Storage. For Azure authentication, query strings are not signed in the `CanonicalizedResource` portion of the signature if the method is POST. This option exists for edge cases and should rarely be needed. | ```.kurl.sync (url; `POST; ``sign_query!(::;1b))``` |
| `region` | `10 -11h` | Explicitly set AWS signing region. If unspecified, it is parsed from the request URI. Typically set together with `service`. | ```.kurl.sync (url; method;``region`service!(::;"us-east-1";"s3"))``` |
| `service` | `10 -11h` | Explicitly set AWS signing service. If unspecified, it is parsed from the request URI. Typically set together with `region`. | ```.kurl.sync (url; method;``region`service!(::;"us-east-1";"s3"))``` |
| `tenant` | `10 -11h` | Session name – refer to [Tenants](#tenants) | ```.kurl.sync (url; method;``tenant!(::;"bob"))``` |
| `timeout` | `-6 -7 -12 -16h` | Request timeout in milliseconds. `0` is immediate, `0W` is infinity | ```.kurl.async (url; method; `callback`timeout!(cb; 5000))``` |

> **Warning**
>
> When using HTTP PUT, specify `body` or `file`, not both. Leaving both blank is acceptable if the REST server uses the query string.
>
> **Note — Backoff time**
>
> Backoff time (ms) is calculated as follows, where `n` is the number of failed attempts minus 1:
>
> ```q
> 100 * 2 xexp n
> ```
>
> The first retry occurs after 100ms, then 200ms, 400ms, 800ms, 1600ms, and so on.
>
> The `timeout` flag takes precedence over remaining retry attempts — a request times out after the specified duration regardless of attempts remaining.
>
### Example
```q
.kurl.sync ("https://kx-bucket.s3.us-east-2.amazonaws.com/test_data/data.csv";`GET;::)
200i
"sym,price,size\nFDP,1.2,100\n"

// example with no registration required
.kurl.sync ("https://api.census.gov/data/2020/dec/pl?get=NAME,P1_001N&for=state:*"; `GET; ::)
```

## .kurl.async

### Input parameters

`.kurl.async` accepts the same input parameters as [`.kurl.sync`](#kurlsync), including the [`options`](#options) dictionary.

### Example
```q
.kurl.async ("https://kx-bucket.s3.us-east-2.amazonaws.com/test_data/data.csv";`GET;``callback!(`;{.debug.x:x}))
.debug.x
200i
"sym,price,size\nFDP,1.2,100\n"
```
