Skip to content

Quickstart Guide

This guide outlines the essential steps to using KDB.AI. Before proceeding, ensure your environment is set up as described in Pre-requisites and has the necessary information to connect to KDB.AI Cloud or KDB.AI Server.

The following instructions apply to both KDB.AI Cloud and KDB.AI Server users.

Required imports

# Copyright (c)2024 KX Systems
#
# Prequisites
# Before running this script ensure you follow pre-requisites in the cloud setup guide
# A valid kdbai endpoint and key which you can obtain following the setup process here https://cloud.kdb.ai/
import sys
import numpy as np
import pandas as pd
import kdbai_client as kdbai
from logging import Logger

# Endpoints created in setup guide
# Cloud instance
# session = kdbai.Session(endpoint="https://cloud.kdb.ai/instance/XXX", api_key="XXX")

# Local server
session = kdbai.Session(endpoint='http://localhost:8082')

Create a new table

Before creating a table you must first set the table schema. This is defined as a python dictionary containing a list of columns. For each column you must define the name and either a pytype or a qtype. The vector embeddings column should contain a vectorIndex attribute with the configuration of the index for similarity search. Full schema definition specifications are available in the manage tables section. 

schema = {'columns': [{'name': 'id', 'pytype': 'str'},
                        {'name': 'vectors',
                        'vectorIndex': {'dims': 8, 'metric': 'L2', 'type': 'flat'}}]}

# Create a test table called 'quickstartkdbai'
table_name = 'quickstartkdbai'
try:
    table = session.create_table(table_name, schema)
    # Check table was created successfully
    if(table_name in session.list()):
        f"Table {table_name} was created successfully"
    else:
        Logger.info(f"Could not create the table \'{table_name}\' in endpoint \'{session.endpoint}\'")
except Exception as e:
    (f'Exception {e} occurred creating the table \'{table_name}\'. Check endpoint and key and that table does not exist already.')
    sys.exit(1)
finally:
    pass


curl "Content-Type: application/json" localhost:8082/api/v1/config/table/quickstartkdbai -d @table.json

The request body is in a file called table.json:

{
    "type": "splayed",
    "columns": [
      {"name": "id", "type": "symbol"},
      {
       "name": "vectors",
       "type": "reals",
       "vectorIndex": {
         "dims": 8,
         "type": "flat",
          "metric": "L2"
        }
       }
   ]
}

Retrieve a list of tables

Display a list of tables, including your recently created table, using the following command.

session.list()
# This command should return an array with the table: ['quickstartkdbai']

curl -s localhost:8082/api/v1/config/table

If you pipe the result to a tool such as jq you can see the result pretty printed:

curl -s localhost:8082/api/v1/config/table | jq
{
  "quickstartkdbai": {
    "type": "splayed",
    "columns": [
      {
        "name": "id",
        "type": "symbol"
      },
      {
        "name": "vectors",
        "type": "reals",
        "vectorIndex": {
          "dims": 8,
          "type": "flat",
          "metric": "L2"
        }
      }
    ]
  }
}

Add data to your table

Generate a vector of five 8-dimensional vectors that will be the vector embeddings.

You can then add these to the pandas dataframe ensuring the column names/types match the table schema.

# Insert a row of data with sample vectors
try:
    ids = ['h', 'e', 'l', 'l', 'o']  # Example ID values
    vectors = np.random.rand(40).astype(np.float32).reshape(5,8)
    df = pd.DataFrame({"id": ids, "vectors": list(vectors)})
    table.insert(df)
# Check we have our row
    if(len(table.query().values) == 5):
        f"Row with ID {id} in table {table_name} was inserted successfully"
    else:
        Logger.info(f"Could not insert a row with ID {id} into table {table_name}.")
except Exception as e:
    f'Exception {e} occurred trying to insert a row with ID {id} into table {table_name}.'
    sys.exit(1)
finally:
    pass

JSON data can be generated in any language. Below .j.j from q is used to generate example data and write to a file to insert.json. 

# start the kdb+ binary with the q command
q
# run the following on the 'q)' prompt
`insert.json 0: enlist .j.j `table`rows!(`quickstartkdbai;([] id:"hello";vectors:(5;8)#(5*8)?1e))
\\

The contents of the generated file should be:

$ cat insert.json
{"table":"quickstartkdbai","rows":
[
    {"id":"h","vectors":[0.3927524,0.5170911,0.5159796,0.4066642,0.1780839,0.3017723,0.785033,0.5347096]},
    {"id":"e","vectors":[0.7111716,0.411597,0.4931835,0.5785203,0.08388858,0.1959907,0.375638,0.6137452]},
    {"id":"l","vectors":[0.5294808,0.6916099,0.2296615,0.6919531,0.4707882,0.6346716,0.9672399,0.2306385]},
    {"id":"l","vectors":[0.949975,0.439081,0.5759051,0.5919004,0.8481566,0.389056,0.391543,0.08123546]},
    {"id":"o","vectors":[0.9367504,0.2782122,0.2392341,0.1508133,0.1567317,0.9785,0.7043314,0.9441671]}
    ]
}

As above, if you pipe the result to a tool like jq you can see the result pretty printed.

$ cat insert.json | jq
{
  "table": "quickstartkdbai",
  "rows": [
    {
      "id": "h",
      "vectors": [
        0.3927524,
        0.5170911,
        0.5159796,
        0.4066642,
        0.1780839,
        0.3017723,
        0.785033,
        0.5347096
      ]
    },
...

{
    "type": "splayed",
    "columns": [
      {"name": "id", "type": "symbol"},
      {
       "name": "vectors",
       "type": "reals",
       "vectorIndex": {
         "dims": 8,
         "type": "flat",
          "metric": "L2"
        }
       }
   ]
}

Query the table

Use the following command to query data from the table.

The query function accepts a wide range of arguments to make it easy to filter, aggregate, and sort. Run ?table.query to see them all.

data = table.query()
print(f'Table data:\n {data}')

curl -s -H "Content-Type: application/json" localhost:8082/api/v1/data -d '{"table":"quickstartkdbai"}'

As above, if you pipe the result to a tool like jq you can see the result pretty printed.

curl -s -H "Content-Type: application/json" 13.93.68.168:28082/api/v1/data -d '{"table":"quickstartkdbai"}' | jq ".payload"
[
  {
    "id": "h",
    "vectors": [
      0.3503749,
      0.7027296,
      0.07756515,
      0.4045712,
      0.5368044,
      0.9517315,
      0.1455024,
      0.2176988
    ]
  },
...

Search for the nearest neighbours using the following command.

The dimension of input query vectors must match the vector embedding dimensions in the table, defined in schema above.

# Run a similarity search
results = table.search(vectors=[[0.1, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]], n=3)
print(f'Similarity search results: {results}')

curl -s -H "Content-Type: application/json" localhost:8082/api/v1/search -d '{"table":"quickstartkdbai","n":3,"vectors":[[0.1, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]],"distances":"dist"}'

As above, if you pipe the result to a tool like jq you can see the result pretty printed.

curl -s -H "Content-Type: application/json" 13.93.68.168:28082/api/v1/search -d '{"table":"quickstartkdbai","n":3,"vectors":[[0.1, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]],"distances":"dist"}' | jq '."payload"'
[
  [
    {
      "id": "l",
      "vectors": [
        0.1692735,
        0.6401389,
        0.9439349,
        0.5964162,
        0.2678166,
        0.7119821,
        0.8890669,
        0.6621503
      ],
      "dist": 0.6006493
    },
    {
      "id": "e",
      "vectors": [
        0.7630598,
        0.645776,
...

The closest matching neighbors for the query vector are returned along with the calculation of L2 (Euclidean Distance) similarity.

The search API supports batch querying and filtered search.

Delete table

Use the following command when you want to delete a table.

# Clean up our table
if(table_name in session.list()):
    session.table(table_name).drop()

curl -s -X DELETE localhost:8082/api/v1/config/table/quickstartkdbai

Once you delete a table, you cannot use it again.

In KDB.AI, when you delete a table, the associated index is also removed.

Next steps

Now that you are successfully making indexes with KDB.AI, you can start inserting your own data and analysing it:

Samples

You can also explore the following: