How to perform a hybrid search

This section details how to execute a hybrid search in KDB.AI.

Added in v1.1.0.

Hybrid search in KDB.AI

In KDB.AI, a hybrid search returns results based on both dense and sparse vector search. The process includes querying and re-ranking of results across both vector sets. A parameter named alpha allows you to control the weight factor on the search for each algorithm.

Before we dive in, go to the understanding hybrid search page to make sure you're familiar with dense vectors, sparse vectors, tokenization, the Best Matching 25 (BM25) algorithm, and inverted indexes.

A hybrid search blends a dense search with a sparse search. Therefore, you need to perform the following:

Dense search.
Sparse search.
Hybrid search.

Setup

Before you start, make sure you have:

a KDB.AI Server license
set up your KDB.AI Server
the latest version of the KDB.AI server and Python client

1. Dense search

For details on how to perform a standalone dense search, go to the How to conduct a similarity search page.

2. Sparse search

This section details how to perform a standalone sparse search.

2.1 Create table with a sparse index in KDB.AI

To use sparse vectors, you need to create a table with a sparse index.

Parameters

Option	Description	Type	Required	Default
`k`	Term saturation	real	false	1.25
`b`	Document length impact on relevance	real	false	0.75

The values of k and b can be modified in place at search time.

PythonJSON

schema = {'columns': [
     {'name': 'id', 'pytype': 'int32'},
     {'name': 'sym', 'pytype': 'str'},
     {'name': 'time', 'pytype': 'datetime64[ns]'},
     {'name': 'text', 'pytype': 'dict',
         'sparseIndex': {'k': 1.25, 'b': 0.75}]}

{
 "type": "splayed",
 "columns": [
    {"name": "id", "type": "int"},
    {"name": "sym", "type": "char"},
    {"name": "time", "type": "timestamp"},
    {
     "name": "text",
 "type": "",
     "sparseIndex": {
        "k": 1.25,
        "b": 0.75,
     }
    }
   ]
  }

You can use these schemas to create the tables as shown on the Manage Tables page.

2.2 Insert sparse vectors

Before the data is inserted, the sparse vectors should be prepared as dictionaries with integer keys and values:

PythonJSON

{1996: 2, 101: 1, 11190: 1, 5598: 1, 2058: 1, 4231: 1, 102: 1}

{'1996': 2, '101': 1, '11190': 1, '5598': 1, '2058': 1, '4231': 1, '102': 1}

These sparse vectors are inserted as seen before.

2.3. Perform a sparse search

The search endpoint is overloaded to accept lists of sparse vectors:

Parameters Index Options

Option	Description	Type	Required	Default
`k`	Term saturation	real	false	1.25
`b`	Document length impact on relevance	real	false	0.75

PythonJSON

table.search(vectors=[{1:2,3:1}],n=3)

curl -s -H "Content-Type: application/json" localhost:8082/api/v1/kxi/search \
-d '{"table":"documents","n":3,"vectors":[{"1":2,"3":1}]}'

3. Hybrid search

Combining the sparse and dense searches is the basis of the hybrid search. You can use the parameter alpha to determine the weighting of each leg of the search. A value of 0 means only the sparse leg is used and a value of 1 means the dense leg only is used.

For intermediate values, the following algorithm demonstrates an example that assumes:

the number of matches is 4.
alpha is 0.6.
the dense search returned vectors in order 3, 2, 1, 5.
the sparse search returned vectors in order 4, 3, 2, 1.

Vector	Sparse	Dense	Re-ranked
Vector 1	3	4	3
Vector 2	2	3	2
Vector 3	1	2	1
Vector 4	null	1	4
Vector 5	5	null	5

We deduced the combined ranking (re-ranked) by the reciprocal rank fusion scores defined by the following formula:

\[ \textrm{score}(i) = \frac{1-\alpha}{1+\textrm{sparserank}(i)} + \frac{\alpha}{1+\textrm{denserank}(i)} \]

We computed the scores as below:

Since vector 1 occurs in the sparse search in position 3 and in the dense search in position 4, it receives a score of 0.22, as calculated below:

${\color{NavyBlue} {\fontsize{10}{10} 0.4\cdot(1/(1+3)) + 0.6\cdot(1/(1+4)) }}$

Since vector 2 occurs in the sparse search in position 2 and in the dense search in position 3, it receives a score of 0.28:

${\color{NavyBlue} {\fontsize{10}{10} 0.4\cdot(1/(1+2)) + 0.6\cdot(1/(1+3)) }}$

Since vector 3 occurs in the sparse search in position 1 and in the dense search in position 2, it receives a score of 0.4:

${\color{NavyBlue} {\fontsize{10}{10} 0.4\cdot(1/(1+1)) + 0.6\cdot(1/(1+2)) }}$

Since vector 4 doesn't occur in the sparse search and in the dense search in position 1, it receives a score of 0.2:

${\color{NavyBlue} {\fontsize{10}{10} 0.4\cdot(0) + 0.6\cdot(1/(1+2)) }}$

Since vector 5 occurs in the sparse search in position 5 and doesn't occur in the dense search, it receives a score of 0.08:

${\color{NavyBlue} {\fontsize{10}{10} 0.4\cdot(1/(1+4)) + 0.6\cdot(0) }}$

Comparing these five scores, the top four results are returned based on the sorted value.

Summary

To perform a hybrid search, you need to define the dense and sparse vectors in the schema. See the Schema section on the Manage Tables page.

PythonJSON

schema = {'columns': [
     {'name': 'id', 'pytype': 'int32'},
     {'name': 'sym', 'pytype': 'str'},
     {'name': 'time', 'pytype': 'datetime64[ns]'},
     {'name': 'dense', 'pytype': 'float32',
         'vectorIndex': {'type': 'hnsw', 'dims': 10, 'metric': 'L2'}},
     {'name': 'sparse', 'pytype': 'dict',
         'sparseIndex': {'k': 1.25, 'b': 0.75}}]}

{
 "type": "splayed",
 "columns": [
    {"name": "id", "type": "int"},
    {"name": "sym", "type": "char"},
    {"name": "time", "type": "timestamp"},
    {
     "name": "dense",
     "type": "reals",
     "vectorIndex": {
        "type": "hnsw",
        "metric": "L2",
        "dims": 10}
    },
    {
     "name": "sparse",
     "type": "",
     "sparseIndex": {
        "k": 1.25,
        "b": 0.75}
    }
   ]
  }

Once you insert the data, perform the hybrid search:

PythonJSON

table.hybrid_search(dense_vectors=[[0,0,0,0,0,0,0,0,0,0]],sparse_vectors=[{1:2,3:1}],n=3)

curl -s -H "Content-Type: application/json" localhost:8082/api/v1/kxi/hybridSearch \
-d '{"table":"documents","n":3,"sparseVectors":[{"1":2,"3":1}],"denseVectors"=[0,0,0,0,0,0,0,0,0,0]}'

Code example

This example demonstrates how to perform each of the three searches on some dummy data.

Click here to expand the code sample

import pandas as pd
import numpy as np
import kdbai_client as kdbai

### start session

session = kdbai.Session()

### create table

schema = {'columns': [
        {'name': 'id', 'pytype': 'int32'},
        {'name': 'sym', 'pytype': 'str'},
        {'name': 'dense', 'pytype': 'float32',
            'vectorIndex': {'type': 'hnsw', 'dims': 10, 'metric': 'L2'}},
        {'name': 'sparse', 'pytype': 'dict',
            'sparseIndex': {'k': 1.25, 'b': 0.75}}]}

table = session.create_table("example", schema)

### insert data

df = pd.DataFrame({
'id': range(1000),
'sym': np.random.choice(['AAA', 'BBB'], 1000),
'dense': np.random.rand(1000, 10).tolist(),
'sparse': [{int(y+1):1 for y in np.random.choice(range(1200),x+1,replace=False)} for x in np.random.choice(range(120),1000)]})

table.insert(df)

### Dense search

table.search([[0.1,0,0,0,0,0,0,0,0,0]],n=3)

### Sparse search

table.search([{1:1,4:1,34:2,2:1}],n=3)

### Hybrid search

table.hybrid_search(dense_vectors=[[0.1,0,0,0,0,0,0,0,0,0]],sparse_vectors=[{1:1,4:1,34:2,2:1}],n=3)

Next steps

Now that you're familiar with hybrid search, you can do the following:

Check out the use cases and learn how to improve accuracy with hybrid search from the Learning hub.
Visit our GitHub repo.
Open the sample.
Run the notebook in Google Colab.
Learn how to perform a Transformed Temporal Similarity Search.
Learn how to perform a Non-Transformed Temporal Similarity Search.