How to use an Index in KDB.AI

This section describes how to use indexes in KDB.AI and key parameters that can be tailored to specific use cases.

Flat

The Flat search performs an exhaustive search against all vectors in the search space. You can configure it with a number of distance metrics. As the search is exhaustive, it finds the exact nearest neighbors without approximations.

Build parameters

Option	Description	Type	Required	Default
`dims`	Number of dimensions	long	true	8

Example: Flat index

schema = {'columns': [{'name': 'id', 'pytype': 'str'},
                        {'name': 'tag', 'pytype': 'str'},
                        {'name': 'text', 'pytype': 'bytes'},
                        {'name': 'embeddings',
                        'vectorIndex': {'dims': 1536, 'metric': 'L2', 'type': 'flat'}}]}
table = session.create_table('documents', schema)

Storage: the Flat index is stored in-memory.

qFlat

The qFlat search performs an exhaustive search against all vectors in the search space. You can configure it with a number of distance metrics. As the search is exhaustive, it finds the exact nearest neighbors without approximations.

Build parameters

Option	Description	Type	Required	Default
`dims`	Number of dimensions	long	true	8

Example: qFlat index

schema = {'columns': [{'name': 'id', 'pytype': 'str'},
                        {'name': 'tag', 'pytype': 'str'},
                        {'name': 'text', 'pytype': 'bytes'},
                        {'name': 'embeddings',
                        'vectorIndex': {'dims': 1536, 'metric': 'L2', 'type': 'qFlat'}}]}
table = session.create_table('documents', schema)

Storage: the qFlat index is stored on-disk.

The qFlat index is the Flat index stored on-disk instead of in-memory.

HNSW

A Hierarchical Navigable Small Worlds (HNSW) index establishes connections between vertices in the graph based on their distances. These links are instrumental in enabling efficient traversal and navigation through the hierarchical graph during the search process.

The HNSW index can be searched to navigate through the layers of the graph to find increasingly similar data in the graph. This approach is extremely efficient with search performance a measure of the complexity of the graph.

Build parameters

Option	Description	Type	Required	Default
`dims`	Number of dimensions	long	true	8
`efConstruction`	Nodes at each step (construction)	int	false	8
`M`	Valence of each node in graph	int	false	8

Search parameters

index_options at search() (and dense_index_options at hybrid_search())

Option	Description	Type	Required	Default
`efSearch`	Nodes considered at each step (search)	int	false	8

For optimal balance between accuracy and performance, choose a value of 2 - 10 times your k for efSearch. As a rule, increase the value for higher accuracy at the cost of slower search times.

For coding example of using the argument index_options at search() (and dense_index_options at hybrid_search()), see the Python API Client page.

Example: HNSW Index

schema = {'columns': [{'name': 'id', 'pytype': 'str'},
                        {'name': 'tag', 'pytype': 'str'},
                        {'name': 'text', 'pytype': 'bytes'},
                        {'name': 'embeddings',
                        'vectorIndex': {'dims': 1536,
                                        'metric': 'IP',
                                        'type': 'hnsw',
                                        'efConstruction' : 8, 'M': 8}}]}
table = session.create_table('documents', schema)

IVF

When using an Inverted File (IVF) search, first you train the index on a set of points that are used to generate cluster centroids using a k-means algorithm. The data is not partitioned into a cluster based on centroid distance. The search is performed by running a flat search against the most relevant clusters. As only a subset of the data is searched, the results are returned much quicker, but as a consequence can be less accurate.

Build parameters

Option	Description	Type	Required	Default
`nclusters`	Number of clusters	int	false	8

Search parameters

index_options at search() (and dense_index_options at hybrid_search())

Option	Description	Type	Required	Default
`clusters`	The number of clusters to be traversed in the search	int	false	2

Training is required to initialize the IVF index.

Example: IVF Index

schema = {'columns': [{'name': 'id', 'pytype': 'str'},
                        {'name': 'tag', 'pytype': 'str'},
                        {'name': 'text', 'pytype': 'bytes'},
                        {'name': 'embeddings',
                        'vectorIndex': {'trainingVectors': 1000,
                                        'metric': 'CS',
                                        'type': 'ivf',
                                        'nclusters': 10}}]}
table = session.create_table('documents', schema)

IVFPQ

You can compress input data using the product quantization method before applying the IVF schema above. This is known as Inverted File Product Quantization (IVFPQ). This can greatly reduce the size of the index held in memory and improve search speeds.

Build parameters

Option	Description	Type	Required	Default
`nclusters`	Number of clusters	int	false	8
`nsplits`	Number of splits	int	false	8
`nbits`	Number of bits	int	false	8

Search parameters

index_options at search() (and dense_index_options at hybrid_search())

Option	Description	Type	Required	Default
`clusters`	The number of clusters to be traversed in the search	int	false	2

Training is required, to initialize the IVFPQ index.

Example: IVFPQ Index

schema = {'columns': [{'name': 'id', 'pytype': 'str'},
                        {'name': 'tag', 'pytype': 'str'},
                        {'name': 'text', 'pytype': 'bytes'},
                        {'name': 'embeddings',
                        'vectorIndex': {'trainingVectors': 5000,
                                        'metric': 'L2',
                                        'type': 'ivfpq',
                                        'nclusters': 50,
                                        'nsplits': 8,
                                        'nbits': 8}}]}
table = session.create_table('documents', schema)