Glossary

Cosine

Cosine similarity is a measure of the cosine of the angle between the two non-zero vectors.

Euclidian

A Euclidean vector is a geometric object that has magnitude, or length, and direction.

Hierarchical Navigable Small Worlds (HNSW)

At a fundamental level a HNSW incrementally generates a hierarchical, multi-level graph structure which allows searches to navigate through the layers of the graph to find increasingly similar data in the graph to the data being searched greedily. This approach is extremely efficient with search performance a measure of the complexity of the graph.

Inverted File with Product Quantization (IVFPQ)

The IVFPQ (Inverted File with Product Quantization) is a data structure and algorithm used in the Faiss library for efficient approximate nearest neighbor search in high-dimensional spaces. It's leveraging the Inverted File and Product Quantization techniques to accelerate the search process.

The IVFPQ algorithm divides the vector space into a set of Voronoi cells (partitions of a vector space) using the Product Quantization method. Then each cell is associated with an inverted list that stores the identifiers of the vectors falling into that cell in an inverted file structure. This ivf structure provides a mapping between each Voronoi cell and the vectors associated with it.

During the search, a query vector is assigned to a specific Voronoi cell, only the vectors in that cell are considered for the nearest neighbor search. This significantly reduces the search space and improves the efficiency of the search process.

To find the nearest neighbors, the algorithm computes the distances between the query vector and the vectors in the assigned Voronoi cell using a distance metric (e.g., Euclidean distance or cosine similarity).

The search algorithm efficiently traverses the inverted file and retrieves the vectors with the closest distances to the query vector.

Large Language Models (LLMs)

Large language models are algorithmic predictors for text, which are able to process enormous amounts of text data. These models come in various forms including: Autoencoder-Based, Sequence-to-Sequence, Recursive Neural Network, and the most well-known, Transformer-Based Models.