Skip to content

Previous KDB.AI Release Notes

This page documents previous updates to KDB.AI Cloud UI and KDB.AI Server. Bookmark it to set yourself up for continued success while using KDB.AI.

KDB.AI Cloud UI

v1.5.0

Release date 2024.11.28

We’re excited to bring you several updates focused on enhancing your user experience with KDB.AI Cloud UI:

UX Improvements

  • YouTube integration: Added a View all button that links directly to our YouTube channel for easy access to all videos.
  • Enhanced navigation: Renamed and reordered tabs for easier access to key features.
  • Streamlined setup: Reorganized and moved Connection panels to the Overview tab for better navigation. Quick-to-watch Video tutorials include duration for added clarity.

The KDB.AI Cloud UI 1.5.0 release also includes the latest capabilities and improvements introduced in the KDB.AI Server 1.5.0 release.

v1.4.0

Release date 2024.10.21

New

The KDB.AI Cloud UI 1.4.0 release contains the latest capabilities and improvements introduced in the KDB.AI Server 1.4.0 release.

v1.3.0

Release date 2024.09.19

New

The KDB.AI Cloud UI 1.2.4 release contains all the latest capabilities and enhancements introduced in the KDB.AI Server 1.3.0 release.

v1.2.4

Release date 2024.08.15

Fixes and improvements

  • This update includes several bug fixes and enhancements to improve KDB.AI performance and stability.
  • v1.2.2 and v1.2.3 introduced a series of internal changes that significantly enhance KDB.AI reliability and performance.

New

  • Request new database button is now available to new users who haven't created a table within 7 days of creating their KDB.AI account.

v1.2.1

Release date 2024.07.02

Fixes and improvements

  • KDB.AI Cloud UI 1.2.1 includes bug fixes and internal improvements.

v1.2.0

Release date 2024.06.27

The KDB.AI Cloud UI 1.2.0 release contains all the new functionalities and performance improvements introduced in the KDB.AI Server 1.2.0 release.

v1.1.2

Release date 2024.05.23

Fixes and improvements

  • KDB.AI sample notebooks can be run from the UI directly in Google Colab. No need to switch windows or install the required libraries on local laptop/environment anymore.

v1.1.1

Release date 2024.05.09

Fixes and improvements

  • During the sign-up process, personal details are now entered after login, rather than on the signup page.
  • You can use Google SSO to sign up and login.

v1.1.0

Release date 2024.03.27

New

  • The KDB.AI Cloud UI 1.1.0 release contains all the latest search capabilities and integrations introduced in the KDB.AI Server 1.1.0 release.

v1.0.3

Release date 2024.03.13

Fixes and improvements

  • KDB.AI Cloud UI 1.0.3 includes internal improvements.

v1.0.2

Release date 2024.01.30

New

KDB.AI Cloud UI now provides three sections:

  • Overview: view Database Capacity and a Quickstart Guide with embedded cope snippets.
  • Connection Details: view instance endpoint URL and manage API keys for your environment. In addition, a Connection Guide is provided with embedded code snippets.
  • Tables: view list of Tables and schemas in your database and delete tables from the UI directly. The delete functionality only allows for deletion of complete tables, not row level data. See Delete table for details. In addition, a Tables Creation Guide is provided with embedded code snippets.

v1.0.0

Release date 2023.12.04

New

  • Introducing REST API support, including insert and train functions.

v0.2.0

Release date 2023.11.03

Fixes and improvements

  • Minor fixes and performance improvements

v0.1.1

Release date 2023.09.27

Fixes and improvements

  • Fixed issues with training functionality surrounding CPU instructions on certain machines.

  • Improved behavior surrounding synchronous API calls, including create and drop.

  • Improved responses from similarity search API by removing unneeded empty rows.

v0.1.0

Release date 2023.09.12

New

Introducing the first public release of KDB.AI Cloud UI.

  • Supported by Python clients with version less than or equal to 0.1.1

KDB.AI Server

v1.5.0

Release date 2024.11.28

Welcome to the KDB.AI Server 1.5.0 release, designed to elevate your data management and search capabilities to new heights! Key highlights include:

  1. Partitioning in KDB.AI (New)
  2. Enhanced non-transformed TSS (Improvement)
  3. Range-based similarity search for qFlat index (New)
  4. Optimized search relevance with reranking capability (New)
  5. Multithreading in KDB.AI (New)
  6. Memory-mapping parameter for qHNSW (New)

1. Partitioning in KDB.AI (New)

Manage large data volumes more effectively by partitioning tables on metadata columns!

  • Improved efficiency: Partitioning distributes the workload across multiple shards on disk, enhancing query performance and resource utilization.
  • Enhanced scalability: By partitioning data, vector databases can handle larger datasets more effectively, allowing for horizontal scaling.
  • Similarity-based partitioning: Group similar vectors together to reduce cross-shards-on-disk search times and improve query efficiency.

Tip: Learn about partitioning and how to partition data in KDB.AI. Use cases include time-series data management, geographic data segmentation, and optimizing data retrieval based on specific criteria.

2. Enhanced non-transformed TSS (Improvement)

Unlock the power of KDB.AI’s enhanced non-transformed Temporal Similarity Search:

  • Expanded numeric search: We've broadened our search capabilities to include all numerical formats (for example, reals, longs, ints, and shorts), not just floats, allowing more flexibility in your data searches.
  • Optional matching vector inclusion: Retrieve the matching vector from your search results for more detailed insights. You’re no longer limited to just the row containing the first element of the matching vector!
  • Search by unique column values: Perform searches on each distinct value of a specific column (for example, each ticker symbol), or multiple columns. This eliminates the need for multiple searches and filtering on each unique value.

Tip: Include the matching vector in search results to help your analysts make better decisions. Search by unique column values to streamline data filtering, save time and reduce financial analysis errors. Perform comprehensive queries and enhance the robustness of research analysis with the expanded numeric column search.

3. Range-based similarity search for qFlat index (New)

Discover the benefits of setting a distance threshold to improve classification accuracy.

  • Customizable queries: Use the keyword range to find all vectors within a distance you define.
  • Efficient search: Perform similarity searches for all vectors within a specified distance threshold, enhancing search efficiency across large datasets.
  • Improved relevance: Range search helps you define relevance more precisely and optimize it for a specific dataset, ensuring more accurate and meaningful results.

Tip: Range-based similarity search for qFlat indexing is particularly beneficial for classification tasks, as it helps provide more precise results compared to traditional k-nearest neighbors searches.

4. Optimized search relevance with reranking capability (New)

Unlock the power of KDB.AI’s integration with top rerankers like Cohere, Jina AI, and Voyage, designed to let you reorder query results to match your preferences!

  • Enhanced relevance: Rerankers use advanced natural language processing to grasp the intent behind queries, ensuring the most relevant search outcomes.
  • Handling complex queries: Rerankers excel at capturing nuances and context, delivering accurate results for even the most complex or ambiguous queries.
  • Optimization for specific metrics: Customize search results to meet your unique business goals by optimizing for specific metrics beyond simple relevance.

Tip: Explore reranking in KDB.AI. Enhance search relevance and personalization by learning how to integrate rerankers like Cohere, Jina AI, and Voyage AI within your KDB.AI databases.

5. Multithreading in KDB.AI (New)

Boost your application's performance with multithreading! Configure the number of THREADS each worker uses to optimize parallel execution of tasks.

  • Parallel task execution: Setting the THREADS environment variable allows tasks to be executed simultaneously across multiple threads, significantly speeding up processing times.
  • Adaptability: Easily adjust the number of threads to match your server's resources and workload requirements, ensuring optimal performance.
  • Optimized resource use: Multithreading distributes the workload more effectively, improving resource utilization and query performance.

Tip: Utilize multithreading for tasks such as qHNSW insert, qFlat/qHNSW searches across partitions, and TSS search across partitions to see significant performance improvements. For a comprehensive understanding of how multithreading works and how to optimize the THREADS variable, refer to the dedicated Multithreading page.

6. Memory-mapping parameter for qHNSW (New)

Take your data processing to the next level with the new mmapLevel parameter for qHNSW! This parameter allows you to control how vectors and node connections are accessed during computation.

  • Flexible memory access: The mmapLevel parameter offers three memory mapping options: 0 for fully in memory, 1 for vectors memory mapped and nodes in memory, and 2 for both vectors and node connections memory mapped.
  • Default setting: If not specified, mmapLevel defaults to 1, balancing memory usage and performance.
  • Enhanced control: Adjust the mmapLevel to suit your specific use case and optimize resource utilization. For example, 0 grants the fastest performance, while 2 is the slowest but conserves memory.

Tip: Use mmapLevel to fine-tune your qHNSW index settings for tasks requiring different memory and performance characteristics. For more details on configuring mmapLevel, refer to the How to use indexes - qHNSW documentation.

Documentation

Summary

We hope you appreciate the enhancements and new features in v1.5.0. Enjoy exploring all the updates! If you need help, reach out to our Support Team at support@kdb.ai or join our Slack channel.

Thanks for reading. Stay tuned for the next release!

The KDB.AI Team

v1.4.0

Release date 2024.10.21

Welcome to the KDB.AI Server 1.4.0 release, crafted to make your vector search experience faster, more consistent and powerful! Key highlights:

  1. macOS support
  2. Database layer
  3. Multiple indexes
  4. q API
  5. Version information
  6. Enhanced REST API
  7. Optimized kdb+ integration
  8. Enhanced symbol management

1. macOS support (New)

We're excited to announce that KDB.AI is now fully supported on macOS via Docker, bringing you the following benefits:

  • Enhanced Performance: Optimized for macOS to ensure smooth and efficient operation.
  • Seamless Integration: Easily integrate KDB.AI with your existing macOS workflows and applications.

Tip: macOS users can set up and leverage KDB.AI’s advanced features directly on their devices, boosting productivity and streamlining data management tasks.

2. Database layer (New)

We're introducing a new database layer above tables for better data management. Also, you can reference external database tables, such as kdb+ HDB tables.

  • Scalability: Reduces redundancy and simplifies handling large datasets, making it more scalable for enterprise use.
  • Automatic Setup: A default database is created automatically to ease the initial setup process.

Tip: Use the database layer to organize multiple tables and indexes efficiently, reducing complexity in large-scale environments.

3. Multiple indexes (New)

We're thrilled to support multiple indexes that can share the same embedding column. This means there’s no need to duplicate the embeddings. Get ready for:

  • Flexible Index Management: Create and manage multiple indexes on a single table for diverse querying needs.
  • Simultaneous Searches: Execute searches across different indexes at the same time, ideal for multimodal datasets.
  • Dimensional Experimentation: Support for indexes with different dimensions to refine search accuracy.

Tip: Leverage multiple indexes to perform hybrid searches, combining dense and sparse indexes for comprehensive results.

4. q API (New)

A fully documented public q API is here, allowing q developers to use KDB.AI’s features within their q environment, empowering you with:

  • Reduced Friction: Provides a consistent toolset for developing advanced applications.
  • Enhanced Capabilities: Leverage KDB.AI’s power directly in q programming.

Tip: Use the q API to integrate KDB.AI’s advanced search functionalities into your existing q-based applications effortlessly.

5. Version information (New)

Quickly access KDB.AI server version information for compatibility checks in q API, Python API, and REST API.

  • Deployment Management: Helps ensure consistency across different environments.
  • Simplified Troubleshooting: Facilitates easier management of deployments.

Tip: Regularly check version information to maintain compatibility and streamline troubleshooting.

6. Enhanced REST API (Improvement)

1.4.0 brings improved adherence to RESTful conventions for a more user-friendly experience:

  • Consistent Error Handling: Better error management and troubleshooting tips for reliable application development.
  • Developer-Friendly: Enhances the overall developer experience with more intuitive API interactions.

Tip: Use the enhanced REST API to build robust applications with consistent error handling and improved debuggability.

7. Optimized kdb+ integration (Improvement)

  • Direct Access: Query kdb+ tables directly from KDB.AI, maintaining data integrity while utilizing advanced search features.
  • Time Series Analysis: Run Time Series Similarity (TSS) searches on kdb+ tables for insightful decision-making.
  • Optimized Indexing: Create indexes on kdb+ data within KDB.AI without altering original tables.

Tip: Use direct table access to seamlessly integrate kdb+ data with KDB.AI’s search capabilities, streamlining your data analysis workflows.

8. Enhanced symbol management (Improvement)

Symbol atoms are stored in symfiles, which are used to enumerate symbols in splayed or partitioned databases. Using symbols instead of strings can cause inefficiencies, especially when symfiles become large due to numerous non-distinct values. Using Python bytes objects to avoid too many symbols helps, providing:

  • Optimized Storage: Helps in reducing symfile size and improving performance.
  • Performance Management: Prevents degradation from large, non-distinct symbol values.
  • Effective Data Handling: Ensures efficient data processing and retrieval.

Tip: Use strings instead of symbols for large numbers of distinct values to keep symfile sizes manageable. Symbols are effective when you have many repeated values. If each value occurs only once, use strings instead. For more details, refer to the Kx documentation.

Summary

We hope you enjoy the v1.4.0 enhancements and new capabilities. Feel free to explore these updates. If you need assistance, reach out to our Support Team at support@kdb.ai or join our Slack channel.

For a complete list of past features and improvements, visit the previous KDB.AI Server release notes.

Thanks for reading. Stay tuned for our next release!

The KDB.AI Team

v1.3.0

Release date 2024.09.19

Requires Python client >= 1.3.0

Welcome to the KDB.AI Server 1.3.0 release! Key highlights:

  1. Fuzzy filtering on metadata
  2. On-disk index: qHNSW
  3. Early Access program
  4. Data migration
  5. Integration with Unstructured.io

1. Fuzzy filtering on metadata (New)

We’ve added support for fuzzy filtering on metadata, allowing for more flexible and approximate matching in queries. This enhancement significantly improves search relevance and result accuracy, making it easier to find what you’re looking for even with imperfect data.

Fuzzy filters are the ultimate game changers when dealing with data that might include errors, typos, or variations. For instance, applying fuzzy filters to searches allows you to locate documents with terms similar to your query, even if they aren’t an exact match.

Tip

Fuzzy filters have a wide range of applications across various domains. Key use cases include spell checking, data cleaning, autocomplete and suggestions, information retrieval, product matching in e-commerce, name matching in databases, geographic search, and code search.

Discover the concepts behind fuzzy filtering and head to our How to use fuzzy filters section to learn how to apply this method to your searches.

2. On-disk index: qHNSW (New)

Introducing the qHNSW index, a high-performance Hierarchical Navigable Small Worlds (HNSW) index that you can store and access directly on disk. This means you can use large indexes without the need for equivalent amounts of RAM, optimizing performance and resource utilization.

When stored on disk, the qHNSW index maintains its hierarchical graph structure, where vertices are connected based on their distances. This setup is crucial for enabling quick and efficient traversal through the graph during searches.

Tip

qHNSW can be extremely handy in several use cases such as web searches, e-commerce, Content-Based Image Retrieval (CBIR), genomic data search, protein structure search, geospatial applications, and financial fraud detection.

Read more about the differences between in-memory vs. on-disk indexes, then learn how to use the qHNSW index.

3. Early Acesss for KDB.AI integration with kdb+ (New)

We're thrilled to unveil an Early Access program for KDB.AI integration with kdb+. Participants will be provided with code samples to facilitate an understanding of how to set up this integration. If you're keen on joining, please send us an e-mail to obtain early insights and contribute to the development journey.

More about kdb+/q

kdb+ is a powerful column-based relational time series database (TSDB) with in-memory (IMDB) abilities. Built on top of the q programming language, kdb+ allows to operate directly on data. kdb+ is widely used in High-Frequency Trading (HFT) for storing, analyzing, processing, and retrieving large datasets at high speed.

4. Data migration (Improvement)

For users of previous versions such as 1.1.0 or 1.2.0 who wish to retain their existing data, please reach out to KX for detailed migration instructions. Our team is ready to assist you in ensuring a smooth transition to the latest version. Feel free to reach out if you have any questions or need further assistance!

5. Integration with Unstructured.io (New)

KDB.AI is now a destination connector on Unstructured.io, a popular ETL (Extract, Transform, Load) platform for unstructured data. This makes it much easier to ingest various complex documents into your KDB.AI vector database.

The benefits of this integration are:

  • Supports formats like plain text (TXT, XML), documents (CSV, HTML, PDF, and PPTX) and images (PNG, JPG).
  • Simplifies and speeds up the efficient transformation of unstructured data into structured formats.
  • More accurate LLM-generated RAG responses based on the ingestion of semantically relevant data into your vector database.

Tip

Rely on this integration to process and analyze unstructured data effectively and achieve effortless document extraction. Get ready to improve recommendation systems and chatbots by seamlessly handling both structured and unstructured data.

To see the integration at work, go to the Unstructured integration page and follow the steps.

Summary

We hope you enjoy the v1.3.0 enhancements, new index and integrations. Feel free to explore these updates. If you need assistance, reach out to our Support Team at support@kdb.ai or join our Slack channel.

For a complete list of past features and improvements, visit the previous KDB.AI Server release notes.

Download the latest KDB.AI Server version!

Thanks for reading. Stay tuned for our next release!

The KDB.AI Team

v1.2.4 release notes

Release date 2024.08.15

Fixes and improvements

  • This update includes several bug fixes and enhancements to improve KDB.AI performance and stability.
  • v1.2.2 and v1.2.3 introduced a series of internal changes that significantly enhance KDB.AI reliability and performance.

v1.2.0

Release Date 2024.06.27

Welcome to the KDB.AI Server 1.2.0 release! Let's look at the highlights:

  1. Higher performance – launching a series of significant performance enhancements.
  2. On disk indexes: qFlat – a Flat index that's not memory bound, leading to major infrastructure savings.

Now check out what's in store for each:

1. Higher performance (Improvement)

KDB.AI Server 1.2.0 brings considerable speed-ups, demonstrating KDB.AI’s ability to help you build fast, efficient, accurate, cost-predictable, and scalable solutions for your organization’s projects.

2. On-disk indexes: qFlat (New)

Managing indexes on-disk, as opposed to in-memory, provides a low-memory solution for working with large datasets of vector embeddings. That's why the 1.2.0 release introduces an on-disk alternative to the in-memory Flat index.

Labelled qFlat, this index supports all the API create and search capabilities. The only difference is that you need to specify qFlat instead of flat if you prefer the on-disk index to the in-memory one.

Tip

Pick qFlat over Flat if you prioritize capacity over speed. Keep in mind that on-disk is slower but cost-effective, while in-memory is faster but more expensive.

Read more about the differences between in-memory vs. on-disk indexes, then learn how to use the qFlat index.

Note

Please ensure that your data explicitly matches the data type required by your table schema. Not doing this may result in a type error. The example below ensures the correct data type for the StockPrice column.

import numpy as np

table_schema = {
    "columns": [
        {"name": "StockPrice", "pytype": "float32"},
        # rest of schema
    ]}

df = pd.DataFrame()
df = ReadStockData('stock_price_data.csv')
# Ensure correct stock price type
df['StockPrice'] = [np.float32(x) for x in df.StockPrice]

Summary

We hope you'll make the best of the v1.2.0 performance improvements and the new qFlat index. Now feel free to try them out. If you need any help, email our Support Team at support@kdb.ai or join our Slack channel.

For a full list of features and improvements we rolled out in the past, check out the previous KDB.AI Server release notes.

Download the latest KDB.AI Server version!

Thanks for reading. Stay tuned for the next release!

The KDB.AI team

v1.1.0

Release date 2024.03.13

The KDB.AI Server 1.1.0 release is here! This time we focused on enhancing your KDB.AI toolset with three new searches and an integration. Here are the highlights:

1. Hybrid search – combines a sparse vector search with a dense vector search.

2. Transformed TSS – temporal similarity search for transformed data.

3. Non-transformed TSS – temporal similarity search for non-transformed data.

4. LlamaIndex integration – you can now find KDB.AI in the LlamaIndex library.

5. Azure ML integration - includes the latest OpenAI release 1.17.0 and more notebooks to run tests on.

Now let's see what's in store for each new addition:

1. Hybrid search (NEW)

KDB.AI Server 1.1.0 brings a specialized vector search that increases the result relevancy across your applications. It's called a hybrid search and it’s a mix of two powerful search methods:

  • The precision of keyword-based sparse vector search.

  • The contextual understanding of semantic dense vector search.

This means that with KDB.AI, you can now combine and re-rank the results based on a user-set alpha value that weighs the importance of each search type.

Tip

Hybrid search is particularly useful in content recommendation systems, customer support automation, legal document search or human resources and recruitment.

Discover more uses cases on our Learning hub, head to our docs to understand the concepts behind this search method, or even better, learn how to conduct a hybrid search.

2. Transformed temporal similarity search (NEW)

Looking for a compression model that dimensionally reduces time series windows by over 99%, while faithfully preserving the original data’s shape? Using the Transformed TSS method, KDB.AI can store and perform lightning-fast vector searches on time-series data.

This breakthrough enables rapid and profound analytical insights, based on the following:

  • 1% of embedding size, 10x increase in search performance.

  • Memory and disk storage can be 1/100th of the original size.

  • Embeddings are auto generated at ingestion; no need for an external model/LLM.

  • When low latency results are more important than low latency data availability.

Leveraging windowed time series data compressed with Transformed TSS on KDB.AI enables the deciphering of emerging trends.

Tip

By providing fast comparison between current data and vast historical datasets, Transformed TSS is a game changer for financial and market predictions; online advertisement optimization, or retail customer understanding.

Head to the KDB.AI Learning hub for detailed use cases, read the docs to understand the main concepts behind this search method, then learn how to perform a Transformed TSS search.

3. Non-transformed temporal similarity search (NEW)

KDB.AI Non-Transformed TSS is a groundbreaking algorithm for near real-time similarity searches across fast-moving time series data. This new feature provides:

  • A precise and efficient method to analyse patterns and trends in time series data.

  • No need to embed, extract, or store time series vectors in the database.

  • The ability to re-configure search without requiring the overhead of re-indexing.

Tip

By enabling pattern matching, outlier detection, and near real-time analysis of datasets, you can count on Non-Transformed TSS for financial market analysis, sensor monitoring, cybersecurity threat detection and prevention, and healthcare monitoring.

Explore more use cases on the KDB.AI Learning hub, read our docs to learn the key terms, and learn how to execute a Non-Transformed TSS search.

4. LlamaIndex integration (NEW)

We're thrilled to introduce the integration of KDB.AI on LlamaIndex, a platform that simplifies the storage and retrieval of public and private datasets for RAG-enabled applications. With LlamaIndex and KDB.AI, developers can easily build popular RAG-enabled applications quickly and at scale, without worrying about the technical details and complexities.

The KDB.AI and LlamaIndex integration:

  • Enables LLMs to return dynamic and timely information.

  • Helps reduce memory requirements.

  • Enhances accessibility when contrasted with processing searches via raw embeddings.

  • Augments the LLM’s native output, resulting in a more precise and contextually relevant response to the user’s question.

Tip

You can combine LlamaIndex and KDB.AI to improve variety of applications, such as: document Q&A, data-augmented chatbots, knowledge base, FAQs, workflows, procedures, structured analytics, and content generation (blogs, articles, books, etc.).

Read more details on our LlamaIndex integration page and try our Advanced RAG with temporal filters using LlamaIndex and KDB.AI vector store notebook.

5. Azure ML integration (Improvement)

The Azure Marketplace package allows you to deploy an Azure ML studio, with an integrated KDB.AI Server 1.1.0. This includes the following third party dependencies:

  • KDB.AI client 1.1.0

  • LangChain OpenAI 0.1.1

  • OpenAI 1.17.0

Discover 6 new example notebooks, together with a much richer and extended set of data you can leverage to run your notebook tests on.

Read more about the Azure setup.

Summary

We hope you’re as excited as we are about the new KDB.AI Server searches and the LlamaIndex integration. Try them out and email our Support Team at support@kdb.ai or join our Slack channel if you need any help.

Download the latest KDB.AI Server version!

Thanks for reading. Stay tuned for the next release!

The KDB.AI team

v1.0.0

Release date 2023.11.27

New

  • Introducing REST API support, including insert and train functions.

v0.2.0

Release date 2023.11.03

Fixes and improvements

  • Minor fixes and performance improvements

v0.1.2

Release date 2023.10.17

Fixes and improvements

  • For KDB.AI Server instances it is no longer required to specify a hostname with an ordinal parameter (-h). If a hostname does have an ordinal parameter then the environment variable RT_NO_ORDINAL must be set to 0.

v0.1.1

Release date 2023.09.27

Fixes and improvements

  • Fixed issues with training functionality surrounding CPU instructions on certain machines.

  • Improved behavior surrounding synchronous API calls, including create and drop.

  • Improved responses from similarity search API by removing unneeded empty rows.

v0.1.0

Release date 2023.09.12

New

Introducing the first public release of KDB.AI.

  • Supported by Python clients with version less than or equal to 0.1.1