SINDI_V2

SINDI_V2 is VSAG’s term-first sparse-vector index. It uses the same windowed inverted-list scoring model and 1 - inner_product distance as SINDI, but persists postings by term so a search can read only the lists needed by the query.

Factory entry: sindi_v2
Source: src/algorithm/sindi_v2/
Required dtype: sparse
Required metric_type: ip

Storage layout

Each term payload stores only non-empty windows:

non_empty_window_count
ordered {window_id, posting_count}[]
local_doc_ids[]
alignment padding
encoded_values[]

At query time, SINDI_V2 expands the sparse metadata into window_count + 1 window offsets. A query loads postings only for its retained terms through reader_io, mmap_io, buffer_io, async_io, or an in-memory backend.

This differs from SINDI’s window-first serialization. The two factory entries therefore require different persisted formats and must not be used interchangeably when loading an index.

Quick start

#include <vsag/vsag.h>

std::string params = R"({
    "dtype": "sparse",
    "metric_type": "ip",
    "dim": 1024,
    "index_param": {
        "term_id_limit": 30000,
        "window_size": 50000,
        "doc_prune_ratio": 0.1,
        "use_quantization": true,
        "use_reorder": true,
        "remap_term_ids": true,
        "term_io": {
            "type": "buffer_io",
            "file_path": "/tmp/sindi_v2.terms"
        },
        "rerank_io": {
            "type": "block_memory_io"
        }
    }
})";
auto index = vsag::Factory::CreateIndex("sindi_v2", params).value();
index->Build(base);

auto result = index->KnnSearch(
    query, 10,
    R"({"sindi_v2": {
        "n_candidate": 100,
        "filter_callback_limit": 10000,
        "query_prune_ratio": 0.1,
        "term_prune_ratio": 0.2,
        "term_retain_threshold": 10000
    }})").value();

Build parameters

Build parameters live under index_param.

Parameter	Type	Default	Description
`dim`	int	required	Maximum number of non-zero elements per sparse vector, not the vocabulary size.
`term_id_limit`	int	`1000000`	Upper bound on term IDs; maximum 50 000 000.
`window_size`	int	`50000`	Documents per window, from 10 000 to 60 000.
`doc_prune_ratio`	float	`0.0`	Fraction of lowest-weight document terms removed during build (`[0.0, 1.0)`).
`use_quantization`	bool or string	`false`	`false` stores FP32, `true` stores SQ8, and `"fp16"` stores FP16 values.
`use_reorder`	bool	`false`	Store high-precision vectors and rerank coarse candidates.
`rerank_type`	string	`"fp32"`	Rerank storage type: `fp32`, `fp16`, or `dmq8`. FP16 stores document values in half precision and scores in FP32.
`dmq_shared_codebook_threshold`	int	`1024`	Low-frequency term threshold for the shared DMQ codebook.
`remap_term_ids`	bool	`false`	Compact sparse or widely separated external term IDs.
`avg_doc_term_length`	int	`100`	Memory-estimation hint only.
`immutable`	bool	`false`	Select the compact read-only in-memory term DataCell.
`term_io`	object	`{"type": "reader_io"}`	Backend used for term-first postings after load.
`rerank_io`	object	`{"type": "block_memory_io"}`	Backend used for rerank vectors.
`rerank_layout`	uint32	`0`	Number of leading terms in the top-terms-signature layout; `0` disables it.

File-backed term_io supports mmap_io, buffer_io, and async_io, and requires file_path. If a file-backed rerank_io omits file_path, VSAG derives it as <term_io.file_path>.rerank.

rerank_type: "fp16" requires use_reorder: true and supports the same rerank I/O and layout options as fp32. rerank_layout > 0 requires use_reorder: true. rerank_type: "dmq8" requires rerank_layout: 0 and the default block_memory_io rerank backend.

Host filtering

SINDI_V2 supports the same string host build and KNN-query metadata contract as SINDI. Both mutable and immutable indexes are supported. It uses the shared host grouping and routing component while retaining its own term-first posting storage. Host membership is enforced during posting-window search, including multiple ranges created by mutable Add(). The empty-string missing-host value, persisted internal dictionary, mutable Add() metadata rules, streaming serialization, and the KNN-only scope are the same as SINDI.

Publish-time filtering

SINDI_V2 supports the same SINDI Unix-second input, UTC-day matching, and inclusive time-range contract. The two indexes share one metadata filtering component for time-only, host-only, and combined time-and-host KNN routing. SINDI_V2 retains its term-first posting layout. Publish-time filtering supports mutable and immutable indexes with either use_reorder setting and is preserved by standard and streaming serialization; range search remains unfiltered. A mutable time-aware SINDI_V2 is build-once and follows the same initial Build() or first-Add() and later-Add() rejection rules as SINDI. Base timestamp 0 uses the same missing-time semantics: unfiltered and host-only queries include those documents, while time queries exclude them. Exact day filtering disables term-level posting pruning in each selected window, so time queries may scan more postings than host-only queries. Host-only queries on time-enabled indexes share the boundary-window behavior documented for SINDI.

Search parameters

Search parameters live under {"sindi_v2": {...}}.

Parameter	Type	Default	Description
`n_candidate`	int	`0`	Coarse candidate count; `0` uses the index default derived from `topk`.
`filter_callback_limit`	uint64	`0`	Maximum user `Filter::CheckValid` callback invocations for one filtered search. `0` disables the limit. Reaching a positive limit stops candidate and window scanning after processing the final callback result and returns the candidates accepted so far, so the result may be partial. The limit applies to the regular KNN and range-search APIs.
`query_prune_ratio`	float	`0.0`	Fraction of the lowest-weight query terms skipped (`[0.0, 1.0)`).
`term_prune_ratio`	float	`0.0`	Fraction of the lowest stored values skipped in each term list (`[0.0, 1.0)`).
`term_retain_threshold`	uint64	`0`	Maximum postings for one term across all windows. `0` disables the limit; positive values allow each non-empty window posting list to scan at most `max(1, floor(threshold / window_count))` postings.

After combining the ratio and threshold limits, SINDI V2 scans at least one posting from every non-empty term list.

SINDI V2 chooses the heap-insertion strategy automatically from the build-time doc_prune_ratio and search-time query_prune_ratio. With the current 0.1 threshold, it uses distance-array insertion when both ratios are <= 0.1; if either ratio is greater than 0.1, it uses term-list heap insertion.

Choosing between SINDI and SINDI_V2

Use SINDI when the full sparse index remains in memory and window-first serialization is appropriate. Use SINDI_V2 when persisted postings should be organized by term or when searches should fetch only the query’s term lists from external storage.

Keyboard shortcuts

VSAG Documentation