Architecture

This page describes the storage, search, embedding, and SDK layers of Vecnest.

---

Overview

Vecnest is a browser-based vector database. Vectors are stored in IndexedDB and queried via cosine similarity. HNSW (Hierarchical Navigable Small World) indexing provides approximate nearest neighbor search; a brute-force path is used when HNSW is unavailable or disabled.

---

Core components

Vecnest
├── Storage Layer (IndexedDB)
├── Search Layer (HNSW + brute-force fallback + Web Workers)
├── Embedding Layer (Transformers.js / optional)
└── SDK (JS API + React hooks)

---

1. Storage layer (IndexedDB)

• Database name: Configurable (default vecnest). Set via VecnestDB constructor or openDB(name).
• Object store: vectors; keyPath id (auto-increment).
• Record shape: { id, vector: number[], metadata: object, created: number }.
• Indexes: created for ordering (e.g. list newest first).
• HNSW index store: hnsw_index stores serialized HNSW graph data per dimension.
• Persistence: Same-origin IndexedDB. Data survives reloads; cleared when the user clears site data.

Vectors store (conceptual)

erDiagram
  vectors {
    int id PK
    string vector
    string metadata
    int created
  }

(In practice, vector is stored as a number[] and metadata as a plain object; the schema indicates logical types.)

---

2. Search layer

• HNSW (primary): Approximate nearest neighbor via hnswlib-wasm (WebAssembly). O(log n) query time. Used when the index exists and is valid.
• Brute-force (fallback): Full cosine-similarity scan over all vectors when HNSW is unavailable or disabled. O(n).
• Web Workers: Brute-force search can run in a background thread (useWorker: true) to keep the UI responsive.
• Metadata filtering: Filters applied after retrieval using operators $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin. See Metadata filtering.
• Index rebuilding: HNSW is rebuilt asynchronously after updates and deletes. You can also call rebuildIndex manually.

---

3. Embedding layer (Transformers.js)

• Library: @huggingface/transformers (Transformers.js).
• Usage: Optional. Use addText / searchText for built-in embeddings, or supply your own vectors via insert and search.
• Model: Default is typically Xenova/all-MiniLM-L6-v2 (384 dimensions). Fetched on first use and cached.
• Runtime: WebAssembly in the browser; no server required.

---

4. SDK design

Low-level (core) API

• openDB, insert, insertBatch, search, update, remove, count, list, getStorageEstimate, cosineSimilarity, embed, embedBatch, rebuildHnswIndex.
• Use when you need direct control over transactions or storage.

High-level API (VecnestDB)

• Wraps the core API: connect, insert, addText, addTexts, search, searchText, update, delete, count, list, rebuildIndex, close.
• addText / addTexts: embed via Transformers.js, then insert (HNSW updated).
• searchText: embed query, then search.
• React hooks: useVecnest, useSearch, useSearchText in vecnest/react.

---

Data flow

flowchart LR
  subgraph Input
    Text[Text]
    Vectors[Vectors]
  end

  subgraph Embedding
    HF[Transformers.js]
  end

  subgraph Store
    IDB[(IndexedDB)]
  end

  subgraph Search
    Cos[Cosine Similarity]
    HNSW[HNSW]
  end

  Text --> HF
  HF --> Vectors
  Vectors --> IDB
  IDB --> Cos
  Cos --> HNSW
  HNSW --> Results[Results]

---

Technology stack

Layer	Choice	Notes
Storage	IndexedDB	Persistent; large per-origin quotas
Embeddings	Transformers.js	WASM; local; optional
Search	HNSW (WASM)	hnswlib-wasm for approximate NN
Workers	Web Workers	Background brute-force when needed
SDK	JS (ESM)	React hooks optional
Build	Vite	Examples and tooling