Indexing & Querying

Query everything. Reprocess nothing.

MindooDB's append-only store and cursor-based change tracking let you build indexes that stay up to date by processing only what changed — never rescanning the entire database. The same primitive powers Virtual Views, fulltext search, external system sync, and time travel queries.

Incremental indexing: databases feed changed documents through a cursor into Virtual Views, external indexers, and time travel queries
For decision makers

Why incremental indexing matters

In end-to-end encrypted databases, the server cannot run queries — all data is ciphertext. MindooDB solves this with client-side incremental indexing: a single cursor-based API that processes only changed documents. This powers everything from hierarchical reporting views to fulltext search to compliance audits, without ever requiring a full database scan. The result is predictable performance that scales with the rate of change, not the size of your data.

One primitive, many patterns

iterateChangesSince() is the single foundation for Virtual Views, fulltext search, external system sync, and custom analytics. Learn one API, unlock all querying patterns.

Cross-boundary queries

Virtual Views span multiple databases within a tenant, across tenants, or mix local and remote data — without moving documents. Ideal for consolidated dashboards and cross-organization reporting.

Built-in time travel

The append-only store preserves every change. Query any past state, compare document evolution over time, create point-in-time database snapshots, and build complete audit trails — all from the same data.

The foundation

iterateChangesSince() — process only what changed

Every query strategy in MindooDB starts with the same primitive: a cursor-based async generator that yields documents in modification order. On the first call it walks the entire database; on subsequent calls it picks up exactly where it left off. Deleted documents are included with a deletion marker, so downstream indexes can clean up. The cost of each run is proportional to the number of changes since the last cursor — not the total database size.

How it works
  • Cursor-based — pass null for initial scan, then the last returned cursor for incremental updates
  • Modification order — documents yielded oldest-change-first, so indexes see consistent progression
  • Deletion-aware — deleted documents appear with isDeleted() flag for clean removal from indexes
  • Consumer-driven batching — async generator lets you stop at any point and resume later
Performance characteristics
  • O(changed) — each incremental run processes only documents modified since the last cursor
  • No full scans — the internal index tracks (lastModified, docId) for efficient cursor resumption
  • Pluggable consumers — one change stream can feed multiple indexers in a single pass
  • Works offline — indexes update locally; sync brings in remote changes, then indexers process the delta
Virtual Views

Hierarchical views with categories, sorting, and totals

Virtual Views organize documents into an in-memory tree structure — think of a dynamic table of contents that categorizes, sorts, and aggregates your data. Inspired by the proven view paradigm from HCL Notes/Domino, they support nested category hierarchies, ascending and descending sort on multiple columns, and built-in SUM and AVERAGE aggregations on category rows. Updates are incremental: when a document changes, the view updates only the affected branches.

What you get
  • Category columns — group documents into nested hierarchies (e.g. Department > Year > Quarter)
  • Sorted columns — sort entries within each category by one or more fields
  • Total columns — automatic SUM or AVERAGE per category, updated incrementally
  • Display columns — additional data shown alongside each entry
  • Value functions — compute column values dynamically from document data
Navigation and access control
  • Expand/collapse — drill into categories or collapse them, like a file explorer
  • Position-based navigation — jump to "1.2.3" (first category, second sub-category, third entry)
  • Selection — select individual entries or entire categories for batch operations
  • Access control callbacks — filter visible entries per user without changing the view structure
  • Forward and backward iteration — traverse the tree in either direction
Example output: categorized employee view with salary totals
Engineering (Total: $260,000)
Johnson, Alice — $130,000
Smith, Bob — $130,000
Sales (Total: $200,000)
Brown, Charlie — $100,000
Williams, Diana — $100,000
Cross-boundary queries

One view, many databases — even across tenants

One of Virtual Views' most powerful features is the ability to combine documents from multiple MindooDB instances into a single, unified view. Each data source is identified by an "origin" string, so you always know which database (and which tenant) a document came from. This makes it straightforward to build consolidated dashboards, cross-regional reports, and inter-organization analytics — without moving or duplicating data.

Multi-database views

Combine a US products database and an EU products database into a single product catalog. The view categorizes and sorts across both sources. Each entry carries its origin, so your UI can show the source region.

Multi-tenant views

Span views across different MindooTenants for cross-organization reporting. Two organizations share data in a third tenant; a consolidated view aggregates revenue across all three — each tenant retains independent admin control.

Incremental across all sources

Each data provider tracks its own cursor independently. Calling view.update() processes only documents that changed in each source since the last run. You can also update a single origin with view.updateOrigin("us-products").

External system integration

Push incremental updates to any indexer or pipeline

The same iterateChangesSince() primitive that powers Virtual Views can feed any external system. Use it to keep a fulltext search index in sync, push changes to an analytics pipeline, or replicate data to an external service. Because the cursor tracks exactly which documents have been processed, your integration never misses a change and never reprocesses data unnecessarily.

Fulltext search integration

Client-side fulltext search is the natural complement to end-to-end encryption. Documents are decrypted locally, then fed into a search index that runs entirely on the client — no plaintext ever reaches the server.

  • FlexSearch — high-performance, memory-efficient, incremental
  • MiniSearch — lightweight alternative with fuzzy matching
  • Lunr.js — client-side search with language support
  • Custom indexers — any system that supports add/update/remove
Custom pipelines

Build data pipelines that react to document changes. The async generator pattern means you can process changes at your own pace, with backpressure built in.

  • Analytics feeds — push aggregated metrics to dashboards
  • Webhook triggers — notify external systems when specific documents change
  • ETL pipelines — extract and transform data for reporting systems
  • Replication — mirror data to secondary systems with exactly-once semantics
Index orchestration

An index manager can coordinate multiple indexers from a single change stream. Process each changed document once, and fan out updates to all registered indexes — Virtual Views, fulltext search, analytics, and custom consumers — in a single pass. Each indexer tracks its own state, so adding or rebuilding one indexer doesn't affect the others.

Time travel

Query the past, compare changes, build audit trails

MindooDB's append-only store preserves every document change. This enables three powerful capabilities: retrieving a document at any historical timestamp, traversing the complete change history of a document, and listing all documents that existed at a specific point in time. Together, they support compliance audits, version comparison, undo/redo, and point-in-time database snapshots.

Point-in-time queries
  • getDocumentAtTimestamp() — retrieve a document snapshot at any past timestamp, applying all changes up to that moment
  • getAllDocumentIdsAtTimestamp() — efficiently get all document IDs that existed at a specific time, without loading content
  • Deleted document handling — distinguishes between "didn't exist yet" (returns null) and "was deleted" (returns doc with isDeleted() flag)
History traversal
  • iterateDocumentHistory() — walk through every change from creation to current state, in chronological order
  • Author metadata — each change includes timestamp and the public signing key of the user who made it
  • Independent clones — each yielded document version is an independent snapshot, safe to store and compare
  • Change detection — only yields when the document actually changed (skips no-op updates)
Point-in-time snapshots

Create separate database instances synced to any date

Because MindooDB's sync protocol transfers individual change entries with timestamps, you can create a new database instance and sync it only up to a specific point in time. This gives you a frozen snapshot of the database at that moment — useful for regulatory audits, reproducible analytics, or comparing the state of your data across time periods. The append-only store ensures the snapshot is complete and tamperproof.

Regulatory snapshots

Create a frozen copy of your database at the end of each fiscal quarter. Auditors can independently verify document states, authorship signatures, and change history — all cryptographically provable.

Diff across time

Compare document sets at two timestamps to find what was created, modified, or deleted in between. Combine with getDocumentAtTimestamp() to see exactly how individual documents evolved.

Reproducible analytics

Run the same Virtual View or query against database snapshots at different dates. Compare results month-over-month or quarter-over-quarter without maintaining separate reporting databases.

Code examples

Copy-pasteable patterns

These snippets are derived from the MindooDB documentation and test suite to stay aligned with real usage patterns.

Architecture fit

How incremental indexing fits the MindooDB architecture

Why client-side indexing?

In MindooDB, document content is end-to-end encrypted. The server stores only ciphertext and cannot execute queries. This is a deliberate security tradeoff: confidentiality over server-side convenience. Client-side indexing restores the querying capabilities you expect — with the guarantee that your data is never exposed to the server.

The incremental approach keeps this practical at scale. Instead of rebuilding indexes from scratch after each sync, the cursor picks up exactly where it left off, processing only the delta. For a database with 100,000 documents where 50 changed since the last sync, the indexer processes 50 documents — not 100,000.

The append-only advantage

MindooDB's append-only store is what makes all these patterns possible. Because changes are never overwritten:

  • Cursors are stable — the modification order doesn't change, so resuming from a cursor is always consistent
  • Time travel is free — the full history is already stored; no additional logging or snapshots needed
  • Audit trails are built in — every change is signed and timestamped by its author
  • Incremental updates are correct — the change stream is complete and ordered; no missed updates

This stands in contrast to mutable databases where implementing change tracking, history, and incremental indexing requires additional infrastructure (WAL, CDC, change streams, audit tables).