Query everything. Reprocess nothing.
MindooDB's append-only store and cursor-based change tracking let you build indexes that stay up to date by processing only what changed — never rescanning the entire database. The same primitive powers Virtual Views, fulltext search, external system sync, and time travel queries.
Why incremental indexing matters
In end-to-end encrypted databases, the server cannot run queries — all data is ciphertext. MindooDB solves this with client-side incremental indexing: a single cursor-based API that processes only changed documents. This powers everything from hierarchical reporting views to fulltext search to compliance audits, without ever requiring a full database scan. The result is predictable performance that scales with the rate of change, not the size of your data.
iterateChangesSince() is the single foundation for Virtual Views, fulltext search, external system sync, and custom analytics. Learn one API, unlock all querying patterns.
Virtual Views span multiple databases within a tenant, across tenants, or mix local and remote data — without moving documents. Ideal for consolidated dashboards and cross-organization reporting.
The append-only store preserves every change. Query any past state, compare document evolution over time, create point-in-time database snapshots, and build complete audit trails — all from the same data.
iterateChangesSince() — process only what changed
Every query strategy in MindooDB starts with the same primitive: a cursor-based async generator that yields documents in modification order. On the first call it walks the entire database; on subsequent calls it picks up exactly where it left off. Deleted documents are included with a deletion marker, so downstream indexes can clean up. The cost of each run is proportional to the number of changes since the last cursor — not the total database size.
- Cursor-based — pass
nullfor initial scan, then the last returned cursor for incremental updates - Modification order — documents yielded oldest-change-first, so indexes see consistent progression
- Deletion-aware — deleted documents appear with
isDeleted()flag for clean removal from indexes - Consumer-driven batching — async generator lets you stop at any point and resume later
- O(changed) — each incremental run processes only documents modified since the last cursor
- No full scans — the internal index tracks
(lastModified, docId)for efficient cursor resumption - Pluggable consumers — one change stream can feed multiple indexers in a single pass
- Works offline — indexes update locally; sync brings in remote changes, then indexers process the delta
Hierarchical views with categories, sorting, and totals
Virtual Views organize documents into an in-memory tree structure — think of a dynamic table of contents that categorizes, sorts, and aggregates your data. Inspired by the proven view paradigm from HCL Notes/Domino, they support nested category hierarchies, ascending and descending sort on multiple columns, and built-in SUM and AVERAGE aggregations on category rows. Updates are incremental: when a document changes, the view updates only the affected branches.
- Category columns — group documents into nested hierarchies (e.g. Department > Year > Quarter)
- Sorted columns — sort entries within each category by one or more fields
- Total columns — automatic SUM or AVERAGE per category, updated incrementally
- Display columns — additional data shown alongside each entry
- Value functions — compute column values dynamically from document data
- Expand/collapse — drill into categories or collapse them, like a file explorer
- Position-based navigation — jump to "1.2.3" (first category, second sub-category, third entry)
- Selection — select individual entries or entire categories for batch operations
- Access control callbacks — filter visible entries per user without changing the view structure
- Forward and backward iteration — traverse the tree in either direction
One view, many databases — even across tenants
One of Virtual Views' most powerful features is the ability to combine documents from multiple MindooDB instances into a single, unified view. Each data source is identified by an "origin" string, so you always know which database (and which tenant) a document came from. This makes it straightforward to build consolidated dashboards, cross-regional reports, and inter-organization analytics — without moving or duplicating data.
Combine a US products database and an EU products database into a single product catalog. The view categorizes and sorts across both sources. Each entry carries its origin, so your UI can show the source region.
Span views across different MindooTenants for cross-organization reporting. Two organizations share data in a third tenant; a consolidated view aggregates revenue across all three — each tenant retains independent admin control.
Each data provider tracks its own cursor independently. Calling view.update() processes only documents that changed in each source since the last run. You can also update a single origin with view.updateOrigin("us-products").
Push incremental updates to any indexer or pipeline
The same iterateChangesSince() primitive that powers Virtual Views can feed any external system. Use it to keep a fulltext search index in sync, push changes to an analytics pipeline, or replicate data to an external service. Because the cursor tracks exactly which documents have been processed, your integration never misses a change and never reprocesses data unnecessarily.
Client-side fulltext search is the natural complement to end-to-end encryption. Documents are decrypted locally, then fed into a search index that runs entirely on the client — no plaintext ever reaches the server.
- FlexSearch — high-performance, memory-efficient, incremental
- MiniSearch — lightweight alternative with fuzzy matching
- Lunr.js — client-side search with language support
- Custom indexers — any system that supports add/update/remove
Build data pipelines that react to document changes. The async generator pattern means you can process changes at your own pace, with backpressure built in.
- Analytics feeds — push aggregated metrics to dashboards
- Webhook triggers — notify external systems when specific documents change
- ETL pipelines — extract and transform data for reporting systems
- Replication — mirror data to secondary systems with exactly-once semantics
An index manager can coordinate multiple indexers from a single change stream. Process each changed document once, and fan out updates to all registered indexes — Virtual Views, fulltext search, analytics, and custom consumers — in a single pass. Each indexer tracks its own state, so adding or rebuilding one indexer doesn't affect the others.
Query the past, compare changes, build audit trails
MindooDB's append-only store preserves every document change. This enables three powerful capabilities: retrieving a document at any historical timestamp, traversing the complete change history of a document, and listing all documents that existed at a specific point in time. Together, they support compliance audits, version comparison, undo/redo, and point-in-time database snapshots.
getDocumentAtTimestamp()— retrieve a document snapshot at any past timestamp, applying all changes up to that momentgetAllDocumentIdsAtTimestamp()— efficiently get all document IDs that existed at a specific time, without loading content- Deleted document handling — distinguishes between "didn't exist yet" (returns
null) and "was deleted" (returns doc withisDeleted()flag)
iterateDocumentHistory()— walk through every change from creation to current state, in chronological order- Author metadata — each change includes timestamp and the public signing key of the user who made it
- Independent clones — each yielded document version is an independent snapshot, safe to store and compare
- Change detection — only yields when the document actually changed (skips no-op updates)
Create separate database instances synced to any date
Because MindooDB's sync protocol transfers individual change entries with timestamps, you can create a new database instance and sync it only up to a specific point in time. This gives you a frozen snapshot of the database at that moment — useful for regulatory audits, reproducible analytics, or comparing the state of your data across time periods. The append-only store ensures the snapshot is complete and tamperproof.
Create a frozen copy of your database at the end of each fiscal quarter. Auditors can independently verify document states, authorship signatures, and change history — all cryptographically provable.
Compare document sets at two timestamps to find what was created, modified, or deleted in between. Combine with getDocumentAtTimestamp() to see exactly how individual documents evolved.
Run the same Virtual View or query against database snapshots at different dates. Compare results month-over-month or quarter-over-quarter without maintaining separate reporting databases.
Copy-pasteable patterns
These snippets are derived from the MindooDB documentation and test suite to stay aligned with real usage patterns.
How incremental indexing fits the MindooDB architecture
In MindooDB, document content is end-to-end encrypted. The server stores only ciphertext and cannot execute queries. This is a deliberate security tradeoff: confidentiality over server-side convenience. Client-side indexing restores the querying capabilities you expect — with the guarantee that your data is never exposed to the server.
The incremental approach keeps this practical at scale. Instead of rebuilding indexes from scratch after each sync, the cursor picks up exactly where it left off, processing only the delta. For a database with 100,000 documents where 50 changed since the last sync, the indexer processes 50 documents — not 100,000.
MindooDB's append-only store is what makes all these patterns possible. Because changes are never overwritten:
- Cursors are stable — the modification order doesn't change, so resuming from a cursor is always consistent
- Time travel is free — the full history is already stored; no additional logging or snapshots needed
- Audit trails are built in — every change is signed and timestamped by its author
- Incremental updates are correct — the change stream is complete and ordered; no missed updates
This stands in contrast to mutable databases where implementing change tracking, history, and incremental indexing requires additional infrastructure (WAL, CDC, change streams, audit tables).