The docenta handbook

How to read this#

Commands in this handbook are written for macOS and Linux shells. Windows notes appear where the two differ. Anything you type is shown like this:

docenta status --content

and the output that proves it worked is shown right after it.

1. What docenta is#

Docenta is a private search-and-memory engine for everything you own: every file on every drive, decades of email, and the conversations you have had with AI assistants. It runs entirely on your own machine. It has no network code at all, so nothing can leave, even by accident.

Classic file search answers *"which files exist and where"*. Docenta answers *"which files contain this"*, and every answer comes with a citation that points at the exact passage it came from.

The one design idea#

Originals are never copied. Docenta reads your files and stores only *derived* searchable material: compressed text sidecars, a full-text index, and a small database mapping content back to paths. Your files stay exactly where they are, untouched. Everything docenta stores can be deleted and rebuilt from your files at any time.

Identity is by content, not by path. Ten copies of the same document across ten drives cost one extraction and one index entry, and moving a file is a database row update rather than a re-index.

Who it is for#

The primary user is not a person typing in a search box: it is an AI agent working on your behalf. Docenta speaks MCP (Model Context Protocol), so Claude Code, Claude Desktop, or any MCP-capable local model can search everything you have indexed and quote it with proof. Everything the agent can do, you can also do yourself from the command line: the two surfaces return the same data.

What it is not#

• It is not a cloud service, and it has no accounts or sync-to-server.
• It is not a file manager or a backup tool. It never modifies or moves your
files.
• It is not a chatbot. It retrieves and proves; the reasoning is your agent's
job.

2. Before you start#

What you need#

The two halves you will install#

Docenta ships as two programs that work together:

1. docenta - the engine. Finds files, extracts their text, indexes it,
answers keyword searches, and serves the MCP tools to your agent. This half works with no models and no AI at all.
2. docenta-semantic - the meaning layer. Adds semantic ("what I mean,
not what I typed") search, the entity and timeline layers, and evidence packs. This half needs the ONNX runtime and the models, which onboarding fetches for you (chapter 3). You reach it as docenta semantic <verb>, docenta turbo <verb> and docenta ask: one binary name to type, the sibling runs beside it with the same output (the direct spelling docenta-semantic ... works the same and stays until the CLI arc's third phase).

You can stop after installing the first half and have a genuinely useful tool. The second half is what makes it feel like memory.

A note on time and patience#

3. Installing docenta#

docenta arrives ready to run. There is nothing to build, no compiler to install and no code to fetch: every program is compiled, signed and checked before it reaches you. Two commands do the whole job, and this chapter says exactly what each one does.

3.1 Install the programs#

macOS (Apple silicon or Intel) and Linux x64. One line in the terminal:

curl -fsSL https://dl.docenta.ai/install.sh | DOCENTA_CHANNEL=family bash

What that line does, and nothing else:

1. It downloads the current build for your machine from dl.docenta.ai.
2. It checks the download against a signed checksum list, using the docenta
release key that is built into the script. A file that was changed or swapped on the way is refused.
3. It unpacks eight binaries into ~/.docenta/bin, and this handbook, the
readme, the changelog and the licence into ~/.docenta/doc.
4. It remembers which channel you installed from, for later updates, and
prints the next command.

It does not touch your shell configuration, asks for no password and installs nothing outside ~/.docenta.

Windows 10 or 11, x64. There is no package manager entry yet. Six lines in PowerShell put the same eight binaries in %LOCALAPPDATA%\docenta\bin and on your PATH:

$dir = "$env:LOCALAPPDATA\docenta\bin"
New-Item -Force -ItemType Directory $dir | Out-Null
Invoke-WebRequest https://dl.docenta.ai/family/windows-x64/docenta-windows-x64.zip -OutFile "$env:TEMP\docenta.zip"
Expand-Archive -Force "$env:TEMP\docenta.zip" "$env:TEMP\docenta-unpacked"
Move-Item -Force "$env:TEMP\docenta-unpacked\docenta-windows-x64\*" $dir
[Environment]::SetEnvironmentVariable("Path", "$dir;" + [Environment]::GetEnvironmentVariable("Path", "User"), "User")

Open a new PowerShell window afterwards.

3.2 Onboard: one command does the rest#

docenta setup onboard

First it asks how you want it to run: autopilot, where you answer every question now and it then runs through on its own, or step by step, where each phase is explained and confirmed by you before it starts. Either way, nothing long starts while a question is still open.

Then, in order:

1. It says where docenta will live (its store and its models) and how much
room that volume has. When that volume is small and a roomier one is mounted, it lists the roomier volumes and asks which to use; a number picks one, a path is taken as written, Enter keeps the default. The choice is recorded in the config, so nothing lands on a full system disk.
2. It fetches the ONNX Runtime and the models (about 3 GB, once).
3. It sizes your machine and your files in a read-only walk.
4. It asks the rest of its questions, every one now, before anything runs.
The same questions stand on a page in your browser beside the terminal: the terminal prints the page's address and opens it, every question on the page starts not set with its default marked, nothing is chosen for you, and every toggle you make shows in the terminal as a line and is kept in a draft at once. Answer on the page and press its button (it points at any question still open and refuses until all are answered), or press Enter in the terminal to answer the open ones there, each with its default (Enter takes it); an answer typed in the terminal shows on the page, a click on the page answers the question the terminal is on, and when the terminal took part it shows the final selection and waits for your Enter before anything starts. The "answered" tag on a question unsets it again. --cli-only keeps everything in the terminal. The questions:

• OCR - "N images and M documents would go through OCR": on runs
the text recognition over scanned PDFs and pictures in the background, and keeps running it: every update's produce step reads what arrived since, within its budget, until the answer changes; later (the default) leaves those files findable by name and metadata until you turn it on (docenta setup answers --set ocr=on).
• Mail - connect a mail account (IMAP, kept fresh hourly): on the
page, now unfolds the whole mail setup under the question - the address this machine knows pre-filled, the provider's key steps, the app password typed once into the OS credential store - and the account is connected there and then (the question's note says connected: <address>; the terminal lists it); in the terminal, now runs docenta setup mail after the first scan; later (the default) prints the one line to run when you want it.
• Agents - connect an AI agent: the page lists every agent found
on this machine (Claude Code, Codex, Cursor, ChatGPT, each with where it was found) with a tick, so now connects the ticked ones after the build (none ticked: every one found, each shown and asked as docenta setup agent does); later (the default) prints the one line. The transcript the agent sessions keep (full: every turn, tool call and result; prose: the conversation's words) is asked once here, beside the agents, for all of them; the per-agent word is docenta setup answers later.
• Mailbox archives - asked only when a .mbox file is in scope,
with the count and the size: now explodes each archive into its messages after the scan (each message searchable on its own; a big archive is hours of work) and the hourly pass does the same for an archive that appears later; later (the default) records each archive by name and docenta ingest mailbox <file> explodes one when you want it. The archive file itself is captured like any other file at the explosion (its facts and access land on its row; the messages inside carry none by design, they are members, not files).

1. It tells you how long the first build will take and how much disk it
needs, and asks for your word.
2. It builds your first store in the foreground, with a live progress line,
and prints the moment you can start asking. Right after, still in the foreground, it renders the AI agent sessions on this disk (Claude Code, Codex) into the chats scope - the same step the hourly pass runs, so they answer now and not an hour later - and says how many it rendered. Once, it also says what this disk cannot hold: sessions run on your other devices and the providers' hosted chats come in through a data export you request from the provider, fed to docenta ingest claude-ai <manifest> (claude.ai) or docenta ingest chatgpt <download> (ChatGPT). The ledger's chats row records the count.
3. It starts the semantic layers in the background.
4. It registers the hourly refresh with your system's scheduler, for the
store it is onboarding: the machine's store under the label every install carries, a named store or another data dir under a job of its own (section 7.2). The schedule : line names the store the job serves.
5. It runs the hand-offs you asked for: the agents, then the mail account.

Every answer lands in the capability ledger, which docenta setup answers prints: the semantic rows follow the word you gave (turbo and local are on, lazy is lazy, later is deferred), OCR and the archives say what you answered, and mail and agents say whether the hand-off connected anything or what to run later. A scope without an archive has no mailbox row at all.

On macOS the system asks, folder by folder, whether docenta may read. Answer once for each, or give your terminal Full Disk Access (System Settings, Privacy and Security) and the questions go away for good. Windows and Linux ask nothing.

Useful flags: --step (always confirm each phase), --cli-only (every decision in the terminal, no page in the browser), --fresh (discard an interrupted run's answers), --reset (start the store over). A script answers the questions without a terminal: --yes takes every default, --embed local|turbo|lazy|later gives the word, and --ocr on|later, --mail now|later, --agents now|later, --mbox now|later answer the four questions above (the mail and agent setups still need a window you can type in to connect anything; without one the ledger says so).

3.3 The runtime and the models#

docenta runs its AI models on your own machine, through the ONNX Runtime. Onboarding fetches both for you. Everything comes from dl.docenta.ai and from nowhere else, and every file is checked against a signed index before it is used. Nothing is downloaded while you search, and no model ever sends anything anywhere.

On a Mac with Apple Silicon the embedder runs on the GPU. Apple's GPU cannot execute the model as it ships (its input sizes vary), so the assets carry a fixed-size copy of the same weights and docenta-setup installs it on Apple Silicon only. The first embed run compiles it for your chip and macOS, which takes one to two minutes and is kept (in the variant's coreml-cache folder); every later run starts from that. Before its first vector counts, the lane is checked against the CPU on probe texts, and a machine without the variant, or where the check fails, embeds on the CPU and says why in its armed line. Measured on an M4 Max: 57 passages a second on the GPU against 25 on all sixteen CPU cores, with the CPU left free. The sparse encoder does the same with its own variant (3,131 passages a minute against 1,760), checked term by term against the CPU before it counts. DOCENTA_EMBED_ACCEL=cpu and DOCENTA_SPARSE_ACCEL=cpu keep either on the CPU.

To check what is installed, or to fetch anything that is missing, run the fetcher by hand. It is safe to run at any time and downloads only what is absent:

docenta-setup

It prints one line per item, for example runtime : onnxruntime 1.28.0 present and model : arctic-embed-m-v2.0 ... present.

Where they land:

macOS   ~/Library/Application Support/docenta/
Linux   ~/.local/share/docenta/
Windows %LOCALAPPDATA%\docenta\data\

with the runtime under onnxruntime and the models under models. docenta setup config prints the exact places on your machine. If you keep the runtime somewhere else, set ORT_DYLIB_PATH to the full path of the library file and every binary uses it.

3.4 Verify the installation#

docenta --version
docenta setup config

docenta setup config prints every location docenta uses and what decided it. Before onboarding has run, most rows say (absent); that is expected.

3.5 Staying current#

docenta setup update

That one command is the whole upkeep (docenta setup update runs the docenta-setup update binary beside it; either spelling is the same command). It follows the channel this machine was installed from (the installer records it; --channel family picks another one and records that). In order, it:

1. holds the hourly sync for the swap when a new build is due (it writes
the pause marker ~/.docenta/sync.pause and lifts it at the end; a pause you set yourself with docenta sync pause is left exactly as it was; a sync pass already running finishes, because the swap is atomic);
2. fetches the newest build, verifies its signature and digest, and swaps
every binary in place;
3. brings the models and the ONNX runtime current: every model and
accelerator variant this platform installs, the ones published since your install included, each verified the same way the install was;
4. checks the result: every installed binary runs and names the channel's
build, every model is present and current, and the installed runtime carries the platform's lane (cpu, coreml on Apple Silicon, cpu, directml on Windows - never a quiet fallback to the CPU, section 9.5);
5. migrates the scheduled jobs: every docenta job this machine's
scheduler holds (launchd, Task Scheduler, systemd) whose command still spells the pass verb an older build wrote is rewritten by the NEW binary to update --scheduled and reloaded, with its store flags, its interval or time of day, its log paths and its priority exactly as they were (docenta sync migrate is the same step by hand; --dry-run shows what it would rewrite). A job that is not docenta's is never read; a job already current is counted as such;
6. refreshes the agent configs docenta already manages (Claude Code, Codex,
Cursor), the same merge docenta setup agent makes, each written file read back and checked. A client docenta was never set up in is left alone; adding one stays docenta setup agent;
7. starts the timed pass: for every store whose job it migrated or whose
binary it swapped, ONE docenta update --scheduled --after-swap runs detached (its console goes to the store's sync.log). It waits for the setup's own hold to lift (a pause you set yourself stands, and the pass stands down), then for a pass already running on the store (the reloaded job runs at load), each wait bounded at fifteen minutes; then it runs the whole pass, records its wall time as every whole update does and, because it runs outside the job, re-registers the job with the cadence it earns (section 7.2: under thirty minutes earns hourly, longer the nightly setting). A store with a pass running when the setup looks is skipped with a line: that pass's own time is the measurement, and the next by-hand docenta update applies it. The setup does not wait for the pass; docenta sync status shows the result when it ends (measured: last pass 12 s).

It prints a short summary and one verdict:

build       : docenta 0.3.1 (aaaaaaa1) -> v0.3.2+bbbbbbb (family)
changed     : 6 binaries, 1 model
checked     : every binary runs v0.3.2+bbbbbbb, all 7 models current, lane coreml carried
jobs        : 1 rewritten to `update --scheduled` (net.docenta.sync), 1 current
agents      : claude-code, codex refreshed
measuring   : a timed pass runs detached for the machine's store (pid 4242); the schedule follows it (docenta sync status shows the result)
ready       : yes

When something fails, the last line is ready : no - followed by what failed and what to do, and the command exits non-zero. -v (or --verbose) prints every step and check as it happens. A running docenta swaps to the new build by itself; chapter 9 has the details.

docenta setup verify (docenta-setup verify) is the read-only check: the channel's signature and digest, every binary in its build runs and names it, every model present and current here, the runtime's lane carried. It installs nothing and ends with the same verdict line.

3.6 Taking it off again#

Two commands, because they undo two different things. docenta remove walks back what onboarding built on this machine, part by part, after telling you what your first build cost. docenta uninstall takes the programs themselves, the inverse of the install line. Your own files are never a part of either. Chapter 8 describes both.

4. First run: from zero to your first answer#

4.1 Ask what it will cost (optional but recommended)#

docenta setup estimate --preset repo

profile    : Full (16 cores, 64 GiB RAM, 55 GiB free, hash 1724 MiB/s)
scope      : /Users/you/code/project - 25000 files, 10 GiB (sample capped; scale estimates up)
lexical T1 : est 1 min (searchable with receipts as it streams)
semantic T2: est 25 min background

It measures your machine, classifies it (Full, Standard, Constrained, or Refuse if there is not enough disk to work safely), counts the scope by the ingest's own exclusion policy (the directories the ingest will enter and no others - the hygiene globs, the marker-qualified names, the crumbs, the gitignore chains and the store's own data; the counted row says "by the ingest policy"), reads a sample of its files through the real extraction lanes, and estimates every step. The semantic steps (sparse terms, dense chunks) are sized the way the pipeline itself counts them: a uniform draw of the scope's files is read and its units counted by the index step's own rules, the scope's largest files are read outright rather than inferred, and the rest is projected per byte and per extension - a file's units follow its text, not its being a file, so a sample's few large files never stand for every small one. Files larger than the sample reads (16 MiB) are not projected at all: a mail archive or a database dump the sample never opened is named on the screen as unmeasured rather than guessed from a neighbour's rate. The units row on the screen says what it rests on: the sample's size, how much of its text sat in its largest files, what lies beyond the sample's reach, and whether a ceiling decided (a scope cannot hold more chunks of 200 characters than its text-bearing bytes allow; when the sample's rate says more, the ceiling is quoted and the row says CAPPED). The rate lines name their lane (on cpu, on coreml): on an Intel Mac both model lanes run on the cpu, and a rate in the low hundreds a minute is that machine, not a bad day. Presets are repo (the git repository you are standing in), essentials (~/Documents), and machine (every fixed volume: / on macOS and Linux, each drive letter on Windows; everything says the same). docenta setup onboard without --root takes the machine, so the first build and the hourly refresh walk the same roots and the sizing quotes what the refresh will bring; the root row then reads / (the whole machine) or lists the volumes. --root DIR names any folder, and the hourly refresh then keeps to that folder. The operating system's own trees (/usr, /opt, /System, the system /Library, C:\Windows, C:\Program Files and their kin, section 5.8 rule 2) are not part of a machine scope; your home, every mounted drive and every other drive root are.

4.2 Create the store#

docenta store init

One time, idempotent. It creates the data directory layout and the state database. Safe to re-run - it is also how schema migrations get applied after an upgrade.

4.3 Your first ingest - start small#

docenta ingest folder ~/Documents -v

-v shows progress. What happens: docenta walks the folder, identifies each file's family, hashes its content, extracts text from the formats it knows, and indexes it. Re-running costs almost nothing because unchanged files are skipped by content.

Useful flags: --family document (only PDFs, Office files, mail), --ext pdf,eml, --full (ignore the change journal and re-walk everything), --json (machine-readable output).

4.4 Your first search#

You can search while the ingest is still running:

docenta search "something you know is in there"
docenta status --content

If you get hits, the engine half is working. This is the point where docenta is already useful.

4.5 Build the meaning layer#

docenta semantic pipeline -v

(docenta semantic ... runs docenta-semantic ... beside it; the direct spelling works the same.) This runs eight steps in dependency order, each resuming from its own cursor:

Watch it from another terminal:

docenta semantic status --watch

4.6 Now go wide#

Once the small scope worked end to end, index the machine:

docenta ingest machine          # every fixed drive, into one shared collection
docenta ingest machine --plan   # dry run: show what it would do first

ingest machine gives each root its own incremental cursor, so later runs only look at what changed - on macOS it uses FSEvents, on Windows the NTFS change journal. The cursor reads a folder renamed out of the scope as the departure of everything under it (the journal names the folder and none of its files), so those rows are held as missing at the next update, the way a file's own delete holds its row; a folder renamed in is walked. Every path the cursor reports is spelled under the root as you gave it, even where the filesystem resolves the root elsewhere (/tmp to /private/tmp on macOS). A change record the Windows journal cannot place - its parent folder no longer resolves by id - is not dropped: it is a resolve row of docenta status --waiting, and the recover step returns to it by what the record says.

4.7 Connect your agent#

claude mcp add --scope user docenta -- ~/.docenta/bin/docenta serve

Then ask your agent something only your files would know. See chapter 5.7 for other clients and the full tool list.

5. Using docenta day to day#

There are two ways to use docenta: the command line (you type, you read) and an AI agent over MCP (your agent searches on your behalf). They serve the same data. Start with the command line so you can see what the agent sees.

5.1 Keyword search: docenta search#

The workhorse. Fast, literal, and available the moment your first ingest starts, with no models involved.

docenta search "water heater invoice"

Real output from a real machine:

total 18893 hit(s) - more available (use --offset 8)
  1. 114.468  /Users/you/mail/All mail.mbox/msg-47ed12.eml  Water heater  [document, archived]
  From: ... Subject: Water heater Date: 2016-01-02 ... tankless water heaters ...
  id: blake3:47ed129d10c0ec7f165e0ec727a0788876a551359db8b8b10564f2a02f42c26b
  2.  86.027  /Users/you/mail/.../att0-water heater sticker.pdf  [document, archived]
  PLAN AHEAD! AVOID A BROKEN WATER HEATER EMERGENCY! ...
  id: blake3:df556f04e6e068e51ec584b781d187390da013d2747867994b1bbe06455237c5

Read a hit like this:

• The number (114.468) is the relevance score. Only the ordering
matters, not the absolute value.
• The path is where the file lives right now.
• The snippet is real text from the document, not a summary.
• id: blake3:... is the content id. It identifies the *content*, not the
path, and it is what you pass to fetch to read more.
• The tags in brackets tell you what you are looking at: the family
(document, text, image, ...), plus honesty flags:
• archived - this copy was found only in a backup or export, so the
values in it may be out of date.
• reference - vendored or system material (toolchains, dependency
trees), ranked slightly lower on purpose.
• conversation - an AI chat transcript; claims in it date from when the
conversation happened.
• N dup(s) / near-identical folded - identical or near-identical
copies were folded into this one row.

With --json every hit that can say so also carries source_role: what kind of source it is (governing, historical_record, derived_session, external_reference) and whether your store's [[serve.authority]] table declared it or its path only suggests it (inferred). No role means unknown, not "current". A page whose every hit is an agent session or a system or vendored file says so in its notice: your own note may hold all but one of the words.

Useful flags:

The habit that works: run several cheap, narrow searches instead of one broad search and deep paging. Docenta is built for iteration - a query costs milliseconds.

Identifiers stay whole. When no document holds every word, docenta drops words one at a time until a combination holds. A word joined by -, _ or . (live-overlaid, AAPL_100) counts as one word in that climb when it occurs whole somewhere, so it is never cut in half; only a joined word that appears nowhere whole has its parts searched apart. The NOTICE: line says which (kept live-overlaid whole).

When a search comes back empty or weak, read coverage before rewording. The page then says what the store knows about capture, from receipts only, and the first finding opens the NOTICE: line: a capture step that failed on its last run, a date window that starts after the last successful capture (the mail is not captured yet), captured documents still waiting for the index, a scope the store holds nothing of, and on a mail question the accounts the store holds (mail of any other account is simply not here). --json carries it all under freshness, with each asked scope's newest document date (the document's own date, not when it was ingested), its last ingest stamp and its capture step's last success, null when no receipt exists. No receipt means "unknown", never "behind". A sealed collection speaks only of its seal. Nothing here starts a pass; docenta update --scheduled (or the agent's refresh) does.

5.2 Reading the evidence: docenta read#

Search tells you *where*; fetch shows you *what*. Pass the content id from a hit:

docenta read blake3:df556f04e6e068e51ec584b781d187390da013d2747867994b1bbe06455237c5 \
  --around "water heater"

--around TEXT centers the reading window on the first occurrence of that text, which is almost always what you want. Other options:

--raw is how you recover an attachment out of a twenty-year-old mailbox: docenta re-derives it from its container and verifies the BLAKE3 digest before serving a single byte. If the source file changed since it was indexed, the command refuses and tells you how to fix it, rather than handing you the wrong bytes.

5.3 What is in there: status --content, dev families, status --waiting#

docenta status --content

content objects : 842434
file instances  : 2878922
by family       : text 331705, document 280326, image 223269, animated 1576, media 5534, archive 24
last ingest     : 1786485814175 (unix ms)

Content objects are distinct pieces of content; file instances are paths pointing at them. The gap between those two numbers is your duplication (here: 2.88 million paths hold 842 thousand distinct contents).

• docenta status --storage - how much disk each store uses.
• docenta status --by-drive - per-drive breakdown.
• docenta dev families - the modality ladder, with which extensions belong to
each family.
• docenta status --waiting image - files identified as a family but not yet
processed by their lane: the "pre-approved queue". Each row carries the reason it waits (no lane reads .dylib; a binary format no lane reads, over the sniff size limit (limits.sniff_max_bytes); ...), and --waiting unknown lists the files of no family at all.
• docenta setup config - every path docenta uses and *what decided it*. This is
the always-correct answer to "where does everything live".

5.4 Semantic search: meaning, not spelling#

Once chapter 4's pipeline has run, the second binary answers questions where you do not know the words used in the document.

docenta dev probe search "how do we compress sidecars"   # keyword over units
docenta dev probe dense "how do we compress sidecars"    # meaning only
docenta dev probe hybrid "how do we compress sidecars"   # all channels fused

(docenta dev probe ... is the same command; the probes are an instrument and docenta semantic --help does not list them.)

Use hybrid by default. It runs the keyword, meaning, and learned-sparse channels together and fuses them by rank agreement - when independent methods agree on a document, that agreement is the strongest signal available. Its output shows exactly which channels voted:

[ 415583] lex#4+dense#4+sparse#1 content=d4c65f12a80e (+2 same content)
[ 266483] dense#18+sparse#3 content=877554acf5aa

Read that as: the top hit was ranked 4th by keyword, 4th by meaning, and 1st by the sparse channel. A hit that only one channel found is worth more skepticism than one all three found. Agreement says where to read first, not that the document answers the question: two channels can agree on a passage from the wrong year. Over MCP, search_meaning reports this as admission.verdict: empty (nothing was served, no evidence either way), channels_agree, single_channel (read each hit on its merits) or unassessed (a layer is still building).

Common flags: --limit N (default 10), --json, and --rerank to enable the cross-encoder re-ranking stage (off by default: it was measured to cost most of the query budget without improving answers on this material).

5.5 The evidence pack: one assembled briefing#

The flagship read surface. Instead of a list of links, it assembles one answer-shaped object: the route it chose, the cited evidence, connections between items, conflicts it found, and - importantly - an explicit list of what it could not see.

docenta ask "what did we decide about the firewall design" --budget 2000

(docenta ask runs the sibling's hidden answer verb beside it with the same flags; the direct spelling works the same.)

route      : fallback (no structural signal)
budget     : 782/800 tokens [trimmed]
  blake3:d0de68d4...  1786240805  <subject line>
   > <a short excerpt of the passage, word for word>
     /path/to/the/source @1840 unit 6f2a...
  blake3:91c0e2aa...  1786100114  <subject line>
  ...
  ! no structural route: evidence is lexical/hybrid ranked
  ! budget trimmed evidence: 10 of 20 units shown, 0 excerpts left out

Up to four of the cited units carry an excerpt: a short passage copied word for word from the document, the place in the document where the question's distinctive words meet. The line below it is the locator: the source's path, the character offset the excerpt starts at, and the passage's unit id, so docenta read <id> --offset-chars <offset> (or the MCP read tool with that unit_id) opens exactly that place. The excerpt slots go first to one passage for each distinctive word of the question, then to one for each kind of source the pack cites (commits, sessions, a repository's files, mail), so two documents that each answer half of a question both show their half; a passage repeated word for word takes no second slot, but its document stays cited. The other units are listed by id, date and subject.

The lines starting with ! are the honest part: the pack tells you when its route was a guess and when the token budget forced it to leave evidence out. --budget N (default 2000) sets the token ceiling for the whole pack, excerpts included; the pack enforces it rather than overflowing your agent's context, and when it must cut, the units without an excerpt go first.

--scope SCOPE (repeatable: repo:<name>, git, chats, mail, origin:<name>, the spellings search takes) draws the evidence from those worlds only. Without it the pack applies a scope the question names itself (a repo: token, a repository named in a question about code, a path like crates/x/src/lib.rs) and says so; if that scope holds nothing, it searches everything and says that too. The pack also says what its evidence can and cannot carry, about the passages it cited and nothing more:

  ! 4 citations hold none of the question's distinctive words (...); they are listed last
  ! never seen in repo:docenta, git, chats: kafka
  ! assessment: these passages do not establish the answer: none of the 12 examined holds more than 1 of ... together. A lexical check of the cited passages, not a search of repo:docenta, git, chats: ...
  ! limit: the budget trimmed the pack: more evidence was found than is cited
  ! limit: no excerpt holds kafka: read the cited documents; the answer may use other words or lie in a document the pack does not cite
  ! dates: the question asks about in 2023; the 9 dated passages examined carry document dates from 2025 to 2026 ...

The assessment is a word check, not a judgement of truth. When the question's distinctive words meet in one cited passage it says to read that passage first, which is not proof that it answers: a note that discusses a migration you decided against holds every word of "when did we migrate". When they do not meet, these passages do not establish the answer, which is not the same as the answer being absent: a passage in other words or another language can still hold it, so try search_meaning. A limit: line names what the check could not see (a trimmed budget, documents not indexed yet or not yours to read), and a dates: line appears when the question names a year that every dated passage lies outside; those dates are the documents' own, not the dates of the events they describe. In JSON and over MCP the same is the pack's assessment object (anchors: anchors_cooccur, anchors_partial, anchors_absent or not_assessed).

A route that found an entity or a time window joins the keyword evidence instead of replacing it. The MCP evidence tool takes the same scopes.

5.6 People, timelines, and themes#

These layers are built by the pipeline in chapter 4 and answer the questions that plain search cannot.

docenta semantic graph entities --card "Acme Plumbing"     # everything about one entity
docenta semantic graph claims --timeline "Acme Plumbing"   # what happened, when, with evidence
docenta semantic graph communities --list                  # themes running through it all
docenta semantic graph modules --list                      # code modules, if you index code

(docenta-semantic graph ... is the same command.)

The entity layer is deterministic: every link traces back to the exact mentions that produced it, and you can overrule it. --accept and --reject record a human verdict that the machine will not undo.

5.7 Giving it to your AI agent (MCP)#

This is what docenta is really for. The server runs over stdio and is started by your MCP client, not by you.

Claude Code (one command, from anywhere):

claude mcp add docenta ~/.docenta/bin/docenta serve

Claude Desktop or any other MCP client - add this to the client's config file (on macOS, ~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
 "docenta": {
   "type": "stdio",
   "command": "/Users/YOU/.docenta/bin/docenta",
   "args": ["serve-mcp"],
   "env": {}
 }
  }
}

Once connected, your agent has twenty tools:

The four tools marked "writes" change the store or the session. A store several people or agents share can be served without them: docenta serve --read-only, or read_only = true under [serve] in the config, lists only the other sixteen and refuses the four by name.

Two things worth knowing:

• The tools are always registered, even before the models are installed.
If a tool needs a layer you have not built yet, it says so in plain words and names the command that builds it, rather than disappearing.
• The server re-checks on every call, so a layer you build while the
agent is running mounts by itself. No restart needed.
• Every result fits one tool call. A search, search_meaning,
locate or evidence reply is bounded to 32 KiB of JSON by default (a detailed 50-hit search measured up to 130 KB before): the server keeps concise rows - which document, where in it, what caveats - and as much detail as fits, and its reply_budget says what it cut and the exact read call that brings it back. A reply that fits is unchanged. The 32 KiB is docenta's default, chosen from measured replies and client behaviour, not a limit of the MCP protocol.
• docenta serve is self-upgrading: when you install a new binary, the
store's hub hot-swaps the brain underneath every connected session, so neither the next session nor the running ones need a config change.
• The handshake is answered at once, before the store is open. A
cold start (no hub running yet) opens the store and loads the semantic engine, which on a large store takes well over ten seconds, and Codex gives a server ten seconds to answer initialize before it gives up on it ("timed out handshaking with MCP server after 9.999999959s", field 2026-10-09). So docenta serve answers initialize and the tool list right away, from what this store's brain said on its last start (or this build's own roster the first time), and reaches the hub behind it; the first tool call waits for the store to open and is answered then, or, past 45 seconds, with "docenta is still starting (the store is opening, N s so far); retry" rather than hanging. If the brain that comes up is another build than the one the handshake named, the session is asked to list its tools again.
• A connected session is asked to re-list its tools when a new build
serves: the server declares tools.listChanged and sends notifications/tools/list_changed - from the hub to every session attached at the swap, and from the session's own shim when the binary changes while the hub has dropped the session for a day's silence, or when a reconnect lands on another build. An agent client that honours the notification picks up renamed or new tools by itself; one that does not keeps its connect-time roster until it reconnects. If an agent's toolset and its instructions disagree about the names, one status call settles it: tool_names is the date of the name generation the serving build uses (2026-09-22 for search, read, who_is, ...), and a roster that still shows content_search is the client's cache - tell it to reconnect.
• Upgrade with docenta setup update (docenta-setup update) and with nothing else. It replaces
each binary temp-then-rename, which is ATOMIC: an observer sees the whole old file or the whole new one. Never rm the binary and then cp a new one over the gap - a 31 MB copy leaves seconds where the file on disk is half-written, and anything that inspects it then (the supervisor included) sees a binary that cannot run. Measured 2026-08-18: that non-atomic pattern is what dropped two connected agent sessions during a run of rapid redeploys.

5.8 What gets indexed: the inclusion policy#

Docenta walks everything under its roots and keeps a record of every file it saw; what it does NOT record is decided by one policy, applied identically on the walker lane (macOS, Linux) and the NTFS lane (the UFFS producer on Windows). The policy is a precedence ladder, top rule wins:

1. Store roots and other accounts' homes - docenta's own data,
index and staging directories, and every sibling of your home under the users directory. Never indexed; not configurable.
2. Config exclude globs ([discovery].excludes) - the hygiene set:
.git, node_modules, package caches, a project's vendor/ tree, application bundles (**/*.app/Contents), editor extension stores (**/.vscode/extensions and its Insiders and Cursor siblings, **/JetBrains/*/plugins), browser and OS caches, trash and system reserves, an Obsidian vault's .obsidian and .trash, and the operating system's own trees, anchored at the volume root: on macOS /usr, /opt, /System, /Library (the system library; ~/Library with Mail and Messages stays in), /bin, /sbin, /private, /etc, /var, /tmp, /cores, /dev, /nix; on Linux /usr, /opt, /etc, the system parts of /var (/var/mail, /var/spool and /var/www stay in), /proc, /sys, /dev, /run, /boot, /lib, /lib64, /lib32, /bin, /sbin, /snap, /nix, /tmp, /lost+found; on Windows, on every fixed drive, \Windows, \Program Files, \Program Files (x86), \ProgramData, \Recovery, \PerfLogs, \MSOCache, \Intel, \AMD, \NVIDIA. Gitignore syntax; a bare name matches a whole directory component at any depth, **/a/b a pair, and a leading / anchors at the root of the volume being walked - /usr is the system's /usr, never a project folder named usr, and a --root inside a pruned tree still walks it. Setting the key replaces the whole list.
3. Marker-qualified directories ([discovery].qualified_excludes) -
a name that counts only with proof inside it: target holding Cargo's CACHEDIR.TAG. A customer folder named target is indexed.
4. .docentaignore crumbs - your per-directory policy, in gitignore
syntax, evaluated nearest-first and OUTRANKING the .gitignore beside it. An empty crumb hides its whole subtree (the privacy crumb); a crumb with patterns excludes what they name; and a !pattern line un-ignores something the repo's .gitignore hides - the way to keep a gitignored quarantine of design documents in the corpus:

# UltraFastFileSearch/.docentaignore
!_trash/
!_trash/**

Both lines, because git semantics judge every path component on its own: when the repo's rule is /_trash/* it names the children, so whitelisting the directory alone leaves them ignored. docenta explain on a file inside says which line decided.

1. .gitignore chains ([discovery].respect_gitignore, on by
default) - what git ignores, docenta ignores, unless a crumb or an include says otherwise. Build outputs and dependency trees stay out without any configuration. Nothing vanishes in silence: every folder or file the chain hides is recorded as an admit gap with the ignored reason, the ignore file and the pattern, and docenta status --waiting lists it. A person's .env, a notes folder or a data/exports tree that the repository ignores is admitted with [discovery].includes, gitignore-syntax patterns anchored at the ingested root that win over every .gitignore:

[discovery]
includes = [".env", "notes", "data/exports/**"]

A pattern with a literal head (data/exports/**) reaches into the ignored data folder above it, and the folder's other children stay ignored; a pattern that starts with ** (**/.env) admits the file wherever its own folder is walked. The crumb's !pattern is the per-directory spelling of the same thing. docenta explain names the include that admitted a file.

1. The NTFS name prune ([discovery].name_excludes) - file names
the Windows producer never enumerates: extension-less non-content such as .dirstamp, uevent, __data, manifest-*. Note the matcher: | alternation, case-folded, a bare literal matches as a SUBSTRING, a pattern with * or ? as the whole name - so never put lock or master there.

Whatever survives the ladder is recorded. Files docenta cannot read (binaries without an extension, formats no lane handles) keep a metadata row - the record that it looked - and are never re-read until they change.

To know what the policy decided for one file, ask:

docenta explain "/Users/me/Github/UltraFastFileSearch/_trash/docs/design.md"

path    : /Users/me/Github/UltraFastFileSearch/_trash/docs/design.md
stored  : no
verdict : included (crumb whitelist)
reason  : whitelisted by "!_trash/" in /Users/me/Github/UltraFastFileSearch/.docentaignore (the crumb outranks the .gitignore beside it)

--json gives the same as a record. The MCP check_path tool reports the same reason in not_stored_because. The rule the two enforcement points must agree on is pinned by a test that feeds every default glob through both spellings.

6. Bringing in more than files#

Files on disk are only part of your life. These surfaces bring in the rest. All of them work the same way: material is rendered into ordinary email files in a staging area called the *spool* (~/.docenta/spool), and then ingested as normal content. Nothing invents a special storage format.

6.1 AI chats#

docenta status --sources     # every source, its state, and the command that brings it in

status changes nothing. It lists each source docenta knows, for example:

SOURCE    MODE      STATE                                    COMMAND
claude    dynamic   55 sessions under ~/.claude/projects     docenta ingest claude
chatgpt   snapshot  nothing dropped yet                      docenta ingest chatgpt <export.zip>
claude-ai snapshot  no export ingested yet                   docenta ingest claude-ai <manifest.json>

Two kinds of source:

• Dynamic - Claude Code and Codex sessions are found on your machine and
need nothing from you: docenta ingest claude, docenta ingest codex. The hourly sync picks up new ones by itself.
• Snapshot - ChatGPT requires *you* to export, because docenta cannot log
in as you. On chatgpt.com: Settings, Data controls, Export data. OpenAI mails a download link that expires in 24 hours. Then:

docenta ingest chatgpt ~/Downloads/<the export>.zip

It unpacks the archive safely, remembers it by its fingerprint so a re-run costs nothing, and refuses an archive that tries to escape its folder. The [chats] transcript word applies here too: under full the model's thoughts and every tool's output (a code cell's result, a browse result) are turns of their own; prose, the default, keeps the conversation. The files in the export come along: what you uploaded (under its own name), the images it generated and your voice recordings are placed beside the conversations, so a photo you sent is read like any other photo and a PDF you uploaded is searchable. Drop a later export beside an earlier one and each conversation renders from the copy updated last; an export you unzipped yourself can be dropped as a folder.

• Snapshot, the Claude apps - the chats you have on claude.ai, in the
desktop app and on the phone live in your Claude account, not on your machine, so they come home the same way. On claude.ai: Settings, Privacy, Export data. Claude mails a small *manifest* file that names one download link per part of the export (conversations, projects, design chats, pages you published, account records). Save the manifest, then:

docenta ingest claude-ai ~/Downloads/manifest-<...>.json

For every part that is not yet beside the manifest, docenta opens its link in *your own* browser, which must be signed in to claude.ai, and waits for the file to land in the same folder. It downloads nothing itself: the links only answer a signed-in browser, and each works once. You can also point the command at a folder holding the ZIP files, or at one already unpacked. Every conversation is rendered, an edited prompt's other branch as a document of its own; so is every project with its documents, every design chat, and every page you published with all its versions.

6.2 Messenger chats#

Separate command, same machinery, different category: messenger chats are ordinary documents, NOT part of the chats scope (which holds AI agent sessions only). Search them unscoped, or by their path.

docenta ingest-chats --status
docenta ingest-chats --provider whatsapp

WhatsApp needs a local iPhone backup: create one on this Mac, materialize the WhatsApp app-group subtree (iMazing, WhatsApp-Chat-Exporter, or chat4n6 all work), and drop the subtree - or a .zip of it containing ChatStorage.sqlite - into ~/.docenta/drops/whatsapp. Docenta reads that database read-only, checks the schema before touching it, and renders one message file per chat-month. A photo, a voice note, a document, a location or a contact card is rendered as what it is (the file's name and caption, the coordinates, the card), and when the subtree carries the media files they are placed beside the chats, so a photo you were sent is read like any other. In a group chat every message names the member who sent it. For a named store (docenta --store NAME ingest whatsapp) the drop folder is NAME.docenta-drops/whatsapp beside the store, and the renders land in that store's own spool.

6.3 Mail#

Two routes, and you can use both:

1. Files you already have. Any .mbox, .eml, or .msg in a folder you
ingest is picked up, and so is a mailbox that carries no extension: a mailbox is recognised by how it opens (an envelope line, then a header line), so Thunderbird's folders (Inbox, Sent, Archives under the profile's Mail/<server>/) and an Apple Mail export (Name.mbox/mbox) count the same as a .mbox. Large mailboxes explode into one searchable item per message automatically; a mailbox over the 256 MB ceiling prints a note and waits for you to run docenta ingest mailbox explicitly. When you answered on to the archives question at onboarding, a mailbox that grows later (a Thunderbird folder, a fresh export at the same path) is exploded again on the next pass, and only its new messages are read.
2. Live IMAP. One command, once, at the keyboard:

docenta setup mail

The address this machine already knows is offered before you type: on the page as the pre-filled value (the others it knows as the field's suggestions), at the terminal as the Enter default (the others listed). Where it looks, in order: the accounts docenta already keeps up to date here, then the mail clients installed on this machine - Apple Mail (the addresses its inbox is delivered to), Thunderbird and Evolution (their identities). Never a git identity: a noreply address is no mailbox. Change it or take it.

(docenta setup mail runs the docenta-mail-setup binary installed beside it: the page and the Google sign-in are the two things in docenta that listen on an address, and they live outside the program that reads your files. A hardened install can leave that binary out.)

A page opens in your browser with four numbered steps in plain words: your address; switching on the provider's 2-Step Verification (a button opens the page); making the key on the provider's page, click by click (Gmail, Apple and Yahoo all issue one; it is a separate password for one program, and you can throw it away any time); pasting it into a box. The page is served by docenta on your own computer and only once: what you paste goes into the operating system's credential store - the macOS Keychain, the Windows Credential Manager, the Linux keyring - and nowhere else. --cli-only gives the same four steps in the terminal window, with a pause after each. Nothing secret is written to any file. It then writes the connector settings (~/.docenta/spool-state/imap.toml, which names the account and the command that reads the secret back), proves one login, and from then on every scheduled pass brings the mailbox current.

Every folder. docenta asks the server for its folders (once a day) and fetches every one: Inbox, Sent, Archive, Junk and every folder you made. On Gmail, whose "All Mail" already holds your inbox, your sent mail and every label, it fetches All Mail, Spam, Trash and Drafts; your labels arrive on each message. The login check lists the folders it will keep current. To fetch fewer, name them in imap.toml (mailboxes = ["INBOX", "Sent"]); mailboxes = ["*"] on an account means every folder again. A machine set up before this keeps the list the setup used to write (["[Gmail]/All Mail", "[Gmail]/Trash"] on Gmail, ["INBOX"] elsewhere); docenta reads exactly those lists as no setting and fetches every folder without a new setup, says so once in the pass's log, and docenta status --sources shows it on the imap row. Any other list is yours and is kept. Outlook.com is refused with the reason: Microsoft accepts no password over IMAP any more.

A Google account has two ways in, kept apart on the page: Sign in with Google comes first and is the way; the app password is folded away under it as "Use an app password instead", and a question inside a folded path is not asked. The sign-in uses docenta's own Google client, built into the binary the way every desktop program carries one; an operator who brings a client of their own stores it with docenta setup mail --google-client ID:SECRET (a Desktop-app client from the Google Cloud console with the scope https://mail.google.com/) and that one wins. A build made without a built-in client says so on the page and offers the app password until a client is stored. Every other provider has the app password only. With the button, the page shows "Sign in with Google" and docenta setup mail --google does the same in the terminal: a Google window opens, you choose the account and click Allow, and the window says you may close it. What Google hands back (a refresh token) goes into the credential store like a key, and docenta renews its access before every fetch. While the app is not yet verified by Google the window shows an "app not verified" warning; Advanced, then Go to docenta, continues. The permission is taken away at myaccount.google.com/permissions, and docenta says so at the next fetch.

The cost, before anything is fetched. The login check also sizes the account: the message count of each mailbox, and its size from the sizes of the newest 200 messages (read from the server's own list, no message is downloaded), extrapolated over the count. A time is given only from a rate this machine measured: every scheduled fetch records the bytes its body session landed and how long it took, and once at least 1 MB over at least a second has moved, the setup says "about N h over IMAP at this machine's measured rate". On a machine that never fetched mail it says no rate is measured yet and gives no time. The verdict: a history that takes more than four hours over IMAP is large (without a rate, more than 10 GB is); a large one is steered to a mailbox dump first - request a Google Takeout (Mail, mbox), bring it in with docenta ingest mailbox <file>, then docenta setup mail --anchor, so the hourly pass only tops off from the dump's newest date (the mail anchor); docenta sync pause keeps the pass from starting the whole pull meanwhile. A small one is fetched by the next hourly pass, or at once with docenta update --only imap. Onboarding's mail hand-off records the sizing and the advice in the ledger's mail row. The server's quota is not used: on Gmail it is the whole Google storage, Drive and Photos included.

The hand-off from a Google Takeout: ingest the export first (route 1), and the ingest offers this step at once, with the export's own address prefilled. The export records how far it reached (the mail anchor) and whose mail it is (the address its messages were delivered to), and the live fetch of THAT account starts from there, not from the beginning. Every other account fetches its whole history: a Takeout of one account never shortens another's. If the export's messages name no address, or you set the account up by hand, docenta setup mail --anchor (or use_mail_anchor = true in imap.toml) says the export is this account's; use_mail_anchor = false always fetches the whole history. docenta sync status says whether mail is live or a top-up is waiting.

A machine with a password manager may still point imap.toml at it by hand (password_command = "gopass show -o docenta/gmail-app-password"); the setup command leaves such an entry alone and adds its account beside it.

The fetch uses your system curl over IMAPS. It never deletes or modifies anything in your mailbox. A message whose download fails is kept on a retry list in the account's cursor and tried again on every following pass until it lands or the server no longer has it; the pass's summary names it (RETRY 1: uid 4711). A download has sixty seconds plus one second per 16 KiB of the message's size (a 100 MB attachment gets about 1 h 45 min), so a large message on a slow link is given the time it needs rather than dropped.

6.4 Git history#

docenta ingest git              # every repository under your home, now
docenta ingest git --root DIR   # every repository under DIR
docenta update --only git       # the same, as the step of the pass

Every git repository the scope's walk admits is ingested end to end: a clone of someone else's project counts the same as your own work (a repository in your pile is your data). Repositories are found at any depth; only the exclusion policy's hygiene set (caches, build output, package trees, app bundles, editor stores) and cloud-synced mounts are not entered. Bare repositories and initialized submodules are included.

Every commit renders as one item: every commit of every author on every branch and tag (not only the checked-out line), with the full message (never cut), the author and the committer with both dates, the parents, the refs that point at it, and every changed path with its added and deleted line counts. There is no cap on the commit count: the log is streamed off git, and after each repository its tips are recorded, so the first pass renders the whole history and every pass after it only the commits new since the recorded tips (a rewritten branch whose old tip is gone falls back to the whole history once; the renders are idempotent). The history lives in the chats spool under git/, one .eml per commit, so "why did we change this, and when" is a search, and search --scope git keeps to the commits.

One repository, one history. A linked work tree (git worktree add) shares its main work tree's refs, so only the main one renders. Clones of one remote share a tag (the origin URL's repository name; a repository without a remote keeps its directory name), so a commit both hold is one item and a commit only one clone holds still renders from that clone. A commit's item is written once and never rewritten: its ref decoration is a snapshot of the moment it was rendered. A render whose tag belongs to no repository any more (a removed work tree, a clone that learned its remote) is removed on the next pass and retired from the index with it; the step's summary says N orphaned renders removed. A store built before 2026-10-11 may hold such duplicates (the owner's Mac had 24,061 items for 13,304 commits); its next docenta update removes them.

Your own addresses are a facet, never a filter. The lane learns them from git config user.email and user.name (global and per repository) and from the histories themselves (every address a commit carries under a name you are known by), and records them in ~/.docenta/spool-state/git_identities.json:

{ "addresses": ["me@example.org", "me@old-job.example"],
  "names": ["My Name"], "not": [] }

Add an address by editing that file (or by setting DOCENTA_GIT_AUTHORS=a@x,b@y for one run); put one under not to keep it out for good. The addresses join your mail addresses for direction: sent, so search --scope git --direction sent is "my own commits".

A store built before 2026-10-09 holds zero commits of the repositories the old authorship rule skipped; its next docenta update renders the whole history (the summary's "came along" line says how many commits).

Into another store (a foreign history as a corpus of its own): docenta --store NAME ingest git --root DIR renders into that store's own spool beside it, never into the machine's.

6.5 OCR: making scans readable#

Images and scanned PDFs hold text no search can reach until it is extracted. Docenta does this in a separate producer so your search engine never waits on it:

# 1. run the producer (long-running; safe to stop and restart)
ORT_DYLIB_PATH="$HOME/.docenta/onnxruntime/current/lib/libonnxruntime.dylib" \
docenta-analysis \
  --analysis-root "$(docenta dev layout | grep analysis_root | cut -d'"' -f4)" \
  --state-db "$(docenta dev layout | grep data_root | cut -d'"' -f4)/state.db" \
  --models "$(docenta dev layout | grep data_root | cut -d'"' -f4)/models/ocr-ppocrv5"

# 2. fold the results into search (fast - about a minute for tens of thousands)
docenta update --only analysis

After that, a scanned invoice ranks by the words printed on it. Files that are cloud placeholders (0 bytes locally) are skipped: no bytes, no text.

6.6 Moving material between your own machines#

A bale is one archive of your rendered chats and git history, digest- manifested so tampering is detected:

docenta bale export ~/mymachine.bale.zip
docenta bale import ~/mymachine.bale.zip --ingest    # on the other machine

Import verifies every member before landing any of them; a tampered bale lands nothing. Identical files are skipped, so re-importing is free.

6.7 A whole collection in a bale, ready to use#

A bale of a store (docenta bale export, or docenta store ship for a named collection) carries everything that was expensive to make: the extracted text, the search terms, and the meaning-vectors together with the search graph built over them, packed as finished *segments*. The machine that imports it builds nothing of that again. It unpacks the segments and takes them in as they are, so a collection of millions of passages is ready in the minutes it takes to copy it, on a small old laptop as on anything else. Only the word index and the people-and-places layers are made locally, by one docenta semantic pipeline run, and that needs no graphics card.

Three things follow from a collection arriving as its own segments:

• It is called what it is. A collection keeps its own name wherever it
lands (sleep, not the computer that built it). Where and when it was cut is recorded beside the name, and docenta bale list shows both.
• It stays itself. The segments of an imported collection are *locked*:
your own store's housekeeping never merges them with your own material or with another collection. You can hold several (a sleep-research library, a law library) side by side.
• A newer bale replaces the older one of the same origin as whole
segments, without touching anything else.
• It leaves as one piece. docenta bale forget LABEL --yes removes the
collection's files and its segments at once; nothing is left to clean up.

When you export only *part* of a store (one folder, or the public part of a store that also holds private material), docenta builds fresh segments for exactly that part while exporting, so nothing that was not selected can travel along, and the importing machine still builds nothing.

7. Keeping it fresh automatically#

Ingesting once is not the point; staying current is. One command runs the whole live spine, and a second registers it with whatever scheduler this machine already has.

7.0 One pass, three stages: docenta update (new)#

docenta update runs everything that keeps a store current, in one pass and three stages, from the answers given at onboarding (the capability ledger, the lane word and its turbo consent, the recorded scope):

1. ingest: backfill (a pre-bundle store only), the scan over the
roots the store was given, recover, finish, stale, chats, imap, explode (the mailbox archives, on the word given), produce (OCR, by the ledger), analysis, enrich, promote-bodies (once per build), reindex (only when the search index predates this build's schema), expire (the absence hold's end and the per-pass gc).
2. semantic: repair (once per repair generation), the unit-index
reset only when the index predates a field, the pipeline to the feed head within the lane word and its consent.
3. tidy: docenta tidy --auto - the reclaim without the walk when
the compaction is due by the coverage threshold (gc, the semantic prune, the compaction, the catch-up, the semantic compact, the stack once more), else the semantic prune and the final layer check; always on an update you type, by threshold on a scheduled one (section 8.0). gc and the compaction alone when named.

While it runs it holds THIS store's schedule (a marker under the store's data root, released when it ends, on an error too), so the hourly job of this store stands down and no other store's job notices. It is quiet by default:

ingest     : 11 steps ran in 1 s (scan 1 s, recover 0 s, ...); 3 skipped (backfill, imap, reindex)
semantic   : 1 step ran in 0 s (pipeline 0 s); 4 skipped (repair, prune, reset-units, tail-windows)
tidy       : 1 step ran in 0 s (tidy 0 s); 2 skipped (gc, compact)
came along : content objects 20 (unchanged); file instances 20 (unchanged); sessions rendered 0 (Claude Code 0, Codex 0)
layers     : all 5 current
not counted: changed files, OCR pages, mail fetched - each step's own numbers are in the transcript (`-v` streams them)
embed      : current - nothing to embed; the next delta is measured at the line
line       : under the turbo word, or with no word on record, a delta projected under 2 h 0 min at this machine's measured rate is built here instead of shipped or asked (sparse terms, dense vectors and code vectors alike); over it the turbo word ships all three lanes inside the consent and no word asks. The line never applies under later (both lanes wait for a word) or lazy (what is asked about is embedded first). Rates: ...
done       : update complete in 2 s (transcript: /tmp/arc1/after/store/logs/sync.log)

(A frozen 20-file scratch store on 2026-10-09; the stage lines of a real store carry the minutes.)

-v streams every step's own output as the pass always has. The "came along" numbers are measured before and after the run, never read off a child's prose; what this build does not count says so.

docenta update                      # the three stages, by hand
docenta update -v                   # every step streamed
docenta update --stage semantic     # one stage alone
docenta update --only stale --only pipeline   # named steps; a step the plan skips by default runs when named
docenta update --scheduled          # what the scheduler's job runs: nothing asked, turbo only inside the consent, tidy by threshold

The step names --only takes: backfill, scan, recover, finish, stale, chats, imap, explode, produce, analysis, enrich, promote-bodies, reindex, expire, repair, prune, reset-units, pipeline, tail-windows, gc, compact, tidy. A step the plan leaves out of a default run (gc, which the expire step already ran; promote-bodies under a build that judged the bodies already; prune and tail-windows, which the pipeline's own steps cover) runs when named. A step that CANNOT run (OCR deferred at onboarding, a sibling binary not installed) is said and stays skipped.

The local line, stated, and when it applies. Under the turbo word, or with no word on record, a delta the pipeline projects under two hours at this machine's measured rate is built here instead of shipped or asked - sparse terms, dense vectors and code vectors alike. That is what the line is for: the turbo word ships only a delta OVER it, so a few thousand units never go to a GPU, and the summary prints both the line and the rates it is measured against. The line never applies under later (both lanes wait for your word; nothing embeds and nothing is encoded, however small the delta) or lazy (what you ask about is embedded first, the rest waits): those words hold the lanes until another word replaces them, an unattended run leaves them alone, and a run in a terminal offers the word as the default and asks again. --embed turbo at an onboarding whose every lane projects short says so on the spot: both lanes run here, nothing is quoted or sent, and a later delta over the line is asked for turbo again.

The scheduled pass is the same plan. docenta update --scheduled is a projection of the update's step table under the scheduled posture, so the two never drift; the scheduler's command moves to docenta update --scheduled in a later phase of the arc.

7.0.1 One page: docenta status (new)#

docenta status is one page: the store (objects, instances, families, the seal or the hold), every semantic layer with its percentage and the vectors it holds, the pending dense work with the basis of the measure and the rate it is timed at (the last embed pass of the model on this machine, or the record's probe until one has run), the code partition's own line (its pending count and lane state, or "no code model installed"), the standing turbo consent, the sync (installed, cadence, next run, the last pass, a pause or a hold), and what the next update would do, stage by stage, from a read-only plan. Nothing on it is a second implementation: the semantic lines are docenta-semantic status --json read back, the store lines the stats reader's, the plan the update's own.

sealed          : no
store      : 54 content objects, 55 file instances (text 50, document 3, image 1)
layer      : unit-index                                  75.9%
layer      : dense:arctic-embed-m-v2.0@snowflake-95c2741    75.9%  held 40
layer      : mentions                                    75.9%
layer      : entities                                   100.0%
layer      : gazetteer                                  100.0%
layers     : 3 of 5 behind
pending    : 13 units to embed (counted on the feed tail)
consent    : you consented to 0.5 GPU-hours (turbo consent, 100 texts and 100 units, 2026-10-09); 0.0 shipped on it, 0.5 left, 0.1 of allowance for growth between passes; an unattended pass ships only inside that and asks beyond it
sync       : NOT installed for the store at /tmp/arc1/after/store (`docenta sync install`); last pass 0 s ago
next update: ingest    runs scan, recover, finish, stale, chats, explode, produce, analysis, enrich, expire; skips backfill, imap, promote-bodies, reindex
next update: semantic  runs pipeline; skips repair, prune, reset-units, tail-windows
next update: tidy      runs nothing; skips gc, compact, tidy (the scheduled pass)
next update: tidy      runs tidy; skips gc, compact (a `docenta update` you run)

(A scratch store on 2026-10-09 whose word is turbo with a pack waiting for consent, so three layers sit behind. A stage both postures plan alike prints one next update line; where the scheduled pass and a docenta update you run differ, each gets a line naming its posture: the scheduled pass tidies by threshold, a docenta update you run tidies always, and its line is the one docenta update --dry-run prints.)

docenta status                 # the page
docenta status --json          # the page as one object (store, layers, pending, consent, sync, next_update, next_update_by_hand)
docenta status --content       # the full content table (`stats`)
docenta status --semantic      # the semantic layer page in its own words (`docenta semantic status`)
docenta status --sources       # every ingest source, its mode and state (the source map)
docenta status --waiting [FAMILY]   # the files awaiting their extractor lane (the pre-approved queue; --limit N rows)
docenta status --progress      # what the last long-running command is doing (the maintenance journal)
docenta status --runtime       # the ONNX runtime and its lanes (`docenta setup runtime`)

7.0.2 The answers: docenta setup answers (new)#

The questions onboarding asks up front - OCR, the mailbox archives, a mail account, the agents, the lane word - and the two choices the config holds (the chats transcript, the JSON reading) are one answer set. docenta setup answers shows it: the capability ledger with each row's why, the lane word with its record, the standing turbo consent, the chats transcript and the JSON reading. At a terminal it then asks each question again with the current answer as the default (Enter keeps it) - on a page beside the terminal when a browser can be opened, as onboarding and the mail setup do (below), at the terminal otherwise or under --cli-only; a script sees the table and nothing waits on it.

docenta setup answers                       # show; at a terminal, ask again on the page beside it (the terminal when no browser opens)
docenta setup answers --cli-only            # ask again at this terminal, never on a page
docenta setup answers --page                # insist on the page (below); --fresh discards an interrupted edit's draft
docenta setup answers --set ocr=on          # one answer without a terminal: ocr=on|later|off
docenta setup answers --set mbox=now        # mbox=now|later (the archives), mail=now|later, agents=now|later
docenta setup answers --set chats=full      # chats=prose|full, json=records|whole (the config's two choices)
docenta setup answers --set embed=turbo     # embed=local|turbo|lazy|later (the lane word, for every lane: sparse, dense and code)
docenta setup answers --reset               # forget every answer, the consent included; the next build asks fresh
docenta setup answers --json

The page behaves as the onboarding page does: every toggle on the page prints in the terminal as it is made, the page's button hands the set to the terminal, and Enter in the terminal takes over - each question is asked there with the word the page shows as the default, a typed word shows on the page, a click on the page answers the question the terminal is on - and the final list waits for your Enter before anything is recorded (q keeps it as a draft). The agents found on this machine are ticks under the agents question, an agent whose config already carries docenta ticked and marked connected (the question then reads connected, as the mail question does for an account kept up to date); now with ticks connects the ticked ones right after the record. A mail account connected on the page is recorded. The page's draft (<data dir>/answers-draft.json) survives an interruption: the next run offers the changes not yet on record once (continue, or f for fresh; --fresh decides unasked) and the draft goes when they are recorded.

It writes RECORDS only - the ledger, the lane records, the config keys - and says what applies them: the next docenta update applies this for OCR, the archives, the chats transcript and the lane word (the produce, explode, chats and pipeline steps read them); docenta setup mail and docenta setup agent for the two hand-offs it records but cannot run; docenta setup answers --set json=records|whole for the JSON files already in (new files follow the setting at the next update). Whatever the reading, not every JSON file grows vectors. Each is judged once, at extraction, by what it IS: a *sidecar* (a .json beside a photo or video with the same stem, as Google Takeout writes one per photo) has its facts folded into that file's own record - the description becomes the photo's title, the people, the place and the taken time join its searchable metadata - and grows no vectors itself; a *manifest* or lockfile by its name (package.json, vcpkg.json, tsconfig.*.json, *.lock.json) is a program's wherever it lives; the rest is read by *shape*: sentences among its string values make it prose, which is embedded, while ids, hashes, numbers, paths and timestamps make it a program's output, kept for keyword search only. Where a file sits (app state, caches, package and build trees, the system's folders) stays the last tier, and [storage] unembedded_json still sets whole folders aside by hand. The report under json names each tier with its largest folders and counts the files not judged yet (rows from before this rule; the next update judges them as it re-reads). --set embed=turbo is a word given by hand: the next pass packs and sends inside the standing consent and holds beyond it with the numbers - with no consent on record it holds, as the consent ceiling of 2026-10-09 says.

7.1 What the pass does#

docenta update --scheduled walks these in order, each step resuming from its own cursor so an idle pass costs seconds:

1. docenta-spool chats - re-render AI chats
the recorded tips (the whole history on the first pass)
2. docenta-spool imap - bring mail current (it indexes what it fetched)
3. docenta ingest machine - refresh what the store was given: the folder
onboarding was told, the machine an onboarding without --root walked (the same fixed volumes, the OS trees pruned on both passes), or the roots the ingests recorded (docenta ingest folder DIR, ingest machine --root DIR; a scan without roots records the machine). A store that recorded nothing refreshes nothing on its own, and the pass says so.
4. docenta update --only analysis - fold in analysis that arrived since last pass
5. docenta semantic pipeline - update every semantic layer

1a. docenta-spool git - every repository in scope, the commits new since

There is no separate step to index the spool: the connectors ingest their own output, and the machine-wide scan would pick it up regardless.

Each step's end is recorded beside the pass transcript, in sync-receipts.json (~/.docenta/ for the default store, <store>/logs/ for a named one): when the step last succeeded and when it last failed. A search that comes back empty reads it to tell "not captured yet" from "not there"; until a pass has written it, capture reads as unknown.

7.2 Setting it up#

docenta sync install                 # hourly, the default
docenta sync install --every 240     # a pass every four hours
docenta sync install --daily-at 03:00  # once a night (a constrained machine)
docenta sync install --print-units --user svc  # a server: systemd system units to stdout
docenta sync status                  # installed? paused? when did it last run? every job and its command
docenta sync migrate [--dry-run]     # rewrite a job an older build registered (see below)
docenta sync pause                   # maintenance window: passes refuse to run
docenta sync resume                  # and back
docenta sync uninstall               # stop it again

A job from before the verb map. The scheduled pass is docenta update --scheduled since 2026-10-09; a job registered by an older build spells the verb that build knew, and that verb is gone without an alias, so such a job would fail every hour. docenta setup update rewrites every docenta job on the machine after the binary swap (section 3.5), and docenta sync migrate is the same step by hand: the registration text is read back and the one span that holds the command is rewritten to update --scheduled with the store flags it had; the interval or the time of day, the log paths, the PATH and the priority stay byte for byte, and the job is reloaded. --dry-run prints the old and the new command line of each job it would rewrite and changes nothing; -v prints them on the real run too. sync status lists every docenta job on the machine with its command and marks one still on the old verb OLD COMMAND. A job that is not docenta's is never read.

A server under a service account. The user timer above needs a login session bus, which a service account has none of. --print-units prints the two systemd system units instead, hardened as far as a pass can bear: the account named, no new privileges, no devices, a private temp, the whole system read-only, every home hidden but the account's own, the data disks hidden but the roots the store recorded (bound read-only) and the store itself (read-write), the network shut when no mail account is configured, low priority. Paths are printed canonical, so a symlinked home names the directory it points to. Root places them under /etc/systemd/system and enables the timer; the lines at the end of the output say exactly that, and a root you add later needs its own BindReadOnlyPaths line. Two things follow from the account being the identity. The device key, the lease and the seal live in that account's ~/.docenta/license, so onboard, license and update as the service account itself (sudo -u svc -H docenta setup onboard --root /srv/vault, sudo -u svc -H docenta setup update); a lease taken by your own account does not carry over. And the pass reads what the account can read: give it read access to the vault and nothing more, and write access only to its home and the store ([storage] data_dir).

pause and resume are the two ends of a maintenance window: compact and docenta tidy --only semantic-compact take the writer lock a scan would contend for, so pause first, run them, resume. The switch is a marker file (~/.docenta/sync.pause) that the pass, the MCP trigger and status all honor. It stops what runs unattended: the scheduled pass and every step it spawns, the pipeline an ingest starts detached. A docenta semantic pipeline you start by hand is not a pass: it notes the pause and runs, so an emergency stop of the scan does not cost you the semantic refresh.

One command, three schedulers: Windows Task Scheduler, macOS launchd, and systemd user timers on Linux. Installing is idempotent, so re-running it after an upgrade re-points the schedule at the new binary.

One job per store. The schedule is registered per store, and the command decides which: docenta sync install without --store or --data-dir registers the machine's own store under the label every existing install carries (net.docenta.sync on macOS, the task docenta-sync on Windows, the user timer docenta-sync.timer on Linux); docenta --store legal sync install registers a job of its own for the named store (net.docenta.sync.legal, docenta-sync-legal), and docenta --data-dir /srv/vault sync install one for that data dir, told apart by a short stable hash of the directory. Each job's command carries the store flag it was installed with (--store legal or --data-dir /srv/vault before update --scheduled), so a pass reads only its store, and its transcript and error stream live beside that store (<data dir>/logs/ sync.log and sync.err). uninstall removes only the store's own job; status names the store its first line is about and then lists every docenta job on the machine with the store each serves. Onboarding a store into another data dir or under a name registers that store's job and never touches the machine's.

A store's own pass keeps to that store in its spool steps too. The machine's store renders agent sessions from your home into ~/.docenta/spool and fetches the mail accounts docenta setup mail connected, as it always has. Any other store renders into a spool of its own beside it (<store>.docenta-spool), ingests the renders into itself and no other, and reads the agent sessions about its scope: a store over a folder takes every Claude Code and Codex session whose working directory lies under that folder, from the session roots in your home where the agents keep them (~/.claude/projects, ~/.codex), never the sessions about anything else (a store over the whole machine reads them all). The mail step is the machine store's: another store's pass skips it and says so, and takes mail as an archive (docenta --store NAME ingest mailbox <file>).

Every pass appends to ~/.docenta/sync.log for the machine's store (a named store's or a data dir's pass appends beside that store), streaming as it goes rather than at the end - a long step's progress is only useful while it runs. The transcript rotates one generation back at 8 MB.

Pick the interval by how fresh you need mail to be, not by how much work a pass does: every step is cursor-driven, so a quiet four-hour cadence costs about what four hourly passes cost, minus three wake-ups.

The interval is measured, not assumed (2026-10-09). Onboarding installs the schedule itself and decides the cadence from the pass it just ran: the wall time of its own ingest work (the scan, the agent sessions, the archives) under thirty minutes earns the hourly schedule; longer earns the constrained setting, once a night at 03:00 local, because a pass that long is hours of the whole machine and a person on such a box is happy with any answer, not a real-time one. The decision and the measured time are printed on the schedule line (every 60 min - measured: the pass took 12 s (under 30 min earns hourly)) and recorded beside the store's transcript (~/.docenta/sync-schedule.json for the machine's store, <data dir>/logs/sync-schedule.json for any other). Every whole pass after that is a measurement too, and who ran it decides when the decision is applied: a by-hand docenta update that crosses the line in either direction re-registers the job at the cadence it earned and says so; a scheduled pass records its time (it runs under the very job it would reload) and the next docenta update, docenta sync migrate or docenta setup update applies it; and the pass docenta setup update starts detached after a binary swap (docenta update --scheduled --after-swap, section 3.5) runs with the scheduler's posture but outside the job, so it records AND re-registers

• that is how a box re-measures after a binary change, from a timed

pass after the system had a chance to catch up, never from a stale record. A pass narrowed with --only or --stage measures nothing. docenta update --dry-run opens the store read-only (the state database query-only, no migration, no lock) and leaves its bytes as they were. A cadence you set yourself with sync install --every or --daily-at is recorded as explicit and never moved by a measurement; the bare sync install is the hourly default and may be. A machine asleep at 03:00 runs the pass when it wakes. Every registration runs at low priority (launchd Background with nice 10 and low-priority I/O, systemd Nice=10 with idle I/O scheduling, Task Scheduler priority 7), so the pass never fights the person for the keyboard. sync status prints the cadence in words with the measurement it rests on (every 1 h (measured: last pass 12 s), or set by hand, never re-measured), the last pass's duration step by step, and a suggestion to go daily when a pass eats more than half its interval.

Two hazards worth knowing about, both learned the hard way and both now handled by the command itself: scheduled jobs get a 256 file-descriptor limit that is too low for a recovery pass (each binary raises its own), and an ad-hoc-signed binary makes macOS re-prompt for privacy access on every pass (the pass reports it in the transcript instead of leaving you to wonder about the popups).

7.3 Pausing without disabling anything#

Before benchmarks, big migrations, or when you want the machine quiet:

touch ~/.docenta/sync.pause     # pause: passes exit immediately
rm ~/.docenta/sync.pause        # resume

The switch is just the presence of that file, so it survives reboots and needs no privileges. A second safeguard runs inside the pipeline itself: a lock file means two pipelines can never run at once, which is what once drove this machine into swap.

7.4 Watching the build#

docenta semantic status

The first lines name the runtime and the lane every model step will run on: runtime : onnxruntime 1.24.4-dml at ... - providers cpu, directml and lane : directml available. A lane line that says MISSING means the installed runtime has no provider for this platform's GPU lane and every step runs on the CPU; docenta-setup installs the channel's runtime for the platform. docenta setup runtime (docenta-semantic runtime) prints the same two lines on their own (--json for a tool).

layer                           position        head    built
unit-index                     135101358   135101358   100.0%
dense:arctic-embed-m-v2.0      135101358   135101358   100.0%
mentions                       135101358   135101358   100.0%

Add --watch to re-render every two seconds while you watch a long build. Your AI agent sees this same information through status, which is how it can honestly say "the semantic layer is 60 percent built" instead of pretending it has everything.

Below the table, one rates line per model step says what this machine has achieved on it: dense:<model>: best 1,560/min on directml (2026-09-26), last 73/min on cpu (2026-10-05). The record (rates.json beside the cursors) keeps the best run per step, model and lane, and the most recent one; runs of fewer than 200 units or under a minute inside the model are not recorded. The rate is units over the time spent inside the model, not the wall clock, so a pass that mostly skips unchanged documents does not read as slow. Every embed and sparse pass compares itself against that best once it is past its warm-up (100 units and a minute inside the model): a run below one fifth of the machine's own best says so once, in the log and on the progress line (rate: 73 units/min is 1/21 of this box's 1,560 units/min on directml (recorded 2026-09-26)). That line was the missing one when a machine lost its GPU runtime and embedded on the cpu for a day while every step line still looked normal; the fix it points at is docenta setup verify and docenta setup update.

8. Maintenance#

Docenta is designed to need very little of this. Most upkeep happens by itself; this chapter is what to do when it does not.

8.0 Space back in one command: docenta tidy (new)#

docenta tidy is the reclaim as section 8.1 describes it - the reconcile walk of the roots the store owns, the content policy, gc, the semantic prune, the compaction, the semantic catch-up, the semantic compact - and then two things the door rebuilds of 2026-10-08 taught: the semantic stack runs ONCE MORE after the compaction (the pipeline to the feed head, so the run ends current by the pipeline's own word), and the final check prints every layer's state with the vectors each partition holds.

The two breakers stay. gc refuses a pass that would collect an implausible share of the store (the mass-orphan limit: from 10,000 orphans at over 5 percent), and the semantic layers refuse a removal wave of the same shape. One flag is the reviewed acceptance of BOTH:

docenta tidy                       # the whole sequence; a wave stops it with the numbers
docenta tidy --force-wave          # you have checked the files are really gone (or really excluded): both breakers accept
docenta tidy --only gc             # one piece alone: gc | compact | semantic-compact | expire
docenta tidy --no-reconcile        # a store that names no roots
docenta tidy --repair              # decode-verify every frame first; the walk re-extracts

--force-wave replaces the retired gc verb's --force and the environment variable DOCENTA_ACCEPT_REMOVAL_WAVE=1; the variable is still honoured for one release so a script written before this verb keeps working, and the run says so when it is the variable that accepted the wave. On the Intel Mac on 2026-10-09 the same outcome took the reclaim (gc refused), the forced gc, the reclaim again; it is one tidy --force-wave now.

docenta update runs tidy as its third stage without the walk (the update's own scan just walked): when the compaction is due by the coverage threshold the reclaim runs, otherwise the semantic prune and the final check; always on an update you type, by threshold on a scheduled one.

8.1 The commands, and when to reach for each#

docenta store reset --page writes the same plan as one local HTML file and opens it: what goes, what stays (your files, the models, the runtime, the config, the logs, each said plainly), what the first build cost and what a rebuild costs. The page never decides; the typed confirmation stays in the terminal and nothing listens on an address.

Both reset commands print a plan with sizes and require a typed yes unless you pass --yes. docenta store reset also tells you how long your last full ingest took, so you know exactly what you are signing up to redo.

8.2 Compaction happens by itself#

Two things to know:

• Skip it when you are in a hurry with --no-compact on ingest folder or
ingest machine. There is no other setting.
• Compaction wants headroom. The index force-merge writes the merged
result before dropping the old segments, so it needs roughly the index's own size in free disk. If it cannot, it *skips safely* and tells you - your live data is never left in a partial state.

There is one recorded open issue: on rare occasions an ingest-triggered compaction has spun for hours without writing. If you see that, kill it and re-run the ingest with --no-compact; nothing is damaged.

8.3 What lives where, and how big it gets#

docenta status --storage

The machine every number below comes from, so you can hold your own up against it:

8.3.1 What each thing you add costs#

Docenta is built up in pieces. You start by scanning the machine, and after that every source is something you add when you want it and skip when you do not. Each block below gives the same three answers for one source: how big it is, how much disk docenta spends on it, and how long each step takes.

All figures are from the reference machine described above. Sizes are measured; the per-source disk and timings are those measurements applied to each source's own share.

Note the pattern before you read them: the first build is fast per item because it processes in bulk, while later additions go through the change feed one item at a time. Adding a million things later costs more than having them there from the start - though "more" still means hours, not days.

1. The machine's own files - where everyone starts#

That hour and three quarters is the whole scan side, and at the end of it you can already search every file by its contents.

Everything after it is the semantic side, and it is four decisions, not one. They do not form a single chain: two of them branch off the first, and the last one does not need any of the others at all.

     docenta ingest machine  (1h45, required)
             |
  +----------+------------------------+
  |                                   |
  1. passage index  (30 min)      4. the people layer  (minutes)
  |                               entity cards, timelines, themes
  +--------------+
  |              |
  2. meaning-    3. code
  vectors        vectors
  (13 h)         (2 h)

Three things follow from that shape, and they are the whole decision:

Steps 2 and 3 are independent of each other. Each builds its own separate set of vectors with its own bookmark, so you can have code search without general meaning search, or the other way round, or both. Neither needs the other.

The people layer needs nothing but the scan. It reads the extracted text directly and never touches the passage index or the vectors. So "I want timelines but I am not paying thirteen hours for vectors" is not a compromise

• it is a supported route, and it costs minutes.

Step 2 does not have to be swallowed whole. It works through your material in order and writes down where it got to, so you can run it for an hour tonight and an hour tomorrow, and whatever it has finished already answers queries in the meantime - a partial build is not a broken one, just a smaller one. There is also --only-kind if you want to restrict it to particular kinds of content.

Which models do this work, and can you choose?#

Steps 2 and 3 are the only places docenta uses a model to build anything, and each step uses exactly one model. By default that is two models, run one after the other - the general one over everything you own, then the code specialist over your source code only. They are not run together; each takes the machine in turn, using all its cores while it does.

You can substitute either one, and the trade is speed against how often the right answer comes back:

What "70 percent" actually means, because the number looks worse than it is. It is measured by asking 57 real questions - the kind you would actually ask, phrased the way you would actually phrase them - and checking whether the one document that answers each is in the top ten results, out of 839,000. The right document is the *first* result for about 37 percent of them.

The honest comparison is not against some perfect score, it is against what you have today. Ordinary file search cannot answer these questions at all: it looks at names, so "which insurance letter mentioned the main-drain test" returns nothing unless you happened to put those words in a filename. Going from *cannot ask* to *right answer on the first page seven times in ten* is the whole product. Retrieval research measures itself in the same range on standard test sets; being right most of the time is what good looks like here, not a shortfall.

Two cautions in the same breath, so the number is not oversold. Those 57 questions are our own, and we judged the answers ourselves - a small, self-graded exam, which is why we treat it as a floor to defend rather than a score to boast about. And it measures the *whole* system, keyword and meaning and all, which is the thing you actually use.

Read the speed and accuracy columns together before reaching for the faster model. The lighter alternative saves about four hours once, and gets the right answer for 44 percent of questions instead of 70 - roughly a third fewer answers found, permanently, every day you use the system. Four hours is one evening; the accuracy is forever. Install the default and let it run overnight.

How the code specialist finds your code - it does not read everything. It walks the same list as the other step but checks each entry's label first, and anything that is not a code declaration is dropped before any file is opened or any model runs. Skimming past a million non-code passages that way costs about eight minutes. Only actual declarations - functions, types, methods, about 123,000 of them here - go through the model, which is why the step is two hours rather than the day you would fear from its slower rate. If you keep no source code, it finds nothing and finishes almost immediately.

There is no way to have several general models cover your material at once, and no reason to want it: they would each rebuild the same passages at their own cost, and searches read from one. Sets left behind by an earlier model choice are simply dead weight - see the note on unused models above.

2. Adding an email archive - the best value in the system#

Twenty years of mail becomes individually searchable for two gigabytes. Mail is thousands of small things rather than a few big ones, so the *index* is the slow part here and the vectors are cheap - the opposite of your files. The entity layer pays off most on this material: every correspondent, resolved across address changes.

3. Adding your AI chat history#

Effectively free. Every decision you reasoned through with an assistant becomes searchable evidence instead of something you half-remember.

4. Adding your git history#

Every repository in scope, every commit of every author on every branch: the message, both identities and dates, the parents, the refs, and every changed path with its line counts. Onboarding runs it as its git step and prints the repositories and the commit count of each; the pass renders the delta from then on. This is what makes "why did we change this, and when" a search (section 6.4).

5. Adding messenger chats#

Not yet imported on the reference machine, so these are shapes rather than measurements.

6. Reading the pictures you already have#

No new files - this reads what is already there. For most people it is the difference between scanned paperwork being findable or invisible.

Adding it all up#

Skip the meaning-vectors and the whole thing is a morning's work. Keep them and it is an overnight run you start once and never repeat - and from then on a normal day's changes go through every step in well under a minute, which is why the hourly sync is something you switch on and forget.

Two things make the first build less imposing than the numbers suggest. You can search from the first minutes, while everything else is still building. And every step stops and resumes - close the laptop, kill it, lose power - because each one records how far it got.

8.3.2 The three groups: what each part is for#

Everything docenta writes falls into three groups, and the useful way to think about disk is to decide about the groups, not the individual files.

Group 1: the engine - finding words in your files#

27 GB, 7.5 percent of what it covers.

This is docenta itself; if you use it at all, you have this. It needs no models and no AI runtime: plain extraction and indexing.

It walks roughly 850,000 files and 270 GB per hour, and that covers far more material than the finished size suggests, because most of what any real machine holds is copies. The 1.8 million files on this one contain only 839,000 different things: two and a half copies of each on average. The same attachment sits in four mailboxes, the same document in a backup and a working folder, the same photo across three exports. Every copy is still read - identity comes from content, not from the filename, so there is no way to know something is a copy without reading it - but each distinct thing is extracted and indexed exactly once.

That is the main reason docenta is quick on a real machine, and why its footprint tracks what you actually *have* rather than how many places you happen to keep it.

Re-scans afterwards are a different order of magnitude again: 2.5 million files re-checked in five minutes, because a file whose size and timestamp have not changed is skipped without being opened at all.

Can you skip parts of it? No. These four work as one thing, and together they are the complete product: keyword, phrase, and regex search across everything, with exact citations. Many people never need more than this.

Group 2: the meaning layer - what you can pick and choose#

33 GB of representation plus about 3 GB of models. This is where the real choices are, because it is not one thing: it is two independent branches that both sit on group 1, and you can take either, both, or neither.

               GROUP 1: the core                27 GB   7.5%   2-3 h
               files, extracted text, index     no models needed
                           |
           +---------------+----------------+
           |                                |
     BRANCH A: meaning                BRANCH B: the graph
           |                                |
  passage index  27.5 GB  7.7%   ~30 min    mentions   ..
           |                                entities   ..  0.4 GB
  meaning-vectors 4.7 GB  1.3%   ~12 h      gazetteer  ..  0.1%
                                            claims     ..  minutes
                                            communities..  no models

The two branches do not depend on each other. The graph layers read the extracted text from group 1 directly; they never touch the passage index or the vectors. So the case of wanting people and timelines but not vectorization works exactly as you would hope - and it is the cheapest thing in the system.

Branch A: search by meaning

Branch B: people, timelines, and themes

The whole branch is 0.4 GB, about 0.1 percent of the corpus, and minutes of work - on the reference machine a pass over 34,500 new documents took 18 seconds across all five steps. It needs no models at all. Each step does depend on the one above it, but since the whole chain costs minutes there is no practical reason to stop part way; one command runs all five.

The asymmetry worth staring at

The vectors cost almost nothing in space and almost all of the time. The passage index is the reverse. The graph is nearly free on both axes. So the decision is not "how much disk do I have" but which of the three you actually want, and they price very differently.

So, concretely:

• Want everything? Run docenta semantic pipeline. Budget half a day for
the first pass, mostly the vectors, and let it run overnight.
• Want precise citations and people and timelines, but not meaning-search?
Run docenta semantic pipeline --only index, then --only sparse, --only mentions, --only entities, --only gazetteer, claims, communities. You save 12 hours and 4.7 GB, and you skip the embedding models entirely.
• Want only people and timelines? Run just the five graph steps. 0.4 GB,
a few minutes, no models, no GPU, no ONNX runtime. This is the single best value in docenta and most people do not realise it is available on its own.
• Want none of it? Do not run the second binary at all. Group 1 alone is a
complete citing keyword system.

The timings above are from an M4 with the embedding running on CPU. A machine with a usable GPU lane, or a smaller corpus, will move the 12-hour figure the most - it is the one number here that is hardware-bound rather than corpus-bound.

Group 3: reading pictures - OCR#

Under 1 GB of disk, and about 22 hours of background work for the 223,000 images on this machine - a measured 10,000 images per hour. It runs beside everything else and can be stopped and resumed at will, so it costs you waiting time only if you choose to wait. Optional, and very cheap in space.

Can you skip it? Yes - just never run docenta-analysis. You lose the ability to search the contents of scans, faxes, and photographed documents; everything else is unaffected. For most people this is the best value per gigabyte in the whole system, because scanned paperwork is exactly the material that is otherwise impossible to find.

The models#

About 3 GB for a working installation, and this cost does not grow with your corpus - it is the same whether you index 10 GB or 10 TB.

These are the measured sizes on disk of what onboarding fetches. docenta-setup lists each one and says whether it is present.

The ratio that matters: your files, and docenta's representation of them.

The reference machine has about 800 GB of data on its disk. Docenta covers 1.8 million files holding 358 GB of raw bytes - the rest is skipped by the default hygiene rules (build trees, package caches, system files) or is cloud material with no local bytes. Those 358 GB cost 64 GB, all in:

So a searchable, citable, meaning-aware representation of a third of a terabyte of files costs 18 percent of what those files themselves occupy. If disk is tight, group 1 alone is 7.5 percent - and it needs no models and no AI runtime at all, just the engine. A complete keyword, phrase, and regex search system over everything you own, with exact citations, for a fourteenth of what the files themselves take.

Your ratio will differ, and the reason is simple: only text becomes an index. A photo or video library is recorded from its metadata and costs almost nothing, so a media-heavy machine lands in the low single digits. A corpus of source code, documents, and mail is nearly all text and can reach 40 to 50 percent. The mix on this machine - code, documents, images, mail, and media together - is what produces 18 percent.

One more thing worth planning around: a first semantic build is temporarily much larger than its settled size, which is why docenta tidy --only semantic-compact after a big first pass is worth running. Compaction generally reclaims 25 to 45 percent of the pre-compaction total.

Everything lives under one data root, which docenta setup config will print. You can relocate any store by uncommenting the matching key in config.toml; the guidance in that file says which ones want your fastest disk (the index, the bundles, and the state database are the IO-hot ones). Models are the exception: they take an absolute path and move with docenta setup models --move, which copies, verifies every digest, and only then deletes the source.

8.4 Backups#

The honest answer: back up your files, not docenta. Every byte docenta stores is derived and can be rebuilt by re-ingesting. If you want to avoid the rebuild time, stop all docenta processes and copy the whole data root as a unit - copying it while a scan runs will give you an inconsistent snapshot.

The two things that are *not* rebuildable and deserve a copy: your config.toml, and ~/.docenta/spool-state/imap.toml if you set up mail.

Big documents are read in windows, never held whole, so a multi-gigabyte text, CSV or JSON file costs the same memory as a small one. What it costs instead is index space, and only when the content is worth indexing: a machine-generated body (a log, a minified bundle, generated code) is stored and fetchable in full but gets no search postings.

Size limits, and what they mean: a document a person wrote to be read (text, markdown, Org, JSON, CSV, XML, YAML, HTML, RTF, Office including the macro-enabled kinds, PDF, EPUB, notebooks, mail files and source code; the list is [limits] whole_document_extensions) is ingested whole at any size. A machine-generated file (a log, a trace, a dump, a line-delimited stream, an unknown extension) is recorded metadata-only over max_extract_bytes (256 MiB) and its text is cut at max_body_text_chars. When the text you are served is such a cut, the hit and the fetch say extraction_truncated: true: what you read is real, the tail is not held.

Every bound a lane keeps, stated (the audit of 2026-10-09, docs/architecture/ingestion-coverage-audit-2026-10-09.md, holds the reasoning row by row; nothing else limits what is ingested, and a filter that was someone's guess is gone or listed there as open work):

One more key in config.toml worth knowing: [chats] transcript. Agent sessions (Claude Code, Codex) render as prose by default, the conversation plus a compact facets block folded from the tool traffic (files touched, commands, errors, tests, commits). Set it to full and nothing the agent did is thrown away: its reasoning, every tool call field by field, every tool result whole, each searchable as its own turn. A tool result Claude Code offloaded to a file beside the session (the transcript keeps a short preview) is read from that file, whole; the transcript of every agent a session spawned is a session of its own, with the session that spawned it and the task it was given named at the top. Sessions grow three to twenty times; the next hourly pass or docenta ingest claude re-renders every session in the new mode. A session over about 3 million characters is written in parts (<id>.eml, <id>.part2.eml and so on, one thread). Credentials an agent met on the way (a password in a config it read, a token in a command) are replaced by [redacted secret] before the session is written; the field name stays, so the line is still findable. This happens to AI sessions only: every other file is searchable exactly as you can read it.

9. Upgrading#

9.1 The normal upgrade#

docenta setup update

One command (docenta-setup update is the same one), nothing typed before or after it (section 3.5 lists its six steps: the sync held, the swap, the assets, the checks, the scheduled jobs migrated, the agent configs). It fetches the newest build when there is one, verifies it against the signed checksum list exactly as the first install did, and replaces every binary in ~/.docenta/bin (on Windows, %LOCALAPPDATA%\docenta\bin). When you are already current it says so and changes nothing. The handbook and the other documents in ~/.docenta/doc are replaced with the build, so the book you open always matches the programs you run.

Each file is written under a temporary name beside the binaries and then renamed into place, so replacing a running binary is safe: anything looking at the file sees the whole old one or the whole new one, never half of each. The staging directory sits beside the binaries on purpose: a temp directory on another filesystem (systemd's private temp, a tmpfs) cannot be renamed across, and the updater copies where a rename cannot.

9.2 Your agent does not need restarting#

docenta serve runs as a supervisor that watches its own binary on disk. When you install a new version, it starts a fresh worker, replays the handshake, and swaps traffic over between requests - with no visible downtime and no config change. Your agent keeps its session, and is sent notifications/tools/list_changed so that it lists the tools again: a session that has been idle for days hears of the new build too, because its own docenta serve shim watches the binary while the hub has it dropped for silence and nudges it the moment the binary changes. Whether the agent acts on the nudge is the client's doing; status names the serving generation in tool_names either way.

9.3 The database migrates itself#

Schema changes apply automatically the next time any write command runs. You can also do it explicitly:

docenta store init

which reports already at schema revision N or the from/to pair. Migrations run one transaction each and roll back on failure, so an interrupted upgrade leaves the database at its last consistent revision.

9.4 Do you need to re-ingest? Almost never#

Docenta keys every stored extraction by a recipe: the content, the lane, its version, and the schema. When an upgrade improves an extractor, the recipe changes, and the next walk over those files re-extracts them automatically rather than hiding behind an unchanged timestamp. To refresh one modality deliberately:

docenta ingest folder ~/Documents --family document

The exceptions, all of which announce themselves:

• docenta update --only backfill - only for stores created before the change
feed existed. It synthesizes the missing feed entries from data already on disk without re-reading a single source file.
• docenta update --only analysis - after the OCR producer has run. A bundle
landing beside unchanged bytes is invisible to a normal walk, so this republishes exactly those.
• docenta semantic reset --vectors --yes, then semantic pipeline --only embed - after changing
the embedding model.
• Nothing - after a change to which files get tail windows (the extra embed inputs that
cover a long passage past its first 4,096 characters). The embed step notices a partition built under an older rule and walks the store for the missing windows itself, in the hourly pass's own time slices, and docenta semantic status shows one tail windows line per partition: current, walking N of M, or owed. The walk can also be run by hand (docenta dev embed --tail-windows, with --model-dir and --only-kind for a specialist partition); a partition seeded by the current build needs none, and a partition the store does not hold (a prose-only store has no code units, so the code model never wrote a vector) has nothing to walk: the step says so in one line and goes on, while the manual verb refuses a partition that is not there.
• Nothing - after a model's embed window changes (the tokens of one input it is fed, the
window_tokens of its manifest.json; the inputs and the tail windows are sized by it, 8 characters a token). The partition keeps its name (the window is policy, not the model's identity) and its record says which window built it, so docenta semantic status shows embed window: ... built at 512 tokens, the model reads 1024: a re-embed is owed, and the next embed pass under the lane word re-embeds the store in place in its own time slices (re-embedding, N of M); the partition answers searches all along. The Apple GPU variant is pinned at 512 tokens: a model whose window is wider embeds on the cpu and says why.
• docenta semantic reset --units --yes, then semantic pipeline --only index, then
semantic pipeline --only sparse - after an analyzer or schema change to the unit index. The index step builds the lexical index and attaches the sparse terms the cache already holds; a unit the cache lacks is indexed bare and counted. The sparse step encodes the bare units (--sparse local | turbo | later decides; later leaves the index in place and the step waiting) and upserts them; nothing bare means nothing to do.

9.5 The runtime and its lanes#

The ONNX runtime is an asset like the models, and it is the one asset whose contents decide whether your GPU is used at all: Microsoft builds it per platform with different execution providers inside, and a build without this platform's provider runs every model step on the CPU, ten to twenty times slower, with nothing visibly wrong. On 2026-10-05 the owner's Windows box did exactly that for a day after a runtime bump had moved it to Microsoft's CPU build (there is no DirectML build past 1.24.4). Four things now stand between that and you:

1. The publisher declares the providers each platform's runtime must
carry and inspects every archive before it is published; a runtime without its lane is not published.
2. The channel's index carries that declaration, and docenta-setup
(install, update, verify) asks the installed runtime what it carries - the runtime itself, not the file name - and names a gap: runtime : onnxruntime 1.28.0 lacks directml (the channel declares cpu, directml; the platform's lane directml is MISSING ...). A gap fails verify.
3. A model step that asks for a lane the runtime lacks loads on the
CPU in the CPU's full shape (every core, the CPU's batch budget), and its arming line says so: accel directml (auto) - fell back to cpu: the loaded ONNX Runtime has no directml provider. A lane that fell back is never compared against the CPU as if it were a GPU, so no parity verdict is cached for it.
4. Every embed and sparse step records the rate it achieved on this box
per model and lane, and a run far below the box's own best says so in its first minutes and in status (section 7.4).

The Windows runtime stays at 1.24.4 with DirectML until Microsoft ships a DirectML build again or docenta grows a Windows ML lane; the pinned ort crate loads any runtime from 1.17 on, so the runtime version is the publisher's choice per platform, not the crate's.

10. Troubleshooting#

"Too many open files"#

The scheduler's default file-descriptor limit is far too low for a recovery pass. Both binaries now raise their own limit at startup, and the sync script sets ulimit -n 8192. If you run passes from a custom wrapper, do the same.

The machine is swapping, everything is slow#

Almost always two pipelines running at once. The pipeline takes a lock (pipeline.lock in the semantic directory) and a second instance now exits immediately with another pipeline is already running on this store.

Also check whether a build tree or another job is eating the disk: docenta status --storage shows docenta's own footprint, and if that looks normal, the space is going somewhere else.

"the semantic engine is not wired on this server"#

Your agent is asking for meaning-based search before the layer exists. All three of these must be true: <semantic>/units exists, <semantic>/vectors exists, and your embedding model's manifest.json exists. Run docenta semantic pipeline. The server re-checks on every call, so it starts working without a restart.

"the semantic entity store is not built on this corpus yet"#

Same idea for who_is, timeline, themes, and evidence: run docenta semantic pipeline and they mount themselves.

"loading ONNX Runtime from ...: ..."#

The runtime is missing or somewhere unexpected. Run docenta-setup, which fetches it if it is absent and says where it put it, or set ORT_DYLIB_PATH to the full path of the library file.

Search quality feels poor#

Check which embedder actually armed. Run docenta semantic pipeline --only embed -v and read the armed: line it prints - it names the model directory in use. If it says multilingual-e5-small or baseline Mix (no models installed), the production embedder is not installed: revisit chapter 3.4. This is the single most common quality problem, and it is silent by design (docenta prefers degrading to failing).

Files exist but are not searchable#

Run through these in order:

1. docenta search "<something exact>" --json - is the file even in the
corpus?
2. Is it a cloud placeholder? Files stored only in iCloud, Google Drive,
or OneDrive have no bytes locally. Docenta records them without reading them (which would trigger a full download of your entire cloud drive) and docenta store verify counts them as skips. A file already recorded this way is settled, so a walk never reads it again on its own: docenta update --only finish -- --fetch-from <folder> (the finish step with its folder) fetches and reads the placeholders under that folder, if you truly want the download.
3. Is it a large mailbox? Mail containers over 256 MB are not exploded
automatically; the ingest prints a note and waits for docenta ingest mailbox.
4. Is it an archive other than ZIP? tar, 7z, and rar are cataloged
but not expanded.
5. Is it in an excluded directory? node_modules, target, .git,
Library/CloudStorage and about forty other patterns are excluded by default for hygiene. docenta setup config shows your effective settings.
6. Is it in another person's home directory? On a shared machine those
are skipped by default, even where the permissions would allow reading them - someone else's account is not part of your material. Shared folders and shared volumes are still indexed, because those were shared deliberately. If you genuinely need one, name it: docenta ingest folder /Users/them/Projects still works.

A long run looks hung#

Use -v and watch the heartbeats, or docenta semantic status --watch in another terminal. Two field lessons worth repeating: watch the binary's process and the cursor file, not a wrapper script's memory, and never run bulk passes with a debug build - one measured 150 times slower than release while looking perfectly healthy.

Verify reports divergences#

docenta store verify exits non-zero when stored text no longer matches disk or a sampled object fails its search probe. Run it with --json to see which content ids diverged. The search line reads hit / buried / miss: a probe is a phrase from the middle of the sampled object, and on a corpus with thousands of near-identical files (unpacked tarballs, stock-quote CSVs) the sampled one can sit below the probe depth. Verify then asks the same phrase of that object alone; a match is reported as buried (indexed and searchable, merely outranked - not an error), no match is a real miss. Ordinary causes: the file was edited after indexing (re-ingest fixes it) or it is a cloud placeholder (expected). If sidecar references point at pack files that no longer exist, docenta tidy --only compact --repair deletes those rows and marks the files for re-extraction on the next full scan.

11. Command reference#

docenta - the engine#

The tree: four daily verbs at the top, everything else a verb under a noun (docenta <noun> --help lists the verbs).

The daily verbs

ingest - bringing content in

The source map - every source, its mode, its state, the command that advances it - is docenta status --sources. What used to be an ingest verb and is a STEP of docenta update since the verb map (2026-10-09):

A step's own switches ride on the update (2026-10-09): everything after -- goes to the one step --only names, so docenta update --only finish -- --fetch-from ~/Documents/scans is the finish step with its folder, and docenta update --only stale -- --sparse-text the stale step's empty-text sweep. Naming two steps, or none, with switches after -- is refused in one line. Every step is its own process under the update's runner (for the transcript, the timing, the receipt and the crash isolation); the entry it spawns is internal, like the extract worker, and nothing a person types.

docenta update --dry-run is the whole pass's dry run: it prints the plan (the same read-only plan docenta status shows as "next update", stage by stage, what runs and what is skipped and why) and every planned step that has a dry run of its own runs it - stale says which files are behind, finish which recorded files it would read, promote-bodies how many bodies it would promote - and nothing is written: no transcript line, no receipt, no done marker, no summary of a pass that did not happen. --only STEP --dry-run narrows it to one step's dry run.

sync - keeping it fresh

bale - moving a store

tidy - space back and the store in top shape

store - the store's lifecycle

setup - who, what, where

explain - the inclusion policy's verdict

dev - the instruments (hidden from docenta --help; docenta dev --help lists them)

semantic calibrate - the fusion table, judged by what agents read

Calibration is opt-in and local. It reads the query log this machine wrote, replays it offline against candidate fusion tables, and adopts one only when the evidence clears a bar it names. Nothing is sent anywhere, and --reset puts the shipped constants back.

docenta-semantic - the meaning layer#

Reached as docenta semantic <verb>, docenta turbo <verb> and docenta ask (CLI arc phase 2): docenta resolves the sibling beside itself, passes --data-dir and --store through, inherits the terminal, forwards Ctrl-C and a kill, and ends with the sibling's exit code. Since the arc's third phase (2026-10-09) the sibling's own page lists only what a person reaches through docenta: status, pipeline, runtime, graph, reset, partitions, models and turbo status | receive | consent HOURS

bench, embed) and the operator's and the fleet's turbo verbs are docenta dev ... (docenta dev --help lists them), and the semantic compaction, prune and repair are steps of docenta tidy and docenta update. docenta setup runtime is docenta-semantic runtime, docenta setup models is docenta-semantic models.

Three lanes under one word. The code partition (qwen3-embedding-0.6b over the code_symbol units, the hybrid's code-dense channel) is the third lane since 2026-10-09: the one lane word covers all three (turbo ships them, local builds them, lazy and later leave them), no new question is asked at the onboarding, and the sizing page, the site's decision, the quote, the consent, the ship, the receive, docenta status and docenta semantic status all carry the code lane with its own count, rate and time. A machine without the code model installed says so on the page and skips the lane. The status page's code line is the partition's pending count by count (the index's code_symbol units less the vectors held), the lane's turbo work when any is out, and "no code model installed" otherwise.

The turbo consent and its ceiling. The word turbo stands for the store: a later refresh that finds a delta over the line packs it and sends it on that word, with nobody at a terminal. What it may send is capped by the quote you said yes to. The consent is recorded with its GPU-hours and the job's text, unit and code unit counts when you say yes with the numbers on the screen (at the onboarding, at turbo consent <hours>, at turbo ship --consent <hours>, or with --yes typed at a terminal), every shipped job's quote is spent against it, and a pass with nobody at a terminal (the hourly sync, the onboarding's detached build) ships only a job that fits inside what is left plus a small allowance for ordinary growth between passes: a tenth of the consent, at least 0.1 GPU-hours, and spent like the rest so an hourly trickle cannot add up past it. A job beyond that is not sent and nothing is rented: the pass prints one line with the quote, the consented figure, what is left and the exact command that consents to the new amount (docenta turbo ship --consent <hours>, or docenta turbo consent <hours> for the ceiling alone), raises the desktop notification, and leaves the pack for your word. At a terminal the same job is one question with the numbers, default no. status and turbo status show the standing consent and what is left of it. A new consent replaces the standing one and counts from zero.

docenta-spool - connectors#

docenta-analysis - the OCR producer#

docenta-analysis --analysis-root DIR --state-db FILE --models DIR [--limit N]

Long-running and safe to stop; re-running skips work already done. Follow it with docenta update --only analysis to make the results searchable.

It reads pictures (images, and PDFs with no real text layer) and, since 2026-09-12, PDFs whose text layer is all there but in the wrong order (a plan sheet, a form, a map laid out by column): docenta judges that at ingest and records the verdict on the content object, the producer re-reads those pages, and a hit served from OCR text says extraction: ocr. A store ingested before the verdict existed gets it filled in by docenta update --only enrich -- --only verdicts (it reads the stored text, never the originals).

12. The pages docenta shows you#

docenta is a terminal tool, but a few things read better as a page than as a wall of text: a plan you are about to approve, the book you are reading now, the four steps of a mail login. This chapter names every page docenta or one of its siblings can put in front of you, so none of them is a surprise.

12.1 The rule#

If this machine can show a page, you get the page. If it cannot, you get the same thing as text. Nothing is hidden behind a window that never opens. docenta measures the machine rather than assuming: an explicit DOCENTA_BROWSER=never or always settles it, otherwise macOS and Windows always have an opener, and a Unix box needs a graphical session (DISPLAY or WAYLAND_DISPLAY) with xdg-open on the path. A container, a build agent or an ssh login without forwarding is told which of those it lacks, and gets the text.

A page that lives on this computer never sends anything anywhere: it explains, or it asks and hands your answers to the terminal that opened it. A page that asks never decides for you: every question starts not set with its default marked, the submit points at the first one still open and refuses until all are answered, the "answered" tag on a question unsets it, and what you answered is kept in this browser until it is handed over, so a crash loses nothing (a password is never kept). Where the terminal takes a typed word - the removal and the reset - the page explains and the terminal decides. A page on the site decides only what it says it decides, and you reach it by a link or a short code the terminal prints, never by docenta opening an address on its own.

Every page carries the same bar: the text size (minus, the percent, plus; steps from 85 to 200; Command or Control with plus, minus and 0 on the page itself, the browser's own zoom untouched; remembered in this browser for every docenta page), and the theme (system, light, dark; its tooltip names the active one and the next). A table wider than the text takes the screen's width, centered, before it takes a scrollbar.

12.2 The catalogue#

12.3 Seeing them for yourself#

docenta handbook --samples ~/docenta-pages

writes one example of every page this binary renders, plus an index describing the whole catalogue above, and opens the index. The samples are drawn by the same code the real commands call, from one example machine, so they cannot drift from what you will actually see.

12.4 This handbook, in two forms#

Every install carries both: HANDBOOK.md for reading anywhere, and HANDBOOK.html for reading in a browser, with the glossary live - every term you meet is underlined, says what it means when you rest on it, jumps to its entry when you click it, and every entry has a "back" that returns you to exactly the place you were reading.

docenta handbook picks the form your machine can show and says which it chose and why. docenta handbook --print always writes the text to your terminal.

The words in this book#

Each dotted word in the chapters shows its meaning when you rest on it and jumps here when you click it. Every entry has a back that returns you to the line you were reading.

Agentback

an AI assistant that can use tools for you, such as Claude, ChatGPT or Cursor. docenta is one of the tools it can use: the agent asks, docenta finds the page.

App passwordback

a separate password your mail provider makes for one program only. Your real password stays yours, and you can throw the app password away at any time.

Backgroundback

work that keeps going on its own after the command returns. You can close the window; it continues.

Baleback

a packed box of what docenta learned, made for carrying to another of your own machines. Every item inside is listed with its fingerprint, so a box that was opened or changed on the way is refused.

Binaryback

a program file the computer can run. docenta is eight of them, all in one folder.

Bundleback

docenta's written record of one file it read: what it found inside and when. The meaning layers read these records to learn what is new.

Cacheback

a saved copy of an answer, kept so the same work is not done twice.

Citationback

the exact file, page or line an answer came from, so you can open it and check.

Claimsback

small facts docenta noted while reading, such as who did what and when. Timelines are built from them.

CLIback

short for command line interface: using a program by typing in the terminal instead of clicking.

Commandback

a line you type in the terminal and finish with the Enter key. Everything this book shows in a grey box is one.

Commitback

one saved step in a git history, with a note saying what changed and why.

Compactionback

tidying the store so it takes less room, like closing the gaps on a bookshelf. docenta does it by itself after a big read.

Content idback

the name docenta gives a piece of content: the word blake3: followed by its fingerprint. It names what the content IS, not where it lives, so it stays the same when a file moves.

Content objectback

one distinct piece of content. Ten copies of the same photo are ten files but one content object, so docenta reads it once.

Corpusback

everything docenta has read for you, taken together: your files, your mail, your chats, your code history. When this book says "18 percent of the corpus" it means 18 percent of all of that, measured in bytes on your disk.

Cursorback

a bookmark. Each part of docenta writes down how far it got, so after a stop it carries on from there instead of starting over.

Dense vectorsback

the long lists of numbers a model writes for each passage, used to find text by meaning. See vector.

Doorback

a collection we host so your assistant can ask it over the internet. It is the only thing that lives on our side; your own files never do.

Drop box

a folder, ~/.docenta/drops/<provider>, where you put a download docenta cannot fetch for you, such as a chat export.

Embedderback

the model that turns a passage of text into its list of numbers.

Entityback

a person, a company, a place or an account that docenta recognises as the same one across many files.

Evidence packback

one ready-made briefing for your assistant: the passages that answer the question, where each came from, where sources disagree, and what could not be found. It is kept short on purpose.

Extractionback

pulling the words out of a file, whatever kind it is: a PDF, a Word file, an email, a photo of a page.

Familyback

the kind of thing a file is: text, document, image, animated, media, archive or unknown. Each kind except unknown has its own reader.

File instanceback

one place on your disk where a piece of content was found. Many instances of few content objects means you keep many copies.

Fingerprintback

a short code worked out from a file's contents, also called a hash or a checksum. Change one letter and the code changes, so two files with the same fingerprint are the same file.

Flagback

an extra word on a command that starts with two dashes, like --json. It changes how the command behaves.

Full Disk Accessback

a macOS permission that lets a program read all your folders without the system asking folder by folder.

Gazetteerback

docenta's list of names and their spellings, so that "Bob Smith" and "Robert Smith" are understood as one person.

Gitback

a tool programmers use to keep every past version of their files, with a note for each change.

GPUback

the chip that draws pictures on a screen. It is also very fast at the arithmetic the models need.

Ignore fileback

a file named .docentaignore that lists what docenta must not read, one pattern per line.

IMAPback

the standard way a program reads mail from a mail provider such as Gmail, iCloud or Yahoo.

Indexback

a list of every word and where it appears, like the index at the back of a book, so a question is answered in a blink.

Ingestback

reading every file once and writing what it says into the index. It is the long part of the first run, and it resumes if stopped.

JSONback

a tidy text format that programs read easily. Add --json to a command and it answers in it.

Keyword searchback

finding the exact words you typed, like the search box you already know. Also called lexical search.

Laneback

the reader for one family of files: one lane reads documents, another reads images. Each lane has a version number, and when it improves docenta re-reads that family by itself.

Leaseback

a signed permission file, tied to one machine, that says which edition of docenta may run on it.

Lock fileback

a small file that says "someone is working here", so two runs cannot trip over each other.

Mboxback

one big file holding many emails, the form mail downloads come in.

MCPback

the standard plug that lets your assistant ask docenta questions. One plug, and every assistant fits. The letters stand for Model Context Protocol.

Mentionback

one place where a person's or a company's name appears in your files.

Metadataback

facts about a file rather than what it says: its name, its size, its dates.

Migrationback

a one-time change to the layout of the store when docenta is upgraded. It happens by itself.

Modelback

a file that turns words into numbers, so the computer can tell that "car" and "automobile" mean nearly the same thing. It runs on your machine and is never sent anywhere.

OAuth

letting your mail provider hand docenta a permission of its own, with no password copied anywhere. "Sign in with Google" is this.

OCRback

reading the words in a picture or a scanned page, the way you can read a photo of a sign. The letters stand for optical character recognition.

ONNX Runtimeback

a helper program from Microsoft that lets the models run on your own computer's chips. Fetched once.

Originalsback

your own files. docenta reads them and never copies, moves or changes them.

Partitionback

one shelf of vectors, all written by the same model. A different model gets its own shelf, because numbers from two models cannot be compared.

PATHback

the list of folders your terminal looks in when you type a program's name. Once docenta's folder is on it, typing docenta just works.

Pipelineback

the steps that build the meaning layers, run one after another in a fixed order. Each step picks up where it stopped.

Placeholderback

a file that only pretends to be on your disk: its contents live in a cloud drive until you open it. docenta cannot read what is not there.

Preflightback

the quick estimate before a long job: how much there is to do, how long it will take and how much disk it needs.

Recipeback

the exact way a piece of extracted text was made: which reader, which version, which model. When the recipe changes, docenta makes it again.

Reclaimback

removing from the store what belonged to files that are gone or that you excluded.

Reindexback

building the index again from what docenta already extracted, without reading your files again.

Repositoryback

a folder whose whole history git keeps.

Rerankerback

a last careful reader that takes the best few answers and puts them in the right order.

Rust

the programming language docenta is written in. You never need it: docenta reaches you already built.

Scanback

a quick look for what changed since last time, so only new or changed files are read again.

Schedulerback

the part of your computer that starts a program at set times. On macOS it is launchd, on Windows Task Scheduler, on Linux cron.

Schemaback

the layout of the tables inside the store: which facts are kept and in which columns.

Sealback

freezing a collection, so that nothing is added, changed or removed afterwards.

Semantic searchback

the part that understands meaning, not just spelling: ask about "the roof leak" and it finds the mail that said "water coming through the ceiling". The semantic layers are built after the index, in the background.

Sidecar / blobback

the words docenta pulled out of a file, squeezed small and kept in the store. It is never the file itself.

Signedback

carrying a signature from us that your computer can check, so a file that was changed or swapped on the way is refused.

Snippetback

the few lines around a match, shown so you can see why it matched.

Sparse termsback

a second word list a model writes for each passage, adding related words the text did not literally use. It helps a keyword find text that says the same thing differently.

Spoolback

a folder, ~/.docenta/spool, where the connectors write your mail and chats as ordinary files. docenta then reads them like any other file.

Stdioback

the simplest way two programs on one computer talk: one writes lines, the other reads them. Nothing goes over the network.

Storeback

the folder where docenta keeps what it learned about your files. Your files themselves are never copied or moved.

Syncback

one pass that brings everything up to date: new files, new mail, new chats.

Takeoutback

Google's download of everything in your account, mail included.

Terminalback

the window where you type commands instead of clicking. On a Mac it is the app called Terminal; on Windows it is PowerShell.

Themeback

a topic that runs across many of your files, found by docenta on its own. Also called a community.

Timelineback

what happened, in order, with the file behind each step.

Tokenback

a piece of a word, the unit AI assistants count when they read. Fewer tokens means a faster and cheaper answer, which is why docenta hands over the right passage and not the whole file.

Turboback

renting a GPU for a few hours so the slow arithmetic finishes in hours instead of days. Only the extracted text travels, sealed, and the rented machine is destroyed afterwards. Three lanes ride it: the sparse terms, the dense vectors and, since 2026-10-09, the code partition's vectors.

Unitback

a passage-sized piece of a document. It is what the meaning layers index and cite, so an answer points at a paragraph and not at a whole file.

Vectorback

the list of numbers a model makes from a piece of text, also called an embedding. Texts that mean similar things get similar numbers, which is how a search by meaning works.

WAL

short for write-ahead log: a notebook the store writes into first, so nothing is lost if the power fails halfway through a write.