Search

Chunking

Text files are indexed as markdown-aware section chunks: heading-bounded slices with 1-indexed line ranges, a heading breadcrumb ("Title > Section"), and a token budget that fits embedding models. Chunks are keyed by the blob hash, so unchanged content is never re-chunked — across rewrites, copies, versions, and branches. Chunking is off by default; enable it per instance:

const fs = new PgFileSystem({ db: sql, workspaceId: "workspace-1", chunking: true })
// or tune: chunking: { maxTokens: 480, volatileFrontmatterKeys: ["fetchedAt"] }

await fs.writeFile("/docs/page.md", markdown) // chunks stored with the write
const chunks = await fs.readFileChunks("/docs/page.md")
// [{ chunkIndex, startLine, endLine, headingPath, content }, ...]

Pre-existing content (written before chunking was enabled) is indexed by one idempotent pass: await fs.backfillChunks(). It chunks the text blobs visible at the handle's version and is safe to re-run.

Full-Text Search

BM25-ranked lexical search over the chunks. Requires enableFullTextSearch: true in setup (the default).

const hits = await fs.textSearch("shipping costs", { path: "/docs", limit: 10 })
// [{ path: "/docs/faq.md", startLine: 12, endLine: 34,
//    headingPath: "FAQ > Shipping", content: "...", rank: 1.42 }, ...]

// hydrate a hit with the exact section text
const { content } = await fs.readFileLines(hits[0].path, {
  offset: hits[0].startLine,
  limit: hits[0].endLine - hits[0].startLine + 1,
})

Embedding Pass

Semantic and hybrid search read a per-content vector cache (fs_chunk_embeddings, keyed by chunk content hash) that only an explicit pass fills — never the write path. The one batch embed option serves both the indexing pass and query embedding. The pass is version-scoped: it embeds what the handle's current version serves, so branch → index → promote makes the new main fully indexed the moment it becomes visible, and stale versions never cost an embedding call.

const fs = new PgFileSystem({
  db: sql,
  workspaceId: "workspace-1",
  chunking: { volatileFrontmatterKeys: ["fetchedAt"] }, // keep re-crawls cache-stable
  embed: async (texts) => {                             // one vector per text, batched
    const res = await openai.embeddings.create({
      model: "text-embedding-3-small",
      input: texts,
    })
    return res.data.map((d) => d.embedding)
  },
  embeddingDimensions: 1536,
})

const branch = await fs.fork("crawl")
// ... writes (chunks ride along) ...
await branch.indexChunkEmbeddings()
// { chunks: 120, cacheHits: 118, embedded: 2 } — only changed sections embed
await branch.promoteTo("main")

Requires enableVectorSearch: true + embeddingDimensions in setup. The cache is content-addressed: identical sections anywhere in the workspace share one vector, and content that comes back after a revert still cache-hits.

Semantic Search

Ranks embedded chunks by cosine similarity. Nearest-neighbor semantics: it always ranks something, so irrelevant queries return low-rank hits rather than nothing — and chunks not yet embedded are invisible to it.

const similar = await fs.semanticSearch("how do refunds work", { limit: 10 })
// same result shape as textSearch()

Hybrid Search

Fuses the BM25 and vector rankings with reciprocal-rank fusion — rank-based, not score-based, because BM25 scores and cosine similarities aren't on comparable scales. An exact rare token and a synonym-only paraphrase each still surface, and chunks without a cached embedding stay reachable through the lexical side. Requires both the full-text and vector setups.

const hits = await fs.hybridSearch("delivery options", { perFileCap: 2 })
// same result shape as textSearch()/semanticSearch()

semgrep in Bash Sessions

The semgrep custom command puts all of this inside a just-bash session — hybrid when the handle has an embedder, BM25-only otherwise. See Bash.

ChunkSearchResult

Search Options

All three searches take the same options: