mart_document_chunks

mart table 2 PII columns

Embedded chunks — the semantic-search surface.

Definition

One row per chunk with an `embedding Array(Float32)` column and a `vector_similarity('hnsw', 'cosineDistance', N)` index. AI reaches this mart only via `search_documents` — never via raw SQL. The `embedding` column is initially `[]`; the embedding worker fills it idempotently.

Watch out for

Querying chunks via raw SQL
There is no raw-SQL surface over this mart for AI. The only path is the gateway's `search_documents` MCP tool, which embeds the query and runs persona-scoped cosineDistance. Direct SELECT on `chunk_text` or `embedding` requires operator credentials.
Swapping the embedding model without re-embedding
The HNSW index is built for `cosineDistance` with a fixed dimension. Swap requires updating both the `EMBEDDING_DIM` env var on the worker AND the `embedding_dim` dbt var, then rebuilding the mart (which drops the old index) and re-running the embedder over every row. Mixed-dim or mixed-model rows break the index silently.

Questions this answers

“Find the 10 most relevant chunks about <topic>.”
“How many chunks did we produce per document this month?”

12 of 12

Column

Type

Description

Tests · PII

document_id

UUID

FK to mart_documents.

document +4 PII2 tests

chunk_seq

UInt32

0-based position within the document.

1 test

chunk_text

Nullable(String)

Paragraph-shaped chunk text. Soft cap ~512 tokens.

1 testPII

section

Nullable(String)

Heading path Docling resolved for this chunk.

page_start

Nullable(UInt32)

First page this chunk's content appears on.

page_end

Nullable(UInt32)

Last page this chunk's content appears on.

source_uri

Nullable(String)

gs:// URI of the source binary (denormalised for search ergonomics).

doc_lang

LowCardinality(String)

ISO-2 language code (denormalised).

extracted_at

Nullable(DateTime64(3))

When the extractor ran (denormalised).

extracted_day

Nullable(Date)

Partition key — calendar date of `extracted_at` (denormalised).

embedding

Array(Float32)

Dense vector embedding. Array(Float32), dimension declared by var('embedding_dim') (default 768 for granite-embedding-278m). Initially `[]`; filled by docker/embedding-worker. HNSW vector_similarity index on this column. Stored CODEC(ZSTD).

PII

airbyte_extracted_at

DateTime64(6)

When Airbyte extracted this row from GCS.