maiven gateway · online··
manifest · 1d ago
Sign in Back to catalog
mart_documents
mart table 2 PII columns
Catalog over our unstructured corpora.
Definition
One row per document we've extracted. Use this to look up what we have, what the extractor did to it, and how much PII it likely contains (per the Presidio entity-count summary). The searchable surface is `mart_document_chunks`; this mart is the index of what's been indexed.
Watch out for
- Treating `pii_entities_json` as PII content
`pii_entities_json` is a JSON map of entity-type → count (e.g. `{"PERSON": 1, "LOCATION": 4}`). It carries counts only — never the values. Use the metric `documents_by_corpus` for governance-level reasoning; for the actual text reach the chunk via `search_documents`.
Questions this answers
- “How many documents do we have per corpus this month?”
- “Which corpus is producing the most distinct PII entity types?”
15 of 15
Column
Type
Description
Tests · PII
source_uri
Nullable(String)
gs:// URI of the source binary.
1 testPII
sha256
Nullable(String)
SHA-256 of the source binary.
1 test
mime
LowCardinality(String)
MIME type — LowCardinality.
doc_lang
LowCardinality(String)
ISO-2 lowercased language code.
pages
Nullable(UInt32)
Page count from Docling.
extracted_text
Nullable(String)
Full markdown export. Free-form, unbounded.
PII
extracted_at
Nullable(DateTime64(3))
When the extractor ran.
1 test
extracted_day
Date
Partition key — calendar date of `extracted_at`.
1 test
extractor
LowCardinality(String)
Library + version that ran (e.g. `docling@2.14.0`).
pipeline
LowCardinality(String)
Docling pipeline that ran — `vlm` or `ocr-easyocr`.
pii_engine
Nullable(String)
Presidio version that produced the entity-count summary.
pii_entities_json
Nullable(String)
JSON object — map of entity-type → count. Counts only.
pii_entity_kinds
Nullable(UInt64)
Number of distinct entity types detected.
airbyte_extracted_at
DateTime64(6)
When Airbyte extracted this row from GCS.