maiven gateway · online··
manifest · 2d ago
Sign in Back to catalog
Definition
Cleaned mirror of `raw_unstructured.documents`. Same column set as the raw layer, but with proper UUID/DateTime/LowCardinality types and a derived `extracted_day` for time-partitioned MergeTree downstream. PII columns inherit their full meta from the source layer.
17 of 17
Column
Description
Tests · PII
source_uri
gs:// URI of the source binary.
1 testPII
sha256
SHA-256 of the source binary. Idempotency key.
1 test
mime
MIME type — LowCardinality.
doc_lang
ISO-2 lowercased language code.
pages
Page count. Nullable when Docling cannot resolve it.
extracted_text
Full markdown export of the document. Free-form, unbounded.
PII
extracted_at
When the extractor ran.
1 test
extracted_day
Calendar date of `extracted_at`. Used as MergeTree partition key downstream.
1 test
extractor
Extractor library + version.
pipeline
Extraction pipeline that ran — `vlm` or `ocr-easyocr` for Docling (PDF/image), `text-passthrough` for born-digital text (.txt/.md), which skips Docling and chunks the markdown directly, or `text-local` for the mdt Workflow-driven local extraction (scripts/mdt/extract_local.py) used by the Smithers package.
1 test
pii_engine
Presidio version that produced the entity-count summary.
pii_entities_json
JSON object — map of entity-type → count. Counts only; values never stored.
pii_entity_kinds
Number of distinct entity types detected (== length of pii_entities_json).
source_file_url
Airbyte-supplied `_ab_source_file_url` — the gs:// URI of the JSONL row that produced this document row.
source_file_modified_at
Airbyte-supplied object mtime of the source JSONL.
airbyte_extracted_at
When Airbyte extracted this row from GCS.