§ I · The Data Catalog
Data Catalog
Every Pydantic model that flows through any workflow. One source of truth: same definition produces backend validation, TypeScript types, and the API contracts the agents call against.
Models · total88+3 · this week
Fields · total1,284+38 · this week
ECS components14composable everywhere
Schema migrations142since v0.1
Breaking changes · 30d2deprecation period
§ I · MODEL · CONTENT
ChunkModel
The fundamental unit of retrieval. A passage of text small enough to embed and reason about, with its hierarchy path, source provenance, and confidence.
| Field | Type | Required | Description |
|---|---|---|---|
| id | str | req | Unique identifier · ULID format. |
| document_id | AssetModel.id | req | FK to the source asset this chunk was extracted from. |
| text | str | req | The literal text content of the chunk. |
| chunk_index | int | req | Position within the parent document. |
| page_number | int | None | opt | Page in original document (if paginated). |
| section_heading | str | None | opt | Inherited heading from agentic structural pass. |
| hierarchy_path | list[str] | req | Walk from document root to this chunk · ["Part 1", "Ch.3", "subsection"] |
| chunk_source | Literal["agentic"|"programmatic"|"reconciled"] | req | Which RAPTOR pass produced this chunk. |
| confidence | float | req | 0–1 · agent's confidence in the boundary placement. |
| divergence_notes | str | None | opt | Populated when agentic and programmatic disagree. |
| raptor_level | int | req | 0 = leaf chunk · 1+ = summary node above. |
| is_leaf | bool | req | True if this is original-text leaf, false if RAPTOR summary. |
| parent_ids | list[ref] | opt | RAPTOR ancestry · summary chunks above this one. |
| children_ids | list[ref] | opt | RAPTOR descendants · finer-grained chunks below. |
| timestamps | TimestampComponent | req | created_at, updated_at · inherited from ECS component. |
| tenant | TenantComponent | req | tenant_id, group_id · inherited from ECS component. |
| provenance | ProvenanceComponent | req | workflow_run_id, agent_id, prompt_id · audit chain. |
from pydantic import BaseModel, Field
from typing import Literal, Optional
class ChunkModel(ContentFactoryModel):
# The fundamental unit of retrieval: passage of text small enough to
# embed and reason about, with hierarchy path + source provenance.
id: str = Field(description="ULID")
document_id: str
text: str = Field(min_length=1)
chunk_index: int = Field(ge=0)
page_number: Optional[int] = None
section_heading: Optional[str] = None
hierarchy_path: list[str] = Field(default_factory=list)
# Dual-hierarchical RAPTOR fields
chunk_source: Literal["agentic", "programmatic", "reconciled"]
raptor_level: int = 0
is_leaf: bool = True
parent_ids: list[str] = Field(default_factory=list)
children_ids: list[str] = Field(default_factory=list)
# Confidence & reconciliation
confidence: float = Field(ge=0.0, le=1.0)
divergence_notes: Optional[str] = None
# ECS components: composed in from base classes:
# timestamps: TimestampComponent (created_at, updated_at)
# tenant: TenantComponent (tenant_id, group_id)
# provenance: ProvenanceComponent (workflow_run_id, agent_id, prompt_id)
// Auto-generated from ChunkModel · do not edit by hand.
// Source: backend/models/content.py · regenerated 2 days ago.
export interface ChunkModel extends ContentFactoryModel {
id: string;
document_id: string;
text: string;
chunk_index: number;
page_number?: number | null;
section_heading?: string | null;
hierarchy_path: string[];
chunk_source: "agentic" | "programmatic" | "reconciled";
raptor_level: number;
is_leaf: boolean;
parent_ids: string[];
children_ids: string[];
confidence: number;
divergence_notes?: string | null;
timestamps: TimestampComponent;
tenant: TenantComponent;
provenance: ProvenanceComponent;
}
Where this model is read, written, and validated across the substrate. Click a node to navigate.
Reads from · upstream
←AssetModel · text content
←extract.docling@v2 · structural parser
←refinery.chunking@v3 · RAPTOR pipeline
Writes to · downstream
→ChunkEmbeddingModel · 8× per chunk
→ChunkNLPModel · entities + claims
→RetrievalResult · referenced by id
→EpisodicNode · provenance chain
14 versions since the model was introduced. Every breaking change goes through a deprecation period.
v7 · 2 DAYS AGO
Added
divergence_notesCaptures inline-text explanation when agentic and programmatic chunking disagree. Populated by the reconciler.
+ divergence_notes: Optional[str] = None
v6 · 2 WEEKS AGO
Split into composable ECS components
Pulled timestamps, tenant, and provenance into named ECS components instead of inline fields. Breaking change with deprecation alias.
- created_at, updated_at
+ timestamps: TimestampComponent
+ timestamps: TimestampComponent
v5 · 6 WEEKS AGO
Added
chunk_source enumDual-hierarchical RAPTOR landed. Every chunk now declares which pass produced it.
+ chunk_source: Literal["agentic", "programmatic", "reconciled"]
v4 · 12 WEEKS AGO
Added RAPTOR ancestry ·
parent_ids / children_idsLets summary chunks reference the leaves they summarize. Enables Horizontal retrieval axis.
v3 · 24 WEEKS AGO
Confidence float · added
Agents now report their confidence in the chunk boundary. Downstream consumers can filter by threshold.
Read by · 12 workflows · 7 production endpoints
retrieval.fanout@v4
reads · text, hierarchy_path, parent_ids · 2.2k calls/24h
profile_gen.cluster@v4
reads · text, confidence, divergence_notes · 42 calls/24h
/retrieve/v2 (API)
reads · all fields · 22k calls/24h
Written by · 4 workflows
ingestion.markdown@v3
writes · all required fields on ingest
refinery.rechunk@v2
writes · new versions on re-chunk
refinery.boundary_reconcile@v1
writes · divergence_notes, chunk_source="reconciled"
Validated at · 14 entry points
/api/chunks · POST handler
strict validation · rejects on schema drift
Convex chunks.insert
soft validation · logs drift to telemetry