Data Readiness for GenAI

Teams shipping enterprise GenAI to production consistently identify one blocker as the largest and most persistent: the data platform. Not the model, not the framework, not the guardrail vendor — the data. Datasets are not findable, ownership is unclear, quality is unmonitored, access is manual, PII handling is per-app, retrieval infrastructure is fragile, and compliance evidence is un-producible.

This whitepaper is organized around the five sub-dimensions of data readiness for GenAI, matching the free /tools/data-readiness assessment. Each covers the specific investments that pay disproportionately relative to their cost — the "if we had done this six months ago, we would be a full maturity level ahead" investments.

Data-platform work compounds. Every dataset you catalogue unblocks every future use case. Every guardrail you push down to the platform layer removes duplicate work across every application. Under-invest here and every new GenAI initiative pays the same organizational tax again.

The AI system inventory (see the Governance Playbook) requires a data inventory beneath it. Without knowing what data you have, who owns it, and where it flows, every new GenAI project starts with the same discovery sprint.

The catalog captures: dataset name, description, owner (named role, not just team), classification tier, freshness SLA, downstream consumers, and the schema. Modern catalog tools (Collibra, Atlan, DataHub, Alation) auto-ingest from most cloud data warehouses. The tooling is not the hard part; the sustained discipline of keeping metadata current is.

Column-level lineage matters more for GenAI than for BI. When a downstream RAG system returns an incorrect answer, tracing the answer back through the retrieval chunk, to the source document, to the transformation pipeline that produced it, is the difference between a five-minute triage and a five-day investigation.

Training-corpus provenance is now regulatory-relevant. Under the EU AI Act, providers of general-purpose AI models must publish a training-content summary (Article 53). Even if you are a deployer rather than a provider, downstream questions about copyright-status and license-compliance of training data will land. Track corpus provenance from ingest — retrofitting it later is often impossible.

Retrieval systems amplify data quality issues. A merely-noisy source produces a merely-noisy report in BI; the same source in RAG can produce a confident-sounding wrong answer that a user acts on.

Two disciplines that separate mature data-quality practice from occasional-check practice:

Automated data-quality tests in the pipeline, gated by CI. Great Expectations, Soda, dbt tests, or the equivalent — every critical table gets tests for schema conformance, freshness, uniqueness constraints, referential integrity, and value-range plausibility. Tests fail loud (paging, dashboard) and gate downstream execution.

Retrieval-quality metrics for RAG systems. Recall@k and MRR (mean reciprocal rank) on a labelled query set, run on a schedule against a live index. Combined with downstream generation-quality metrics (faithfulness, groundedness), this catches the case where the retriever silently starts returning less-relevant chunks after a data or index change.

Metric-definition drift is a subtler quality issue: the same metric name computes differently in different tools, and the LLM cheerfully answers with whichever version it happened to retrieve. A semantic layer (Cube, dbt Metrics, LookML) with enforced metric contracts is the reliable fix. Ad-hoc metric dictionaries in wikis do not survive contact with an urgent business question.

The tension in access design is between security (least privilege, auditable, revocable) and velocity (self-service, propagated automatically, no ticket queue). Both are achievable together with the right architecture; separately they are not.

Attribute-based access control (ABAC) at query time. Access policies expressed as attributes (user role, project, geography, data classification) evaluated when a query runs, not when a user is provisioned. New team members inherit access immediately based on role assignment; access decays automatically when the role changes.

Access-control propagation into the vector store. RAG systems commonly expose a broader access surface than the source data because the vector store lives adjacent to the app rather than inside the governance perimeter. Retrieval should filter by user-attribute at query time — retrieving only chunks the user is authorized to see. Retrofitting this is expensive; design it in from the first vector-store deployment.

Data-clean-room patterns for development. Copying production data to development environments is a persistent violation. Synthetic datasets derived from production statistics + governed access to production-view-only environments for dev workloads eliminate the class of "someone put PII in a dev database" incidents.

Agent authentication as first-class. LLM agents accessing data on a user's behalf must authenticate as that user, not as a shared service account. User-scoped OAuth flows (or MCP-with-OAuth-2.1) provide the audit trail and least-privilege enforcement that shared credentials cannot.

Vector databases commoditized in 2026. The choice of Pinecone, Weaviate, Qdrant, pgvector, Milvus (or Chroma, LanceDB, etc.) is driven by operational profile rather than capability differentiator. See /vs/vector-databases for the full comparison.

Design decisions that matter more than product selection:

Hybrid retrieval as default. Pure vector similarity underperforms in the presence of exact-match queries (proper nouns, IDs, technical terms). Hybrid — vector plus BM25 or full-text — with cross-encoder re-ranking is the reference pattern. Ship without it only for prototypes.

Chunk-and-embed strategy. How you chunk source documents dramatically affects retrieval quality. Semantic chunking (splitting at meaning boundaries, not arbitrary character counts) with 20-30% overlap performs better than naive fixed-size chunking on most enterprise document types. Reserve engineering effort for this rather than for switching vector databases.

Freshness architecture. Full re-index on a nightly schedule is the default; event-driven incremental updates on source change are the mature pattern. For workloads where staleness is unacceptable (compliance-critical documents, live pricing, current customer state), event-driven is not optional.

Metadata filtering. Retrieval that supports rich metadata filters (author, date, department, classification) enables per-user access controls, temporal queries, and topic scoping without contaminating the vector semantics. All mainstream vector DBs support this; use it.

Temporal RAG for compliance-sensitive corpora. When "what was the state of this document on date X" is a common query (compliance, audit, legal), track document versions in the retrieval layer. Time-travel retrieval is available in Weaviate, custom-buildable in most others.

The compliance obligations that touched enterprise data platforms in 2024-2025 (GDPR, CCPA, sector regulations) apply with additional urgency to GenAI-adjacent data. The EU AI Act adds new obligations on top.

PII detection and redaction at platform layer. Not per-app. A platform-layer service that scans inputs and outputs against configurable classifiers, applies redaction policies per data class and per tenant, and logs redaction events for audit. Per-app implementations decay to inconsistent enforcement quickly.

Vector-store deletion for data-subject rights. GDPR Article 17 (right to erasure) and CCPA equivalent apply to embeddings derived from personal data. The vector store must support per-record deletion and the DSR pipeline must propagate to it. Retrofitting this is possible but painful; design it in.

Cross-border transfer controls. For EU personal data, the standard contractual clauses + Transfer Impact Assessment framework applies. GenAI complicates it because inference may route through provider infrastructure in unexpected jurisdictions. Automate the routing decision: personal data classified above a threshold routes to EU-hosted inference; below it, any region is acceptable.

Copyright compliance for training corpora. For any in-house fine-tuning or model training, source material must be licensed for the intended use. For hosted models, this is the provider's obligation, but you may be asked for evidence in a downstream audit. Track corpus licensing metadata from ingest.

Documentation ready for audit. Regulatory auditors ask for training-data documentation, evaluation methodology, model performance metrics, and incident history. Producing these on demand — rather than assembling them in a two-week fire drill after a request — separates the mature program from the merely-compliant program. Automate documentation generation from your CI + evaluation harness output.

Data-platform investments compound, but they also have a natural order:

First: Catalog + named ownership + access controls. Without these, everything else is guesswork.

Second: Automated data-quality tests on the datasets feeding your first production GenAI systems. Failures here surface early; failures downstream surface catastrophically.

Third: PII detection and redaction at platform layer. As soon as you have more than one GenAI application, per-app implementations will diverge.

Fourth: Shared vector-store infrastructure with hybrid retrieval and metadata filtering. Do this once, well, and every future RAG system inherits it.

Fifth: Automated DSR propagation, compliance-documentation generation, cross-border routing enforcement. Regulatory obligations catch you eventually; automating them removes the periodic fire-drill tax.

Teams that skip earlier steps to jump ahead reliably regret it. The team that ships a fifth RAG system on ad-hoc chunk-and-embed infrastructure is paying the setup cost five times. The team that shipped shared infrastructure after the second RAG system pays it once.

- /framework/dimensions/data — the framework dimension this whitepaper operationalizes. - /tools/data-readiness — 30-question self-assessment. - /vs/vector-databases — Pinecone / Weaviate / Qdrant / pgvector / Milvus comparison. - /vs/rag-vs-fine-tuning — when RAG is right vs when fine-tuning is right. - /glossary/retrieval-augmented-generation-rag, /glossary/graphrag, /glossary/vector-database, /glossary/embedding — foundational terms.