Building Agentic AI Data Foundations

The transition from static machine learning models to autonomous, multi-agent architectures fundamentally rewrites the rules of enterprise data engineering. Traditional databases were designed for applications controlled by human users or deterministic business logic. In contrast, an agentic AI data foundation is designed for systems where the agent itself is the primary consumer of data. To effectively support orchestration frameworks like LangGraph and CrewAI, organizations must evolve their data stacks from simple retrieval endpoints into dynamic, stateful ecosystems.

First Principles: Three Constraints That Drive Everything Else

Before reaching for architecture diagrams, it is worth deriving the requirements rather than inheriting them. Almost every design decision below falls out of three facts about agents that do not apply to human users.

1. The consumer is a program, not a person. A human given ten loosely relevant search results will skim, discard the noise and find the right one. An agent takes the top-k literally and reasons over whatever it is handed. Recall without precision is not a minor inconvenience; it is a corrupted premise.

2. The model forgets everything between runs. Nothing survives inside the model itself. Anything that must persist has to be written down outside it, which turns memory from a convenience feature into a storage design problem with different latency and consistency needs per layer.

3. The agent acts. A bad retrieval for a human is a wasted minute. A bad retrieval for an agent with tool access is a wrong action taken against a real system. That is why access control and audit have to live at the infrastructure boundary rather than in a prompt.

Constraint two produces layered memory. Constraint one produces hybrid retrieval. Constraint three produces the governed mesh. The rest of this piece works through each in turn.

From RAG to Agentic Memory State

Retrieval-Augmented Generation (RAG) is rapidly evolving to become the core data foundation for agents. However, a standard vector database is insufficient for serving production RAG workflows. Agents require persistent memory, real-time context retrieval, governed tool access, and stateful operation across multi-agent workflows.

A production-grade agentic memory system must be segmented into distinct layers:

• Short-term memory: Stores the current conversational context, including the active session, recent exchanges, and the working state.

• Long-term semantic memory: Stores observations, facts, and user history that should persist across sessions. This data is typically stored as vector embeddings to enable semantic retrieval.

• Profile memory: Stores structured user attributes such as preferences, loyalty status, access rights, and explicit instructions the agent should always be aware of. This information is stored as structured key-value data for deterministic, low-latency retrieval.

The split is not stylistic. Each layer has a different tolerance for approximation. You never want a loyalty tier or an access right retrieved by cosine similarity.

from collections import deque

class AgentMemory:
    def __init__(self, vector_store, kv_store, window=20):
        self.short_term = deque(maxlen=window)  # session only, dropped on close
        self.semantic   = vector_store          # survives across sessions
        self.profile    = kv_store              # structured, exact, low latency

    def observe(self, role, content):
        self.short_term.append({"role": role, "content": content})

    def remember(self, text, metadata):
        # promote a durable fact out of the session into long-term memory
        self.semantic.upsert(text=text, metadata=metadata)

    def build_context(self, query, user_id, k=5):
        return {
            "profile":  self.profile.get(user_id),             # exact lookup
            "recalled": self.semantic.search(query, top_k=k),  # approximate
            "recent":   list(self.short_term),                 # verbatim
        }

Modernizing the Ingestion Pipeline

Traditional ETL pipelines are built to pull structured data into relational tables. The agentic ingestion pipeline must go much further, emphasizing the "Parsing" stage to handle multi-modal, unstructured documents.

• During parsing, the system must convert multi-modal, multi-format raw documents into clean, structured text representations, preserving original logical structure (headings, lists, tables) and metadata (author, date) as much as possible.

• The "Transform" stage then injects intelligence by converting raw text streams into the advanced materials needed for retrieval.

• This includes processes aimed at improving retrieval effectiveness, such as tree structure generation, knowledge graph (entity-relation) extraction, summary generation, and keyword extraction.

The detail that matters most is what you refuse to throw away. Structure and provenance discarded at parse time cannot be recovered at query time, and they are exactly what the retrieval filters need later.

def parse(document):
    """Flatten to text, but keep the structure the retriever will filter on."""
    blocks = layout_parser.extract(document)
    return [
        {
            "text":      b.text,
            "type":      b.type,          # heading | paragraph | table | figure
            "section":   b.section_path,  # ["Policy", "Section II", "Exclusions"]
            "page":      b.page,
            "source_id": document.id,
            "author":    document.author,
            "valid_from": document.effective_date,
        }
        for b in blocks
    ]

def transform(blocks):
    """Turn raw text into retrieval material. This is where the cost sits."""
    for chunk in chunk_by_section(blocks, max_tokens=512):
        chunk["summary"]   = summarise(chunk["text"])
        chunk["keywords"]  = extract_keywords(chunk["text"])
        chunk["entities"]  = extract_entities(chunk["text"])   # feeds the graph
        chunk["embedding"] = embed(chunk["summary"] + "\n" + chunk["text"])
        yield chunk

Hybrid Indexing and Graph Retrieval

Agents need pinpoint accuracy that basic similarity searches cannot guarantee. A robust indexing strategy must support hybrid retrieval, combining vector, keyword, and metadata filtering. Filtering based on attributes like date, department, and type is a rigid requirement in business scenarios.

To achieve this, architectures are adopting a Hybrid Indexing Scheme: building full-text and single-vector indices for text content, alongside separate tensor indices for images, and fusing the results from the three indices during retrieval. Furthermore, integrating GraphRAG extracts entities and relationships from documents to build a knowledge graph, using graph queries and reasoning to discover indirectly related information fragments.

Fusion is the part teams underestimate. Reciprocal rank fusion is a reasonable default because it needs no score calibration between indices that measure completely different things.

def hybrid_search(query, filters, k=10):
    """Metadata filters are hard constraints. The three indices only rank."""
    vector_hits  = vector_index.search(embed(query), top_k=50, filters=filters)
    keyword_hits = text_index.search(query, top_k=50, filters=filters)
    graph_hits   = graph_index.neighbours(extract_entities(query), hops=2)
    return rrf([vector_hits, keyword_hits, graph_hits])[:k]

def rrf(result_lists, k=60):
    """Reciprocal rank fusion: rank-based, so no score calibration needed."""
    scores = {}
    for results in result_lists:
        for rank, doc in enumerate(results, start=1):
            scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)  # ordered doc ids

The Governed Agentic Mesh

As enterprises deploy multi-agent systems in which specialized agents hand off work to one another, shared state becomes the connective tissue. Without a common data foundation, agents duplicate work, drop context at handoff points, and clash on objectives.

By leveraging unified data platforms like Databricks with Unity Catalog or Microsoft Fabric's OneLake, organizations can implement a medallion architecture, enabling agents to ground responses in validated sources. Furthermore, exposing these capabilities via high-performance APIs (such as FastAPI) and adopting the Model Context Protocol (MCP) ensures agents have consistent, standard access to the data mesh.

Ultimately, governance must be enforced at the infrastructure level, not as an afterthought. Role-based access control, field-level permissions, and audit logging ensure that when an agent accesses data or triggers external tools, it operates safely and predictably at enterprise scale. Concretely, that means the permission check sits inside the tool boundary, where an agent cannot talk its way past it.

def agent_tool(required_scope):
    """Scope check, audit and field masking at the boundary, not in a prompt."""
    def wrap(fn):
        def call(ctx, **kwargs):
            if required_scope not in ctx.agent.scopes:
                audit.write(ctx, fn.__name__, kwargs, outcome="denied")
                raise PermissionError(f"{ctx.agent.id} lacks {required_scope}")
            rows = fn(ctx, **kwargs)
            audit.write(ctx, fn.__name__, kwargs, outcome="ok", n=len(rows))
            return mask_fields(rows, ctx.agent.field_permissions)
        return call
    return wrap

@agent_tool(required_scope="claims.read")
def fetch_claim_history(ctx, policy_id: str, since: str):
    return warehouse.query(
        "SELECT * FROM gold.claims WHERE policy_id = ? AND filed_at >= ?",
        [policy_id, since],
    )

An MCP server is essentially this decorator, standardised and moved out of process, so every agent reaches the same data through the same enforced boundary.

Build the memory layers first, because everything else assumes them. Add hybrid retrieval when precision starts costing you wrong actions. Put the governed boundary in before the second agent ships, not after.

References

• Berkeley AI Research / Databricks: The Shift from Models to Compound AI Systems

• Microsoft Research: GraphRAG: Unlocking LLM discovery on narrative private data

• arXiv Preprint: MemGPT: Towards LLMs as Operating Systems (Evaluating Agentic Memory)

• Anthropic / Protocol Labs: Introduction to the Model Context Protocol (MCP) for standardized agent data access