Architecture

Status: Living Document Last Updated: 2026-03-30

---

Overview

A complete AI stack has eight primary layers plus one cross-cutting governance layer. The layers are described here tool-independently - the purpose of each layer is stable regardless of which specific model, framework, or cloud service occupies it. See Reference Stacks for how specific tools map to these layers.

The layers are ordered bottom-up (infrastructure at the foundation, application at the top), reflecting the dependency direction: each upper layer depends on the layers beneath it.

---

Layer 1 - Infrastructure

Purpose: Provide the raw compute, memory, networking, and serving infrastructure needed to run model inference at acceptable latency and cost.

What lives here:

• GPU/TPU/ASIC compute resources
• Model serving infrastructure (endpoints, load balancing, autoscaling)
• Networking and storage
• Container orchestration (Kubernetes, etc.)
• Cost management and usage metering

Key properties to design for:

• Inference-first, not training-first. Production AI stacks are dominated by inference cost, not training cost. Design serving infrastructure for this.
• Latency budgets. Each layer above adds latency; the infrastructure layer sets the floor. Know your acceptable end-to-end latency and work backwards.
• Tiered serving. Expensive large models for planning and reasoning; cheaper small models for execution steps. Infrastructure should support routing between tiers.

Common failure modes: Over-provisioning for peak (expensive), under-provisioning for normal load (reliability issues), single-model serving architectures that cannot accommodate model tiering.

---

Layer 2 - Data & Knowledge

Purpose: Make relevant context available to models in queryable, structured form. This layer is the foundation of RAG (Retrieval-Augmented Generation) and determines the quality of what the model knows.

What lives here:

• Document stores and knowledge bases
• Vector databases (semantic search over embeddings)
• Structured databases and data lakes
• Ingestion pipelines (crawlers, ETL, chunking, embedding)
• Retrieval logic (dense, sparse, hybrid retrieval)
• Knowledge graphs

Key properties to design for:

• Freshness. Knowledge has a shelf life. Design for how and how often context is updated.
• Retrieval quality. The best model cannot compensate for poor retrieval. Precision and recall of retrieved chunks matter as much as model capability.
• Chunking strategy. How documents are split and embedded significantly affects retrieval quality.
• Source authority. Not all retrieved content is equally reliable. Design for source weighting and citation.

Common failure modes: Stale knowledge bases, poor chunking producing irrelevant chunks, no source attribution, treating retrieval as an afterthought.

---

Layer 3 - Foundation Models

Purpose: Provide general-purpose reasoning, language understanding, instruction following, and generation capability. This is the "brain" of the stack.

What lives here:

• Large language models (frontier or open-weight)
• Domain-specific language models (DSLMs) for specialised tasks
• Multi-modal models (vision, audio, code)
• Fine-tuned or RLHF-aligned variants

Key properties to design for:

• Context window. Determines how much can be held "in mind" per reasoning pass.
• Instruction following. The reliability with which the model follows complex, structured prompts.
• Tool use / function calling. Native ability to invoke tools; central to the execution layer above.
• Model tiering. The same task may be handled by a capable (expensive) model for planning and a smaller (cheap) model for execution. Design for this from the start.

Common failure modes: Assuming one model does everything (usually too expensive or too limited), ignoring instruction-following reliability differences between models, treating the model layer as the whole stack.

---

Layer 4 - Orchestration

Purpose: Coordinate multiple agents, models, and tools. Route tasks to the right component, manage state across multi-step workflows, and handle failures gracefully.

What lives here:

• Agent routing and task delegation logic
• Workflow state machines and graphs
• Multi-agent communication protocols
• Error handling and retry logic
• Parallelism and sequential execution control
• Context passing between agents

Key properties to design for:

• State management. Multi-step workflows need state; the orchestration layer owns it.
• Idempotency. Agent actions may fail and be retried; design orchestration so retries are safe.
• Failure modes. What happens when an agent returns an unexpected result, times out, or loops? The orchestration layer must handle this explicitly.
• Observability. Every orchestration decision should be logged - which agent was called, with what input, and what it returned.

Common failure modes: No state management (every step restarts from scratch), unhandled failures causing silent corruption, unobservable orchestration making debugging impossible.

---

Layer 5 - Memory

Purpose: Enable agents to learn from history, maintain continuity across sessions, and apply accumulated knowledge without re-retrieving it from scratch every time.

What lives here:

• Short-term / working memory - the active context window; what the agent currently knows
• Long-term memory - persistent storage of learned facts, user preferences, past interactions (vector store or database-backed)
• Episodic memory - structured records of past tasks, their inputs, and their outcomes
• Policy/procedural memory - reusable plans, strategies, and governance rules

Key properties to design for:

• Memory eviction. Context windows are finite; decide what to keep in working memory vs. what to retrieve from long-term storage.
• Memory reliability. Long-term memory retrieved and acted upon should be verified against current state, not assumed current.
• Privacy and scope. Memory from one user or session should not leak into another.

Common failure modes: Treating each interaction as stateless (agents repeat work), unbounded context growth (costs escalate), no distinction between what was known at time T vs. now.

---

Layer 6 - Execution

Purpose: Translate agent decisions into real-world actions - call APIs, run code, manipulate files, browse the web, send messages. This is where agents make things happen.

What lives here:

• Tool definitions and schemas (what actions are available)
• Tool invocation and response parsing
• Code execution environments (sandboxed runners)
• API integrations (third-party services, internal APIs)
• File system and browser access
• MCP servers (standardised tool exposure)

Key properties to design for:

• Sandboxing. Agents with execution capability can cause real harm. All code execution and file manipulation should be sandboxed.
• Tool schemas. The quality of tool descriptions directly affects how reliably agents invoke them correctly. Invest in clear, unambiguous schemas.
• Reversibility. Prefer reversible actions where possible. Make irreversible actions (deletion, sending, publishing) require explicit confirmation.
• Rate limiting and cost controls. Agents can call APIs in tight loops. Design for usage caps and circuit breakers.

Common failure modes: No sandboxing on code execution, ambiguous tool schemas leading to incorrect invocations, no cost controls on external API usage, no distinction between reversible and irreversible actions.

---

Layer 7 - Planning & Reasoning

Purpose: Decompose goals into actionable strategies, reason about dependencies and constraints, and adapt when intermediate results change the plan. This is where the agent exhibits "thinking."

What lives here:

• Goal decomposition logic (breaking high-level goals into steps)
• Extended thinking and strategy prompting
• Constraint reasoning (what cannot be done, in what order)
• Re-planning when earlier steps fail or produce unexpected results
• Strategy selection (which approach to take for a given problem type)

Key properties to design for:

• Explicit planning vs. implicit. Some architectures plan explicitly (write a plan, then execute it); others reason step-by-step implicitly. Explicit planning is more observable and debuggable.
• Plan validation. Before executing a plan, validate it against constraints (permissions, cost, reversibility).
• Graceful degradation. When a plan fails partway through, the agent should recover to a safe state, not continue with stale assumptions.

Common failure modes: No separation between planning and execution (mixed concerns), plans not validated before execution, no recovery path when mid-plan steps fail.

---

Layer 8 - Application / Agent

Purpose: Expose agent capabilities to end users or consuming systems. This layer defines the user experience, the invocation interface, and the scope of what users can ask the agent to do.

What lives here:

• User interfaces (chat, IDE integration, CLI, web apps)
• API surfaces for programmatic consumption
• Agent personas and scope definitions (what this agent does and does not do)
• Session management and authentication
• Output formatting and presentation

Key properties to design for:

• Scope constraints. Agents should have clearly defined scope - what they can act on and what is out of bounds. Unbounded scope is a safety and cost risk.
• Transparency. Users should be able to see what the agent did, not just its final output.
• Graceful limits. When the agent cannot or should not do something, it should communicate this clearly rather than attempting and failing silently.

Common failure modes: No scope constraints (agent attempts anything asked), no transparency into actions taken, identity and session scope not cleanly separated.

---

Layer 9 - Governance (cross-cutting)

Purpose: Ensure security, regulatory compliance, cost control, auditability, and safety across all other layers. Governance is not a layer you add at the end - it must be designed in from the start.

What lives here:

• Audit logging (who asked what, what the agent did, what it accessed)
• Cost tracking and budget enforcement
• PII detection and data handling compliance
• Rate limiting and abuse prevention
• Safety filters and output moderation
• Policy enforcement (what agents are allowed to do)
• Incident response and rollback capability

Key properties to design for:

• Comprehensive audit trail. Every agent decision, tool invocation, and data access should be logged with enough context to reconstruct what happened.
• Cost visibility. Multi-agent systems can generate surprising bills. Per-task cost attribution and real-time budget alerts are essential.
• Data handling. Know what data is in context at every step and whether it should be there.

Common failure modes: No audit trail (cannot explain what happened), no cost controls (surprise bills), compliance requirements added as an afterthought, no incident response path when an agent does something unexpected.

---

Related

• What is an AI Stack? - what an AI stack is and why it matters
• Trade-offs - local vs cloud, single vs multi-agent, structured vs autonomous
• Reference Stacks - how these layers map to specific tools and frameworks
• Our View - how we currently think teams should approach stack design