Learn / AI Stack
What is an AI Stack?
Start here: what the term means, why it matters now, and how AI stacks relate to older ideas like RAG, LangChain, microservices, and MLOps.
Status: Living Document Last Updated: 2026-03-30
Definition
An AI stack is the layered architecture of infrastructure, data, models, orchestration, memory, execution, reasoning, and governance that together enable an AI system to perceive context, plan, act, and learn. Unlike a traditional software stack - which is largely deterministic - an AI stack contains components that reason, decide, and operate with degrees of autonomy.
The term gained traction in 2025-26 as teams moved beyond single-model integrations into multi-step, multi-agent systems. A single API call to a language model is not a stack; a system that retrieves knowledge, plans a course of action, delegates to specialised agents, executes against real APIs, and remembers the outcome is.
Why it matters now
Three shifts made the AI stack concept necessary:
1. From completion to action. Early LLM use was primarily text generation - summarise this, write that. Modern use involves agents that take actions: call APIs, write and run code, browse the web, send messages, open pull requests. This requires explicit infrastructure for execution, tool use, and safety.
2. From single model to orchestrated systems. The performance ceiling of a single model in a single prompt is real. Complex tasks - multi-step research, end-to-end code generation, multi-domain decision-making - require orchestrating multiple models, tools, and memory systems in sequence or in parallel.
3. From experiment to production. Production AI systems require the same properties as any production software: reliability, observability, cost control, security, and governance. These concerns require dedicated architectural layers, not afterthoughts.
Relationship to older concepts
The AI stack builds on and extends several prior architectural patterns:
| Prior concept | What it contributed | How the AI stack extends it |
|---|---|---|
| RAG (Retrieval-Augmented Generation) | Grounding model outputs in retrieved documents | Agentic RAG adds multi-step retrieval, tool use, and dynamic query refinement - the agent decides when and what to retrieve |
| LangChain / AutoGen | Connecting LLMs to tools, chains, and other agents | These frameworks are now one implementation option for the orchestration layer - not the whole stack |
| Microservices | Decomposing systems into focused, independently deployable units | Agent specialisation mirrors this: each agent has a defined role and interface, coordinated by an orchestration layer |
| MLOps / training pipelines | Data -> feature engineering -> train -> deploy | Modern AI stacks shift emphasis from training to inference, from batch to real-time, from deterministic to reasoning-driven |
| Workflow automation (n8n, Zapier) | Declarative steps chained together | Agentic workflows replace fixed step definitions with agents that decide the next action at runtime |
The layers at a glance
A full AI stack has 8-9 distinct layers (see Architecture for detail):
- Application / Agent layer - what users interact with
- Planning & Reasoning layer - goal decomposition and strategy
- Execution layer - tool calls, API actions, and code runs
- Memory layer - short-term, long-term, and episodic memory
- Orchestration layer - routing, state, and agent coordination
- Foundation Models layer - reasoning, language, and generation
- Data & Knowledge layer - RAG, vector stores, and knowledge bases
- Infrastructure layer - compute, serving, and networking
- Governance - a cross-cutting concern that touches every layer
Not every system needs every layer. A simple AI assistant might only use infrastructure, a foundation model, and an application layer. A production autonomous agent likely needs all of them.
The 90/10 operating principle
A useful heuristic from practitioners: 90% of AI workloads should use structured, predictable workflows; 10% should be genuinely autonomous.
Structured workflows are cheaper, faster, more reliable, and easier to test and audit. Autonomy is powerful but expensive, harder to reason about, and prone to unexpected paths. The stack should be designed so both modes are possible - and so the default leans structured.
Related
- Architecture - the major layers of an AI stack
- Trade-offs - local vs cloud, single vs multi-agent, structured vs autonomous
- Reference Stacks - GStack, Paperclip, five-layer models, and agentic RAG
- Our View - what we think matters most right now
Related terminology
Use the terminology guide for the vocabulary around these pages, especially where products and teams use overlapping terms for agents, harnesses, workflows, tools, and controls.