Learn / AI Stack
Key Trade-offs
The main design decisions teams run into: local vs cloud, single-agent vs multi-agent, structured workflows vs autonomy, model fleets, and async execution.
Status: Living Document Last Updated: 2026-03-30
Overview
The AI stack is a young discipline and several architectural decisions do not yet have settled answers. These are the active debates worth understanding before committing to a design - because the choice made early in each one shapes the rest of the stack significantly.
1. Local vs. Cloud Execution
Should agent execution - particularly tool use, code running, and long-running tasks - happen on local infrastructure or in cloud-hosted sandboxed environments?
The local case:
- Lower latency for inference-heavy tasks with no network round-trip
- Zero marginal cost after hardware investment; better economics above ~5,000 prompts/month
- Full data privacy - nothing leaves your environment
- Complete control; no third-party monitoring of agent behaviour
The cloud case:
- Sandboxed: agents run in dedicated VMs with no access to local filesystem or credentials
- Scales horizontally without provisioning
- Cheaper at low usage volume; no upfront hardware cost
- Parallel execution across many cloud VMs simultaneously
The emerging consensus (2026): Hybrid. Local for sensitive data and low-latency tasks; cloud for parallelism, isolation, and long-running autonomous work. The boundary is blurring as local tools add headless execution and cloud products add deeper workflow support.
Design implication: Do not design for one or the other exclusively. Design the execution layer so local and cloud adapters are interchangeable. The choice should be a configuration decision, not an architectural one.
2. Single-Agent vs. Multi-Agent Systems
Should one general-purpose agent handle the full task, or should work be delegated to multiple specialised agents coordinated by an orchestrator?
| Dimension | Single-Agent | Multi-Agent |
|---|---|---|
| Complexity | Low - one model, one context | High - orchestration, communication, state sync |
| Cost | Lower - one set of model calls | Higher - multiple agents running in parallel |
| Debugging | Easier - one flow to trace | Harder - emergent behaviour between agents |
| Performance (simple tasks) | Good | Worse - overhead for simple work |
| Performance (complex tasks) | Limited by context window and role confusion | Better - each agent optimised for its role |
| Adoption | Still the baseline for most teams | Growing fast where specialization and parallelism matter |
The case for starting single-agent: Most tasks do not require multi-agent coordination. Persona switching and context-aware policies can replicate much of the benefit without the orchestration overhead. The additional complexity of multi-agent systems is only worth it when you have identified specific limitations in single-agent testing.
The case for multi-agent: Complex, multi-domain tasks benefit from specialisation. Parallelism - running multiple agents simultaneously on independent sub-tasks - provides speed gains that single-agent cannot match.
The emerging consensus: Start single-agent. Identify the specific bottleneck (context window, role confusion, parallelism need) before splitting. When splitting, give each agent a narrow, well-defined scope and a clear interface.
3. Structured Workflows vs. Autonomous Execution
Should the agent follow a predefined workflow (structured), or should it decide its own steps at runtime (autonomous)?
| Structured (90% of use cases) | Autonomous (10%) | |
|---|---|---|
| Predictability | High - you know the steps | Low - agent chooses path |
| Cost | Low - fewer model calls, no replanning | High - reasoning at every step |
| Risk | Low - constrained action space | Higher - unexpected paths possible |
| Flexibility | Low - handles only expected inputs | High - handles novel situations |
| Debuggability | Easy - trace the fixed workflow | Hard - emergent behaviour |
| When to use | Well-defined tasks with known inputs | Tasks where control paths are hard to enumerate upfront |
The 90/10 principle: Design 90% of your agent workload as structured, predictable workflows. Reserve autonomous execution for the 10% where the input space or control paths genuinely cannot be predetermined. Most production teams that have tried purely autonomous agents have walked back to a more structured approach after reliability and cost problems.
Design implication: Build structured workflows first; make them composable. Add autonomy as a capability at the orchestration layer for specific task types, not as the default mode.
4. Monolithic Model vs. Fleet of Specialists
Should a single large model handle all reasoning, planning, and execution - or should a fleet of smaller, specialised models be coordinated for different roles?
The monolithic case:
- Simpler orchestration - one model context, no inter-model communication
- Better at tasks requiring broad contextual awareness across domains
- Faster to prototype and iterate
The fleet case:
- Cost reduction when expensive models are used for planning only and cheaper models for execution steps
- Each model optimised for its specific function (coding, summarisation, retrieval re-ranking)
- Failure isolation - a bad execution step does not corrupt the planning context
The plan-and-execute pattern: A capable model generates the plan and breaks it into steps; a cheaper or specialised model executes each step. The orchestration layer manages state between them. This is one of the most common fleet approaches in production systems.
Design implication: Design the model layer with tiering in mind from the start. Even if you start with one model, the orchestration layer should support routing different task types to different models. Switching later is significantly harder if the architecture assumed a single model throughout.
5. Synchronous vs. Asynchronous Agent Execution
Should the user wait for the agent to complete (synchronous), or should agents run in the background and notify when done (asynchronous)?
Synchronous:
- Simpler to reason about - the result is available when control returns
- Appropriate for short tasks
- Natural for conversational interfaces
Asynchronous:
- Required for long-running tasks
- Enables parallel execution across multiple agents simultaneously
- Requires a notification mechanism (webhooks, polling, push)
- Requires the application layer to handle in-progress and failed states explicitly
The direction: More production agent systems are moving toward async for substantial work. Tasks like implementing features, researching reports, or reviewing large changesets are naturally long-running. Synchronous execution can create poor UX, timeout problems, and fragile delivery.
Design implication: Design the application layer to handle async agent results from the start. Retrofitting async onto a synchronous architecture is disruptive.
Related
- What is an AI Stack? - what the term means and why it matters
- Architecture - the major layers of an AI stack
- Reference Stacks - how these debates show up in published systems
- Our View - where we currently land on these trade-offs
Related terminology
Use the terminology guide for the vocabulary around these pages, especially where products and teams use overlapping terms for agents, harnesses, workflows, tools, and controls.