Missing Resource Limits and Segregation

Why the absence of proactive limits makes services slower to recover — and how resource segregation protects cheap operations from expensive ones.

The Counterintuitive Truth

Adding resource limits to a service feels like restricting its capability. In practice, limits protect the service's ability to function at all.

A service without limits will accept work until it cannot accept any more. When that point is reached, the service does not gracefully shed load — it degrades. Requests slow as resources contend. The queue of pending work grows. The service, now under maximum pressure, must drain the accumulated backlog before it can return to normal. That recovery can take many times longer than the original saturation event.

A service with proactive limits rejects work that exceeds capacity immediately, with a meaningful response to the caller. The service never reaches saturation. It continues processing within-capacity requests at normal speed throughout.

What Happens Without Limits

Database connection pool exhaustion. A connection pool has a fixed number of connections. Without concurrency controls, a traffic spike exhausts the pool. New requests queue waiting for a connection. Latency climbs. Each slow request holds its connection longer, further starving the pool. Recovery requires the backlog to drain — which is slow because every queued request is now competing for depleted connections.

Memory exhaustion. Without payload size limits or concurrency caps on memory-intensive operations, a burst of large requests exhausts process memory. The OS begins swapping. Performance collapses. A process restart loses all in-flight work and starts cold into the same traffic pattern that caused the saturation.

CPU saturation. CPU-intensive operations without concurrency limits will peg the processor when enough simultaneous requests arrive. The service continues accepting new work it cannot process promptly, growing the backlog. Each new request makes the situation worse.

In each case: the service accepted the work, attempted to do it, failed to do it well, and then must recover from the attempt.

What Proactive Limits Provide

Immediate feedback to callers. A 429 Too Many Requests or 503 Service Unavailable returned in milliseconds tells the caller: back off, retry later. Retry logic in the caller can handle a fast rejection. An indefinitely slow response cannot be handled — the caller also degrades while waiting.

Predictable service behaviour. A service operating within its limits behaves consistently. Latency is bounded. Resource usage is observable. The service's health is understandable.

No recovery phase. A service that never exhausts its resources never needs to recover. When a traffic spike hits the limit, it is rejected cleanly. When the spike passes, normal traffic resumes. There is no backlog to drain, no restart, no cold start.

The difference in recovery time between these two modes is significant. A service that hit hard limits and crashed can take ten times longer to return to normal operation than a service that shed excess load at a limit boundary.

Types of Limits

Connection pool limits. Set a maximum pool size and a connection acquisition timeout. Requests waiting beyond the timeout receive a fast error rather than an indefinite queue.

Request concurrency limits. Cap the number of simultaneously in-flight requests, or in-flight requests for specific expensive operations. Cheap operations should not share a concurrency budget with expensive ones.

Payload size limits. Reject oversized request bodies before they are read. See File and Media Handling for implementation detail.

Rate limits per client. Prevent a single client from consuming a disproportionate share of capacity. Protects all clients equally and simplifies capacity planning.

Worker queue depth limits. Background workers should cap how many items are processed simultaneously, rather than consuming all queued work at once. An unbounded worker under a large queue has the same exhaustion dynamics as an unbounded request handler.

Resource Segregation

A related principle: different classes of work should not compete for the same resource pools.

A service handling both high-volume cheap operations (reading a user profile) and low-volume expensive operations (generating a report, processing an upload) creates contention when they share the same connection pool and thread pool. A burst of expensive operations starves cheap operations — degrading the experience for all users, even those making trivial requests.

Segregation allocates separate resource pools to different operation classes:

Segregation level	Mechanism
Separate connection pools	Expensive operations draw from a dedicated, lower-cap pool; cheap operations use the main pool
Separate worker processes	Background and batch tasks run in dedicated workers, not inline with request handling
Separate queues	High-priority operations in one queue, bulk operations in another, with independent consumers
Separate services	At sufficient scale, expensive operations become a separate deployable with independent scaling

The key property: a surge in expensive operations does not affect the latency of cheap operations. Limits and segregation enforce this boundary structurally rather than relying on traffic being well-behaved.

Limits as a Contract

A resource limit is not an apology for insufficient infrastructure. It is a contract with callers: the service guarantees reliable, bounded-latency responses for work it accepts, and it communicates clearly when it cannot accept more. This is more useful behaviour than accepting all work and delivering slow, unpredictable responses.

Infrastructure with fixed allocations — containers with memory limits, databases with connection ceilings, thread pools with fixed sizes — is the norm. Designing services that operate correctly within those bounds, and that fail fast when bounds are approached, is a prerequisite for reliability.