File and Media Handling

The Problem with Simple Approaches

File and media handling is one of the few areas where naive implementations work correctly — until they don't. Loading a file into memory, processing it, saving it. This works for small files under light load. It fails predictably under large files, concurrent uploads, or in environments where memory and disk are constrained.

The failures are not immediate. A service that handles 1MB uploads correctly for months may encounter a 200MB upload and exhaust available memory. Or handle ten concurrent 50MB uploads and run out of memory from the accumulation. Or quietly fill ephemeral container disk with files that are never cleaned up, until the disk is full and the service stops responding.

Buffering Entire Files in Memory

The most common form. Reading the full file content before doing anything with it:

# Naive: entire file loaded into process memory
@router.post("/upload")
async def upload_file(file: UploadFile):
    content = await file.read()   # ← full file in memory
    await storage.save(content)
    return {"status": "uploaded"}

For a single 100MB file, this allocates 100MB in the process. Five concurrent uploads allocate 500MB. Under unexpected load or large files, memory is exhausted and the process crashes.

The streaming alternative reads and forwards data in chunks, keeping only a small buffer in memory regardless of file size:

# Streaming: bounded memory usage regardless of file size
@router.post("/upload")
async def upload_file(file: UploadFile):
    await storage.stream_upload(
        source=file,
        chunk_size=64 * 1024,  # 64KB chunks
    )
    return {"status": "uploaded"}

The same principle applies to downloads. Streaming directly from object storage to the response, rather than loading the full file into memory first.

Missing File Size Limits

Without an explicit limit, a client can send an arbitrarily large file. The application attempts to receive and process it, consuming memory and bandwidth proportionally.

Limits should be enforced at the boundary, before the body is read:

MAX_UPLOAD_BYTES = 50 * 1024 * 1024  # 50MB
 
@router.post("/upload")
async def upload_file(
    file: UploadFile,
    content_length: Optional[int] = Header(None),
):
    if content_length and content_length > MAX_UPLOAD_BYTES:
        raise HTTPException(413, "File exceeds maximum allowed size")
 
    # Also enforce during streaming — Content-Length can be absent or wrong
    await storage.stream_upload(file, max_bytes=MAX_UPLOAD_BYTES)

Checking Content-Length is a fast pre-check but not sufficient alone — clients can omit it or send chunked transfers without it. The limit must also be enforced during the actual read.

Ephemeral Disk Without Cleanup

Writing files to local disk in containerised or serverless environments treats ephemeral storage as permanent. Containers have a fixed disk allocation. Files written to /tmp or local paths persist for the container's lifetime, accumulating until disk is full.

Symptoms: services that work correctly for days or weeks, then fail with disk-full errors. Particularly common with temporary files created during processing — thumbnail generation, format conversion, compression — that are not explicitly deleted after use.

import tempfile
 
async def process_upload(file: UploadFile) -> ProcessedResult:
    # Temporary file is automatically deleted when the context exits
    with tempfile.NamedTemporaryFile(delete=True, suffix=".tmp") as tmp:
        await stream_to_file(file, tmp.name)
        result = await run_processing(tmp.name)
    # File is gone here regardless of success or failure
    return result

For authoritative storage, use object storage (S3, GCS, Azure Blob Storage). Local disk should only be used for transient processing, and always with explicit cleanup.

Blocking the Event Loop with File Operations

In async services, CPU-intensive file operations — image resizing, format conversion, compression, virus scanning — block the event loop if called directly. While one request is processing a large file, other requests wait.

Large or CPU-intensive operations should be offloaded to a background worker:

# In the route: accept the upload, enqueue, return immediately
@router.post("/media")
async def upload_media(file: UploadFile):
    file_key = await storage.stream_upload(file)
    await queue.enqueue(ProcessMedia(file_key=file_key))
    return {"status": "processing", "key": file_key}
 
# In a worker: runs independently, does not block request handling
class ProcessMediaHandler:
    async def handle(self, event: ProcessMedia) -> None:
        async with storage.open(event.file_key) as stream:
            processed = await executor.run(self._process, stream)
        await storage.save_processed(event.file_key, processed)

The Practical Checklist

Read Next

• Missing Resource Limits and Segregation — Protecting service health when file operations are a component of the load
• Event-Driven Patterns — Background workers and async processing via domain events