Fixing S3 Latency and Connection Pool Exhaustion at Scale
We had an export system that worked perfectly locally, but S3 latency and later connection pool exhaustion caused serious problems in production. Exporting hundreds of records with imagery, generating PDFs with embedded photos, KMZ files with plume overlays, and aerial photography mosaics, took 4-5 minutes on a local MinIO setup.
Then we deployed it.
Suddenly the same exports started taking hours.
Initially, we suspected S3 latency to be the problem, so we parallelized downloads, optimized concurrency, added semaphores, and tuned connection pools. Performance improved, but only up to a point.
Beyond a certain scale, a different class of problems started appearing like database pool exhaustion, worker starvation, connection resets, long-running monolithic jobs, and infrastructure-wide resource pressure. What started as an “optimize downloads” task slowly became a lesson in distributed systems design.
In This Article
-
What Caused Our S3 Latency Problem
-
Our First Assumption: S3 Latency Was the Bottleneck
-
Reducing S3 Latency With a Python Asyncio Semaphore
-
How Parallelism Caused Connection Pool Exhaustion
-
Why Unlimited Exports Break Distributed Systems Design
-
Fixing Connection Pool Exhaustion With Operational Limits
-
Rate Limiting Best Practices: What Salesforce and Stripe Do
-
The Fix: Semaphores, Connection Pooling, and Queue Fan-Out
-
The Real Lesson: Distributed Systems Design Beats Optimization
What Caused Our S3 Latency Problem
Our export system generates multiple formats for geospatial data like PDF reports with embedded imagery (a format we've written about handling complex PDF documents elsewhere), KMZ files for Google Earth visualization (with plume overlays and aerial photography), large CSV/Excel datasets, and TIFF imagery archives.
Query a database for asset or emissions data, retrieve associated images from object storage, compile into the requested format, and deliver a ZIP file.
Locally, against MinIO running on localhost, a large PDF export with imagery completed in 4-5 minutes. In production, running on ECS workers against S3 in a different region, the same export took over two hours. Sometimes it didn’t complete at all: timeouts, connection resets, mysterious failures.
The mismatch was dramatic, and worse, hard to reproduce. Local development masked multiple real bottlenecks: network I/O, cross-region latency, shared database connections, CPU/memory constraints, and concurrent workload patterns. But network I/O dominated so completely that the others stayed hidden.
We've written before about how deployment strategy changes production behaviour on AWS, and the gap between a local run and a production one is rarely just speed.
Our First Assumption: S3 Latency Was the Bottleneck
S3 latency becomes a bottleneck when an application makes many small, sequential requests to object storage across regions without a CDN in front of it. That was exactly our setup.
Looking at traces and logs, the culprit seemed obvious. We weren’t using CloudFront or any CDN. Every image was fetched fresh from S3, often across AWS regions. For exports with hundreds of items, that meant hundreds or thousands of individual S3 GetObject calls, each incurring tens or hundreds of milliseconds of latency.
We also noticed repeated downloads: the same plume TIFF or mosaic image being fetched multiple times when it appeared in multiple records. Deduplication went on the backlog; we tackled the obvious win first: parallelism.
Network I/O completely dominated export time. CPU was idle. Memory was fine. The workers were just sitting there waiting for bytes to arrive from S3.
In large export systems, object storage latency, rather than compute, is often the bottleneck.
Reducing S3 Latency With a Python Asyncio Semaphore
The solution seemed straightforward, parallelize the downloads. Our export code was originally synchronous, downloading one file at a time. We rewrote it using Python’s asyncio, replacing sequential download() calls with asyncio.gather() to run downloads in parallel.
But unbounded parallelism is dangerous. Opening hundreds of connections simultaneously would exhaust connection pools, trigger rate limits, or destabilize MinIO in local development. So we added a global semaphore to cap concurrency:
DOWNLOAD_SEMAPHORE = asyncio.Semaphore(8)
async def download_object(...):
async with DOWNLOAD_SEMAPHORE:
await asyncio.to_thread(client.download_file, ...)We enlarged the HTTP connection pool from the default 10 connections to 20, ensuring the semaphore wouldn’t be bottlenecked by connection availability. Important distinction: the pool allows up to 20 connections, but the semaphore caps actual concurrent downloads at 8. Since the semaphore is process-wide and workers handle up to 10 concurrent jobs, all exports share those 8 download slots.
The results were dramatic. Medium-sized exports, 50 to 200 assets with imagery, dropped from 45+ minutes to under 10 minutes.
Parallelism helped dramatically for medium-sized exports.
How Parallelism Caused Connection Pool Exhaustion
Connection pool exhaustion occurred because faster exports increased database activity while long-running workers continued holding database connections during extended S3 download operations.
Then we started seeing new problems. Exports were faster, yes, but that meant workers were now hammering the database more intensely. Database connection pools started saturating. Workers waiting for connections. Timeouts appearing in unrelated API requests.
Our database connection pool, sized for normal API traffic (5 connections per process, plus 10 overflow), became depleted during peak usage when long-running export jobs consumed all available connections, starving the API.
And “long-running” was the issue. The worker held a database connection for the entire job, from initial query through final upload, even during long stretches where it was just waiting for S3.
One concrete example: workers would hold Postgres connections idle during S3 download phases, which could span several minutes. Meanwhile, API requests for dashboards and real-time queries would queue waiting for an available connection, eventually timing out. We saw cascading failures where one very large export could make the entire platform sluggish for other users.
Worse, a single very large export could monopolize the infrastructure. One worker processing thousands of emissions with plume imagery, downloading gigabytes of data, would block other jobs from starting, hold a database connection idle during I/O, and leave other users’ exports queued indefinitely.
Optimizing one bottleneck exposed the next one.
Why Unlimited Exports Break Distributed Systems Design
We realized the problem wasn’t just performance. It was the shape of the workload. Our worker design was monolithic in terms of orchestration: one worker handles the entire export from start to finish. Query the database, download all the files, generate the output, ZIP it, upload to S3, send the notification email. If anything fails, the entire job fails and must retry from the beginning.
For large exports, this created compounding problems:
-
Memory pressure: Accumulating hundreds of images in temporary storage, even on disk, consumed worker memory and disk I/O.
-
All-or-nothing retries: If a job failed after 90% completion (a network glitch, a single corrupt file), the entire export message became visible in the queue again and had to start over from scratch. There’s no partial checkpoint.
-
Worker starvation: Other users’ smaller, faster exports queued behind a single massive job.
-
Resource hoarding: A worker processing a huge export held database connections, file handles, and compute capacity for the duration.
As it was workload shape, the issue was no longer just performance.
The same reasoning drives bounding a dependency before it exhausts your resources: a circuit breaker does for a failing third-party API what an export limit does for a greedy job.
Fixing Connection Pool Exhaustion With Operational Limits
Operational limits reduced connection pool exhaustion by setting boundaries based on database capacity, worker memory, network stability, and expected user wait times.
Initially, we made the counterintuitive decision to add hard limits. Instead of pursuing “unlimited exports,” we imposed operational boundaries. Limits vary by export type: imagery-heavy formats get tighter caps than tabular exports, and multi-format requests take the strictest limit across everything selected.
These weren’t arbitrary. They reflected the practical capacity of our infrastructure: database connection pools, worker memory limits, network stability, and reasonable user wait times.
We also added explicit database connection lifecycle management for I/O-heavy exporters. For exports dominated by S3 downloads (PDF mosaics, aerial photography, plume imagery, KMZ), we now close the database connection before beginning long download operations, then reconnect when needed:
# Release DB connection before long S3 download phase
if not db.is_closed():
await db.close()Tabular exports (CSV/Excel) keep the connection open for cursor-based streaming; that’s a different I/O pattern, and the connection is actively used. But for imagery-heavy paths, releasing the connection during long S3 phases dramatically reduced connection pool exhaustion and pool contention.
Mature systems are defined as much by their constraints as by their capabilities.
Rate Limiting Best Practices: What Salesforce and Stripe Do
When we started imposing limits, it felt like we were compromising. Like we were admitting defeat.
Then we looked at how the industry leaders handle this. Salesforce caps concurrent long-running API requests at 25 per production organization and times out all calls at 10 minutes. Stripe enforces rate limits at multiple levels, returning HTTP 429 with headers indicating which limit you hit. High-volume customers can request increases, but it requires advance notice and business justification. Even “Unlimited Edition” has limits.
These are different shapes of problems: multi-tenant API protection versus long-running batch jobs. But the underlying principle is identical: bounded resources require explicit limits. Every platform uses similar patterns: rate limits, concurrency caps, daily quotas, exponential backoff for retries, timeout constraints, tier-based pricing for higher allocations.
Operational limits aren’t a workaround, they’re industry standard engineering practice.
The Fix: Semaphores, Connection Pooling, and Queue Fan-Out
With these changes in place, we reached a stable middle ground.
We implemented semaphore-controlled S3 concurrency, enlarged connection pools, DB lifecycle management for imagery exports, multi-part ZIP chunking, format-level queue fan-out, worker concurrency caps tied to pool size, and hard operational limits at the API layer.
Large imagery exports now finish in under 15 minutes instead of hours or timeouts. The database pool no longer saturates during peak usage. Workers process multiple jobs concurrently without resource starvation. Users can export complex multi-format datasets within our documented limits without mysterious failures.
Very large exports still occupy a worker for extended periods. They’re within our limits and they complete reliably, but the orchestration is still monolithic: one worker generates all parts sequentially and assembles the master ZIP. The next step is chunk-level fan-out, treating each part as its own queue message so multiple workers can build one export in parallel.
We didn’t solve the problem completely, we just made it manageable.
The Real Lesson: Distributed Systems Design Beats Optimization
We initially approached the problem as an optimization issue. But in reality, it was a distributed systems problem.
Parallelizing S3 downloads was the right first step as it addressed the most obvious bottleneck. But systems at scale have many bottlenecks, often hidden until you fix the first one. Faster downloads revealed database pool limits. Optimized workers revealed workload shape problems. “Just make it faster” gave way to “design for the constraints.”
Some will ask, why not just throw more resources at it? Scale horizontally, add more workers, increase the database pool, provision bigger instances. That works yes, but to a point. But unlimited horizontal scaling brings its own costs like infrastructure expenses, operational complexity, and the reality that you’re often optimizing for the 95th percentile use case while most exports are small and fast. Limits let you right-size infrastructure for typical usage while still serving power users reasonably well.
Optimization is sequential. Each fix shifts the bottleneck and trades for the other.
And the moment workloads become large enough, architecture matters more than incremental speed improvements. So we didn’t end up with “unlimited exports”, but a clearer understanding of where infrastructure limits appear, how distributed workloads evolve under scale, and why responsible backend systems need operational boundaries.
This is the same principle we work from when we design and run cloud infrastructure for teams whose workloads have outgrown the architecture they started with: the constraints are part of the design, not a compromise on it.
Sometimes good engineering is actually understanding where limits should exist.