Architecture & Lifecycle:
- Goal: Allow users to upload high-resolution images, generate responsive thumbnails asynchronously, and serve them globally with low latency.
1. Direct-to-Storage Ingestion:
- Avoid bottlenecking API application servers with large file uploads.
- Client requests a pre-signed S3 upload URL from API server with file size, content-type, and MD5 checksum restrictions.
- Client uploads file directly to an S3 raw ingestion bucket over HTTPS.
2. Asynchronous Processing Pipeline:
- S3 triggers an event notification to an SQS queue upon upload completion.
- Processing worker pool (containerized image resizing service using libvips) pulls tasks from SQS.
- Workers strip EXIF metadata (privacy), validate image headers (security), and generate standardized WebP/AVIF variants (thumbnail: 150px, medium: 600px, large: 1600px).
- Transcoded variants are written to a public S3 processed bucket.
3. Global CDN Delivery:
- CloudFront CDN sits in front of the processed bucket with aggressive edge caching (Cache-Control: public, max-age=31536000, immutable).
- Use unique content-addressed hashes in URLs (/images/{hash}_600.webp) so updates invalidate cleanly.
4. Failure Handling:
- If resizing fails, retry via SQS with exponential backoff; poison pills move to DLQ for engineering review.