Architecture & End-to-End Pipeline:
- Goal: Ingest large video files, transcode into adaptive bitrates, and stream seamlessly across diverse global devices and network speeds.
1. Ingestion & Upload:
- Resumable chunked uploads directly to cloud storage (S3) via multi-part signed URLs to tolerate network drops.
2. Transcoding & Adaptive Bitrate Pipeline:
- Upload triggers an event in a distributed queue (Kafka/SQS).
- Distributed encoding workers split video into 5-second segments and transcode in parallel into multiple resolutions (1080p, 720p, 480p, 360p) using H.264/HEVC/AV1 codecs.
- Manifest Generator creates HLS (.m3u8) and DASH (.mpd) manifests describing available bitrates and segment file paths.
3. Storage Tiering:
- Raw source videos archive to cold storage (Glacier/S3 Standard-IA).
- Transcoded segments store in distributed object storage with edge caching.
4. Global Delivery & Playback:
- Geo-distributed CDNs cache video chunks at edge locations.
- Player client dynamically switches bitrates based on real-time bandwidth estimation to prevent buffering.
5. Metadata & Engagement:
- Relational DB (PostgreSQL) for user accounts and video metadata; Cassandra/ClickHouse for high-throughput view counts and telemetry events.