Architecture & Feed Generation Pipeline:
- Goal: Serve personalized feeds of posts, media, and recommendations to hundreds of millions of users with sub-100ms response times.
1. Fanout Models (Push vs Pull):
- Fanout-on-Write (Push): When user posts, write post ID into all followers' timeline caches (Redis). Great for active users with small follower counts; fails for celebrities with millions of followers.
- Fanout-on-Read (Pull): Fetch recent posts from followed users and merge at query time. High compute cost on read.
- Hybrid Fanout: Push for regular users; for high-follower celebrity accounts, merge their posts at read time.
2. Candidate Generation & ML Ranking:
- Candidate Retrieval: Aggregate ~1,000 candidate posts from followed accounts, social graph interactions, and trending topics.
- Ranking Pipeline: Pass candidates through a multi-stage machine learning ranker (LightGBM / neural ranking model) scoring candidates on engagement probability (likes, shares, comments).
3. Caching & Delivery:
- Hydrated top-100 post feeds are cached in Redis.
- Client fetches feed via paginated API with cursor pagination (before_id), eliminating offset performance degradation.