Architecture & Fault-Tolerant Execution:
- Goal: Schedule and execute millions of recurring and one-off tasks reliably with at-least-once delivery, retry mechanics, and worker auto-scaling.
1. Scheduling & Leader Election:
- Scheduler Leader: Use Raft/ZooKeeper for active-passive or partitioned active-active leader coordination to prevent dual scheduling.
- Priority Queue Datastore: Store scheduled tasks in a distributed database indexed by execution timestamp: (scheduled_time, job_id). Redis Sorted Sets (ZADD with timestamp score) or PostgreSQL with index on scheduled_at.
2. Task Dispatcher Loop:
- Dispatcher polls tasks where scheduled_at <= now(). Use atomic state updates (UPDATE jobs SET status='queued' WHERE id IN (...) AND status='pending').
- Dispatches runnable tasks onto distributed worker message queues (RabbitMQ/Kafka) segregated by priority/queue name.
3. Worker Execution & Heartbeats:
- Worker nodes pull jobs, execute code in isolated container/sandbox environments, and report periodic heartbeats back to Redis.
- If a worker crashes mid-job, the heartbeat monitor detects timeout and re-queues the job (At-Least-Once execution guarantee; jobs must be idempotent).
4. Execution History & DLQ:
- Job outcomes (success, return code, logs) persist to an analytical datastore. Repeatedly failing jobs route to a Dead Letter Queue.