Architecture & Resilient Delivery Pipeline:
- Goal: Deliver outbound HTTP webhooks to external customer endpoints reliably, handling slow endpoints, timeouts, and security verification.
1. Ingestion & Event Decoupling:
- Internal services publish event payloads to an append-only Kafka topic. Delivery engine runs asynchronously without blocking core application flows.
2. Cryptographic Security & Signatures:
- Generate a cryptographic HMAC-SHA256 signature for each payload using the customer's shared secret.
- Include signature and timestamp in headers (X-Signature, X-Timestamp) to prevent replay attacks and allow receivers to verify authenticity.
3. Outbound Dispatchers & Failure Retries:
- Worker pool pulls events and dispatches HTTP POST requests with a strict 5-second timeout.
- Exponential Backoff with Jitter: If client endpoint returns 5xx or times out, retry at 1m, 5m, 30m, 2h, and 24h intervals.
- Circuit Breaker: If a customer endpoint consistently fails for over 100 consecutive requests, pause delivery and notify the developer via dashboard to prevent resource waste.
4. Developer Tooling & Audit Logs:
- Store delivery logs (status code, execution duration, payload, response headers) in ClickHouse/PostgreSQL for developer debugging.