Architecture & Data Ingestion Flow:
- Goal: Collect, index, and query terabytes of logs, metrics, and traces per day with sub-second alerting and interactive search.
1. Edge Collection & Transport:
- Lightweight DaemonSet agents (e.g., Vector or FluentBit) run on each server node, collecting logs from containers and system metrics (CPU, RAM, disk).
- Agents batch, compress, and forward telemetry over persistent gRPC connections to ingestion gateways.
2. Stream Buffering & Decoupling:
- High-throughput Kafka cluster acts as a shock absorber during traffic spikes, partitioning logs by tenant_id.
3. Processing & Storage Engine:
- Stream Processing (Flink): Computes rolling aggregates (p50, p95, p99 latencies, error counts per minute) in real time.
- Metrics Datastore: Time-Series Database (TSDB like Prometheus, VictoriaMetrics, or ClickHouse) optimized for timestamp-ordered numeric series.
- Logs Datastore: Columnar storage with inverted indexes (ClickHouse or OpenSearch) for full-text search with hot/warm/cold lifecycle retention.
4. Real-Time Alerting Engine:
- Rule evaluation engine evaluates PromQL/SQL thresholds every 10 seconds against Flink state to fire PagerDuty/Slack alerts with zero query load on storage.