Situation: Our user analytics events were growing at 200% month-over-month, threatening to overwhelm our single PostgreSQL database within six weeks, and we had to select a distributed event datastore without knowing our exact query patterns six months out.
Task: I had to select and implement a scalable datastore architecture quickly without locking the company into an inflexible or overly complex ecosystem.
Action: I categorized our options based on two-way vs one-way door decisions. Migrating data ingestion was a two-way door if we introduced an abstraction layer. I designed an event publishing interface backed by an append-only Kafka topic, isolating consumers from producers. For storage, I evaluated ClickHouse against Cassandra against Elasticsearch. Because analytical aggregations and cost efficiency were our top requirements, I selected ClickHouse for its columnar compression and SQL compatibility. To mitigate unknowns, I built our query client with schema adapters so we could switch or add backends if indexing requirements shifted.
Result: ClickHouse handled our 50M daily events with 80% lower disk consumption than our relational database, and query latencies dropped from four seconds to under eighty milliseconds. The message-bus abstraction gave the data science team easy direct access to the stream without modifying application code.