Situation: Shortly after a regular mid-day release, our API gateway error rates surged to over 15%, dropping incoming user authentication sessions across our web and mobile applications.
Task: As the primary engineer on on-call rotation, my immediate duty was to restore service availability, coordinate team communications, and isolate the root failure without panic.
Action: I declared a major incident in our operations channel, assigned a teammate to handle status page communications, and began triaging the telemetry dashboards. Metrics revealed connection pool exhaustion in the user profile datastore. Recognizing that the recent release introduced an unindexed query on session renewal, I initiated an immediate rollback of the deployment rather than attempting an untested hotfix. Once the rollback finished, database connections stabilized and error rates dropped to normal levels within five minutes.
Result: Total disruption was contained to eighteen minutes. Later that afternoon, I led a blameless post-mortem where we identified missing automated index validation in CI and added synthetic connection pool health alerts to prevent similar regressions.