Handling Incidents
Engineering organisations routinely have numerous alerts configured, some of which, when raised, will trigger alarms for the on-call engineers. There also should be an incident response playbook that gets employed during incidents to ensure a structured response and analysis as well as stakeholder communication. Finally, there should be a post-mortem process and documentation template to document and understand what has happened afterwards. All of these tools serve to ensure business continuity and learning.
What if something happens outside the engineering remit?
This happened: over a weekend, a Google marketing campaign malfunctioned. It happening over the weekend, this was noticed late, after a huge cost spike had amassed.
That qualifies as an incident, to which a quick reaction - triggered by an alert going off - would have been ideal. A structured post-mortem analysis did happen, and Marketing even reused Engineering’s post-mortem template for it.
My point: all areas of the business should give risks and incidents the same kind of treatment. There should be automated alerts. There should be someone “on call” to be able to respond to an alert quickly. Incidents should be analysed post-mortem, regardless of their type, because there’s always something to avoid, something to learn about how to avoid, and something to understand about the failure class rather than the failure instance.
In a way, that’s what controlling is all about.
Tags: work