Levels of Insight
Once upon a time, there was an issue at work with having overlooked SSL certificates that were due for renewal. Once discovered, that was a quick fix, however we missed that the same certificate covered several subdomains, so more breakage was discovered and had to be fixed. In analysing what had happened, we discovered important insights and learnings at different levels.
The obvious one was more of a confirmation: the team understood the technology and issues we had been facing, so that those could be addressed quickly. This was good: we had small knowledge gaps at worst in the technology area.
There were two more levels of insight where we had to pay attention.
The first was the level of infrastructure and process. This covers things like dashboards, automated alerting and reminders, validation, documentation, and inter-team collaboration. These things hadn’t gotten the right attention - or there hadn’t been time for them. Both of these reasons weren’t acceptable, and we worked on fixing this.
The second level was that of plain old behaviour, or culture. It had, unfortunately, come to light that we had, on some occasions, not learned from failure: several of our subdomains had been affected by certificate issues across several subsequent days. Now, failures happen, so that’s “OK” - as long as we learn from them. If the same failure happens again, that’s a problem, because the learning didn’t happen. If the same failure happens yet again, that’s absolutely not OK by any means, because the insight that the learning didn’t happen wasn’t taken seriously.
As a bottom line takeaway, we had to learn that thinking in failure classes sounds plausible and even kind of obvious, but it takes the doing and awareness of the possibility of there being “more issues of a similar nature”.
Fixed.
Tags: work