Levels of Insight
Once upon a time, there was an issue at work with having overlooked SSL certificates that were due for renewal. Once discovered, that was a quick fix, however we missed that the same certificate covered several subdomains, so more breakage was discovered and had to be fixed. In analysing what had happened, we discovered important insights and learnings at different levels.
The obvious one was more of a confirmation: the team understood the technology and issues we had been facing, so that those could be addressed quickly. This was good: we had small knowledge gaps at worst in the technology area.
There were two more levels of insight where we had to pay attention.
The first was the level of infrastructure and process. This covers things like dashboards, automated alerting and reminders, validation, documentation, and inter-team collaboration. These things hadn’t gotten the right attention - or there hadn’t been time for them. Both of these reasons weren’t acceptable, and we worked on fixing this.
The second level was that of plain old behaviour, or culture. It had, unfortunately, come to light that we had, on some occasions, not learned from failure: several of our subdomains had been affected by certificate issues across several subsequent days. Now, failures happen, so that’s “OK” - as long as we learn from them. If the same failure happens again, that’s a problem, because the learning didn’t happen. If the same failure happens yet again, that’s absolutely not OK by any means, because the insight that the learning didn’t happen wasn’t taken seriously.
As a bottom line takeaway, we had to learn that thinking in failure classes sounds plausible and even kind of obvious, but it takes the doing and awareness of the possibility of there being “more issues of a similar nature”.
Fixed.
Tags: work
Latency, Visualised
Here is a beautiful interactive visualisation of how long things take in and around computers, from L1 cache references to an IP packet roundtrip from the Netherlands to California and back. Each duration is a bar that’s proportional in length to all the others. That gives you a sense of relations. What’s even more fun is that you can also go back in time and observe how performance has evolved during the years.
Tags: the-nerdy-bit
Knowledge Sharing in Engineering Teams
The best approach to knowledge sharing in an engineering team is pair programming. Change my mind.
Such a bold stance warrants justification. So here goes. In pair programming, both involved engineers’ minds are simultaneously focused on the code base and problem at hand. The engineer more experienced with the code base imparts valuable knowledge about the code base on the less experienced engineer. This happens while both collaborate to build the next feature increment, fix the next bug, apply the next refactoring - you name it. It’s learning on the job, while it’s being done. The synchronicity of the activity is the key to its twofold benefit.
Other forms of knowledge sharing often used by engineering teams are - in increasing order of utility and value - written documentation, knowledge sharing sessions, live coding sessions, and mob programming.
Written documentation is good and important, but does not adapt to the coding situation: it needs to be consulted as an information source on the side. Also, it’s written by someone at some point, and later consulted, typically by someone else. Two people invest time, independently from one another.
Knowledge sharing sessions are great because they introduce the Q&A element. They take place independently from problem solving activities however, and preparing them takes time (if they are prepared at all).
Live coding - which could be an unprepared knowledge sharing session, so to speak - is good because it’s interactive and in the problem solving situation. It does, however, typically involve a comparatively large and mostly inactive audience and no task switches.
Finally, mob programming: this is pair programming, but with more people. Its main drawback is that it binds much more people than pair programming.
There are valid usages for each of these. I will stand by my initial opinion that if knowledge sharing is needed, pair programming has the best combination of involving the right number of people whilst at the same time guaranteeing productivity.
Tags: work
Speaking Drums
My wife and I once had been to a concert where one of the pieces performed was the concerto for percussion and orchestra “Speaking Drums” by Péter Eötvös. This is quite a remarkable piece. It’s seriously complex modern “classical” music, rather well composed, and hilariously entertaining - all at the same time.
True to the genre, it’s a piece for orchestra and solo instrument. Just that the solo instrument is a multitude of percussion instruments played by one soloist. These instruments - drums, cymbals, marimbaphone, tubular bells, and things that go “doink” (to name just a few) - are spread out over a vast amount of space.
Consequently, the soloist’s job is not only to play all of the instruments, but also to buzz around on stage between them. As if that wasn’t enough, the soloist also has to recite nonsense poetry that is the linguistic inspiration for the rhythms played on the instruments.
Allow yourself to be entertained: here’s a performance featuring the fantastic Martin Grubinger.
Tags: music
Menace
Watching a German quiz show a while back, I learned from this most unsuspected of sources that a man named Donald Michie had, in the early 1960s, devised an entire machine learning solution for playing Tic-Tac-Toe. This thing would actually apply reinforcement learning to improve over time, to the degree that games against a strong human player who knows the optimal strategy would always end in a draw.
The model is called MENACE. What’s particularly fun about it is that it is built using matchboxes filled with coloured beads. Each box represents one possible layout in the grid. The beads are randomly used to decide on the next move, and beads that lead to winning the game are multiplied in the respective box, thereby increasing the chance for being picked. Beads leading to losing the game are removed.
This illustrates nicely how reinforcement learning is about statistics. It is so simple that children can understand it.
Tags: the-nerdy-bit
Parse This
The German language is an example of Kurt Gödel’s incompleteness theorems. The following sentence does make sense, but does not parse. “Die Welt und mir ist schlecht.”
Tags: the-nerdy-bit
Code Documents Itself
“The code documents itself!” is nonsense unless everyone thinks exactly alike, and that would be a sad, sad world indeed. So, documentation it is!
Forgive the snotty intro. I’ve been sounding like a broken record when it comes to spreading knowledge in engineering (and other) teams - wherever I’ve worked for the past 10 years or so. Knowledge sharing can take on many forms, pair programming being one of them; and written documentation, another.
Going from the “self-documenting code” illusion to a usable actual documentation is a piece of work. More yet, it needs to be part of every piece of work: documenting what’s being built has to become a standard task in engineering. Just like (hopefully) writing tests. Only when this habit is ingrained, documentation will flow from everyday work like code. Not effortlessly, but habitually.
A description of a structured journey is found in this nice article.
Tags: work
Large-Scale Choral Music
How about some monumental “contemporary” unaccompanied choral music? Two 20th-century composers (so much for “contemporary”) I admire have written pieces that are impressive for their length and complexity or superficial lack thereof (as we will see).
The composers are Alfred Schnittke and Arvo Pärt. Both were born in the Soviet Union - Schnittke, in Volga; Pärt, in Estonia - and found their unique musical languages partially through their experiences living in the totalitarian system. Both the works I want to introduce are both based on Orthodox Christian texts treating the broad topic of repentance. Schnittke’s lyrics are in 16th century Russian; Pärt’s, in Church Slavonic.
This is where the commonalities end: musically, the two pieces could not be farther from one another.
Take Schnittke’s Psalms of Repentance. The piece is some forty to fifty minutes long and harmonically challenging. There are neither complex rhythm nor counterpoint: the singers basically “read out” the lyrics to the listener, albeit with the composer’s interpretation added. The music directly slams the full emotional bandwidth into the listener’s ears: ranging from self-doubt, consciousness of and sadness about guilt, and trust in forgiveness and eternal love are all there, mostly in the harmonics. Here is a recent (and quite excellent) rendition.
Now, take Pärt’s Kanon Pokajanen. Commissioned for the 750th anniversary of Cologne Cathedral, it had to be monumental, and with a duration of 80-90 minutes, it does not disappoint. (Remember, this is for unaccompanied choir.) The piece is written in Pärt’s personal style, which is not known for huge dramatic effects at all. That makes it immensely hard to perform and listen to. The performers, on the one hand, have to muster the strength to keep very long arcs of subtle suspense that often incorporate actual silence. Getting this wrong makes for a very monotonous and dull experience. It’s hard, very hard, to sing like that for such a long time. The audience, on the other hand, must be ready to endure a long time of meditative no-drama tension. It’s really quite unique. Kanon Pokajanen does have all the drama that comes with the lyrics, but it’s veiled and more spiritual than with the previous piece. Give it a try: I could not find a good complete rendition on the ’tubes, so here’s a collection of excerpts that has appropriate quality.
Tags: music