Musical City Techno

Fanny’s Music, East Nashville

Cicada, cricket, grasshopper, frog. However that particular indeterminate carpet of sound occurs, it is a permanent feature of the American South. I grew up in this soup, so hearing this familiar symphony again after nearly two decades of being away felt surreal and magical.

It is the sound that framed my stay in Nashville, Music City, for three mini-conferences all held at once at the Nossi College of Art: Music City Agile, Music City Code, and Music City Data. Although tickets were sold separately, it was all mixed together, so all attendees had access to everything. I recall being initially impressed by their high-quality social media campaign, and getting to the venue the conference Crew did not disappoint.

art on the walls at Nossi

Music City Tech was well organized, well-staffed, expertly choreographed (at least from my vantage point, whatever demons they found were hidden well), and thoughtfully planned. The personal attention given to speakers made it special. The organizers were friendly, helpful, thoughtful, and courteous. Southern Hospitality was at its pinnacle here.

I would say that this all made it feel like a well-oiled, small event… except that 900 attendees and nearly 200 talks and workshops over three days isn’t exactly small. Not only that, it was the most diverse group I’ve seen yet at a tech conference, which really made me smile, as did the attention to including Music-themed talks!

I have written a few things related to my talk, The Music in Resilience (aka The Practice of Practice), so I won’t really go into it here (blog version, soon enough). Instead, what I want to embolden is this thread of confluence between musical and sociotechnical systems that I see showing up in other peoples’ thinking.

instruments on the show floor!

Before jumping into the other talks, it helps to underscore a central theme from mine that was strongly emphasized in all: being on a technology team is like being in a band. An important tool for a collaborative team to maneuver the challenges of complex systems, software or musical, is to build group intuition.

The Sounds of Software

Brett Berliner (@brettberliner)
Principal Software Engineer, Insight

Right away Brett made the connection that many people make and that I’ve heard all my life: musicians make excellent computer people and computer people love music (he has personally attended 500+ live shows). He started off by drawing a clear parallel between great team communication and people getting together as a band and practicing.

geeks geek out to the afterparty band

I felt a bit like I was lounging in a levitating gondolier, floating low above the surface, with our guide pointing out topography below of how music and coding intersected. Using himself as Exhibit A, we were shown early listening devices (boom box, portable CDs, mp3 player pioneers), logos of usual suspects in online music services (props for Bandcamp mentioned), and even headphone recommendations. He explained the methodology for choosing songs for his shared playlists, crafted for various moods and temperament while working.

Lastly was an introduction to live music coding, including something I’ve used before called ChucK, but mostly featuring a (successful!) live demo of Sonic Pi. This came before my talk on Friday, and although I normally never go to talks before I give one, I wanted to make sure I saw this because I had a feeling I could relate.

Does Your Team Groove?
What You Can Learn from a Bass Player!

David Gentry
Scrum Master, Agile Coach

A good deal of the sponsorship appeared to revolve around Agile practices and general team organization and efficiency. Part of the Agile conference track, this talk was my favorite. He even brought a copy of The Music Lesson by Victor Wooten – source material and inspiration for his subject – to give away. You know I grabbed it.

Good Groove is everything but the notes, Good Agile is everything but the process.

Right away the familiar motif returned, and “playing with other people” underlined everything, again making the astute observation that collaboration in teams is the real action of playing with people. I think he and I share the same view that this is not strictly allegorical. As such, David more strongly resonated with the subject matter of my talk, really digging into specifics, not being satisfied merely by comparison and simile.

Like I do with idiomatic improvisation and complex systems, so does David link bass playing and the Scrum Master, believing in that deep connection, feeling it himself. These relationships aren’t only useful because of being relatable to the prevalence of Music in the lives of humans, they also tickle the same cognitive space as collaboration in software engineering.

David and his bass came to play!

He covers six territories of overlap: 1) Servant Leadership as a Scrum Master is akin to the place of the bass in a band, a foundational but often not obvious carrier of the ground; 2) Congruence as the rhythm and harmony holding things together, not noticed until incorrect or missing; 3) Cadence as tempo, synchronization, pulse, and timeboxed structures; 4) Conflict as dissonance that eventually balances to illuminate resolution and discovery; 5) Space as one of the greatest things to give or receive, with characteristics of trust, self-organization, psychological safety, and freedom to innovate; 6) Groove as everything about music but the notes, or in Agile as anything except the process.

His electric bass was at the ready, playing along with simple accompaniment through a single amplifier, giving examples of each of these six in its musical context, while also showing common language used in the sociotechnical one. Triplets against dotted rhythms for Congruence and walking jazz bass for Conflict were especially good, and hey we were even ‘Journey-rolled’ once he made it all the way to Groove. Don’t stop believin’ in Agile, I guess? 😉

What is Kafka?
TicketMaster’s Data Streaming Journey

Pooja Rallabhandi (@pzralla)
Software Engineer III, TicketMaster

defunct PC store, East Nashville

What I liked about this was its connection to Music through a mature, massive, and borderless sociotechnical system that has grown in complexity over nearly half a century. After steering any hecklers away from TicketMaster’s pricing model, Pooja flew at a high level about Kafka‘s pillars, giving a brief overview of the value in persistent Commit Logs and the relationship of producers and consumers with brokers.

The T.M. Core database surely isn’t the same technology in use 43 years ago, right? Although … [queue time travel sequence soundtrack] … I fondly remember operating the TicketMaster VAX terminal at Tower Records 15 years ago, itself reminding me of the VAX at Virginia Tech we used for email 10 years before that … ok, so maybe it’s not that unbelievable. She explained that attempting to solve their problems with normal queue products like ActiveMQ simply didn’t work, ultimately choosing Kafka to front the read layer of their access model for ticketing.

Ticket order transactions go directly to T.M. Core, maybe through some other logistics prior, but nothing touches Kafka yet. That happens when the data gets aggregated and prepared for display – for example on your phone for scanning right after you bought it or for resellers, where immediate updates would be needed – and a near-real-time stream of data is required. That’s where Kafka’s speed and durability comes to the rescue, which they bolster even further with their own strong consistency guarantees by performing snapshot comparisons. It is an elegant example of the tried-and-true separation of reads and writes so familiar to efficient database usage.

afterparty jugglers and geeks converge

Philosophy vs Practice

PJ Hagerty (@aspleenic)

This one had nothing overtly to do with Music but neatly supports our central theme. The reason I wanted to check this out was not only that PJ was basically the one other speaker there I even remotely knew, not because he recommends the CI/CD tool I have most recently experienced (GoCD), not due to his self-proclaimed musicianship (I’m sure he’s a fine singing saw player), not even because he called out Chaos Engineering with a Netflix tale, but because the general topic touches an exposed nerve of mine.

The key is that teams work together, the most important part of DevOps is communication.

Nevertheless, his presentation attempted to move us away from the catchphrases so often associated with DevOps, instead dialing-in on how modern tooling supports a cross-disciplined cooperative environment. I am particularly interested in this perspective because Chaos Engineering plugs directly into concepts of group learning in a sociotechnical complex system, especially Game Days (outlined in my talk as ‘The Practice of Practice’).

He also gave a shout-out to another sociotechnical subject near and dear to my heart: SRE. Only he called it Site Reliability Expert. I think my tweet on this sums it up nicely:

This ‘mistake’ actually fit into the talk perfectly, because eschewing the buzzwords really helped make it all about practical combinations of tool and theory, and PJ readily handed out toolset recommendations by continuously relating their practicality to the overreaching philosophy of DevOps.

What is that philosophy, you ask? It’s pretty simple, and it was refreshing to hear someone else say so. Above all, the key is that teams work together. The most important part of DevOps is communication. In fact, it makes sense to think about DevOps as something that needs multiple perspectives to exist at all. Maybe DevOps is everything about software engineering except the code.

Music City

The Gambling Stick BBQ

It felt good spreading the word of Chaos Engineering and Resilience to more ears, and it was spectacular to be a part of the conference. Having been at a few conferences lately that happen on the (US) coasts, it is striking that trends and conversations around the resilience of complex systems and new ways of thinking about safety aren’t being as heavily preached in more landlocked locales. At both here and Texas Linux Fest earlier this year, the number of hands going up for the question “who has heard of Chaos Engineering” was one or less, even in casual conversations. I am finding that this subject seems to get a lot less attention in so-called ‘flyover’ areas than it would at a coastal conference. This needs to change, and my team at Verica is in the game to help make that happen.

I decided to stay over in Nashville a bit afterward to experience some of the city (and sorely missed southern BBQ), fortunate to have a friend and coworker there with whom I also wanted to spend time. So imagine my joy when she surprised me on Saturday with a message that “Duet for Theremin and Lap Steel” from Atlanta (@duetonline) would be performing that night at a sake distillery downtown.

Of course, without a moment’s hesitation, I said yes.

DFTALS at Proper Sake, Nashville

Weirdness of Experiment

My job in ops has always been to keep things running. I never considered myself “working in software”, but have recently begun embracing the fact that I do. What I accomplish as an operations and infrastructure engineer is part of the system, it isn’t dislocated from its composition.

Relatedly, I have been considering the nature of the experiment in Chaos Engineering. How continuous verification is becoming a crucial part of the complex systems we build because there really is no end. Developing a software system isn’t just about writing it, it’s also every bit as much about running it. Unless there is some kind of evil catastrophic end-game planned from a volcano island hideout, most of us want to keep them running.

I’m big on experimental music. You probably know what I mean when I say that, but you might not because genres, in general, are horrible overgeneralizations. Similarly, after the composer John Cage had written his “silent piece” in 1952 (see also Living 4’33”), he seemed to have a struggle with the concept of calling the music composed by him and others he admired experimental.

In science, we often think of an experiment as a method to (dis)prove a hypothesis. We perform experiments to answer a question or assertion, often during the process of reaching an end goal. To Cage, this implied that calling something an “experiment” meant it was not complete, not finished. That there is a final state determined by products of the experimentation, and he thought that his (and others’) music was complete when performed. There was no “final state” that was decided as a result of an experiment either succeeding or failing, if it was itself called experimental.

Cage revised this view, however. He began embracing the term and actually ended up preferring it. The reason for this is the way he evolved to think about the context of sound. At the beginning of the decade, he experienced an anechoic chamber (an “echoless” room) and the non-presence of total silence, because he could hear both high and low sounds — explained to him by the engineer as his nervous system in operation and blood circulating, respectively. Whether or not that is physiologically probable, he had the now famous revelation that it is impossible to remove sound completely. 4’33” and an entire philosophy about the nature of sound and silence in music was not far behind.

To him then, the moniker experimental came to mean that which is undiscovered, because even if a piece of music requires certain sounds, environmental sounds are impossible to predict. This experimental music isn’t about the search for failure or success, but an experience of discovery, where questions become more interesting than answers. When applied to composition, each performance of a musical work is always new and different due to its context and sonic environment. Indeed, it is impossible to know ahead of time any structure of the interpenetrating sounds both intentional and not, themselves independent and unique (whether or not they are consonant). It is in fact in a total state of chaos, each and every time.

When complex systems run, they do so at the hand of indeterminacy and randomness. There certainly is a “steady state”, but it is continuously in need of verification. Just like Cage observing that no performance of a musical work is a repeat, the nature (structure and form) of distributed systems we operate cannot in truth be predicted with any kind of regularity.

So while it is useful to be very specific in defining and running our Chaos experiments, the nature of what we’re doing is more about asking questions and making discoveries, not testing for answers we already think we know or think we can guess. The “breaking things in production” mantra implies we are interested in failure when what we’re really interested in is what was determined and what questions arose, good or bad.


Here are a couple of PDFs taken from Cage’s writing that highlight his viewpoint on the subject of “experimental music” as a title of what he did.

  • Experimental Music: Doctrine (1955) ::: This article, there titled Experimental Music, first appeared in The Score and I. M. A. Magazine, London, issue of June 1955. The inclusion of a dialogue between an uncompromising teacher and an unenlightened student, and the addition of the word ”doctrine” to the original title, are references to the Huang-Po Doctrine of Universal Mind.
  • Experimental Music (1957) ::: The following statement was given as an address to the convention of the Music Teachers National Association in Chicago in the winter of 1957. It was printed in the brochure accompanying George Avakian’s recording of my twenty-five-year retrospective concert at Town Hall, New York, in 1958.

The photo above is the album cover from a release by Craque called Meat Hacker.

Musical Intuition meet Technology and Chaos

Today I read through the InfoQ eMag on Chaos Engineering, and was struck by John Allspaw’s (@allspaw) contribution because it reminded me of something I jotted down on a sticky at my desk a few days ago:

Intuition is valid because it is learned like jazz changes.

I’m pretty stubborn and refuse to accept that music is merely a hobby of mine. When people ask me if electronic music or singing is my “hobby”, I am wincing inside. So a question often on my mind is: how does the intuition I have when performing and composing music connect with the work I do as a technologist?

Some musicological background might help. One concept in learning how to improvise (jazz or otherwise) is that you have developed an intuition built around internalizing the materials and form of the piece (or genre) – like scales, chord changes, or rhythm structures. This is different from the more lizard-brainy concept of instinct. Think about a blues progression, the foundation of music you hear every day, everywhere. You know intuitively the chord progression and timing is “right”, even so much that anomalies and departures come across as emotionally significant. The rest is pop history.

But you, homo sapiens, do not have this chord sequence pre-programed in your DNA, it isn’t something that is instinctual. By the same token, great technology leaders develop good intuition (expertise over hundreds of interviews) when hiring engineers but never rely on instinct (oh I just have a good “gut feeling”). The best DBAs have an intuitive understanding of their platform (you want to do X, but did you think of Y+Z?), but there’s nothing instinctual about it.

It is not a stretch, then, to recognize that intuition in improvised music can be directly compared to how Allspaw writes about the “mental map” that engineers develop. They each have their own subjective view on relevant (but overlapping) parts of the system and are challenged when relating each substrate to theirs. For instance, a phenomenon known as “fundamental common-ground breakdown” (Woods & Klein: Common Ground and Coordination in Joint Activity) happens when what I describe as intuitions (accumulated individual learnings about the system) are assumed knowledge among participants, good or bad. Part of the game is learning how to harmonize these separate threads of experience, avoiding costly coordination surprise and re-synchronization… and trust me, I have been in plenty of rehearsals and narrowly saved performances that fit this description!

The important point here is that a system becomes more complex as it grows dimensions, shrinking the capacity of any one person to comprehend the whole thing. Therefore we rely on shared and discovered knowledge to fully grok these fascinating systems. Take any ensemble of musicians: as it grows in membership, individuals gradually lose the ability to contain its myriad relationships in their mental map, so coordination and integration become a matter of listening and rehearsal experience (both modes of communication). Oh and it characterizes the music, too. Building intuition about how to play a part in an opera is much different than in a free improv vocal trio. Orchestrating ten thousand linux containers in a cloud provider doesn’t compare to managing two rows of server racks at the datacenter downtown.

Technologists grapple with the task of building and sharing intuitions about a system because understanding an entire system contributes to what we know about making it more resilient. Communication is key in either musical or engineering teams, collaboration on understanding the whole is no exception. Our mental maps should be adaptable to constant updates, and practices like Chaos Engineering that make discoveries in complex system behavior are supported by this kind of cross-pollination and proliferation of our combined understanding.

A quote from Allspaw’s article highlights it well:

Maybe the process of designing a chaos experiment is just as valuable as the actual performance of the experiment.

– John Allspaw, Recalibrating Mental Models through Design of Chaos Experiments

The use of the term “performance” is apt. We’re familiar with this concept: practice makes perfect. Taken further, the experience of practice is necessary such that the result is merely an extension of practice. It takes meticulous work to understand a piece of music to the level of having an intuition about how it operates, and the same goes for building experimentation that challenges what you think you know about complex software. The results of the “performance” can be enhanced by a focus on understanding the system’s design and steady state (i.e. nominal condition), what we would call the language of the musical work. It is as if the performance of the event naturally evolves from learnings gained preparing for it.

Imagine you are a jazz musician, you have gone through years of studying scales and changes and charts and recordings of a particular artist, and have built a capability for understanding how the language of their music works. One evening at a local club, your dreams are fulfilled, you’re in the audience and invited up for a set with them. You intuitively know how this person plays their music, as it has been a guide for your own. But when you’re jamming together, they do something indeterminately that informs your intuition in a way you would have never discovered yourself. Not only has the process of designing your inevitable collaboration been valuable to understand what you thought you needed to know to play like your biggest influence, but it also served as the basis for learning something new and unexpected.

Whether it is free improvisation or interpreting a through-composed piece of music (and everything in between), there is a certain amount of experience and training informing the performance. Eventually, when we’ve practiced enough, the music itself steps out of the way and intuition takes over. I think this is where my musical performance connection with technology starts: once you understand the fundamentals of the system, let the presentation of the system get out of the way, and you’re in a better place to evolve your mental map and gain further intuition through disciplines like Chaos Engineering.