Welcome to the Verified Machine

Imagine for a moment that you’re a software team, pushing new versions into production. You’re working on multiple pieces independently, so as you make changes automation pulls it all together in orderly Continuous Integration. At the right moment, your team’s work is pushed and Continuous Delivery does the job. Five deploys a day? No problem. Y’all have written, collaborated, tested, and published your glorious stuff. Job well done!!!

Just kidding, you’re not done. Does anyone write software that will suddenly just stop? Ok, maybe Y2K mitigation. But most of us in this industry want to see our stuff keep running into the unforeseeable future. That’s where Continuous Verification enters the picture. In operations, we’ve been given the ball to ensure the reliability of the software you write, but we want to be as invested in running it into the future as you are in writing it.

So, long story short (and trust me it’s long), that’s what I’m doing now at a company called Verica. We are a Continuous Verification company. It’s an area closely aligned with Human Factors Engineering, Resilience Engineering, Safety, and Chaos Engineering. In fact, Chaos Engineering is central to what we do. We are here to help the enterprise get a leg up on Continuous Verification for the complex distributed systems they run. Tested in production, like the glass says.

At Verica, I’m still doing ops. I am SRE, I am Sr. Infrastructure Engineer, I am IT, I am NetEng, I am Purchasing, I am UAT. When we get some, I’ll be DataOps. Nevertheless, I’m beginning to traverse a lot more of the software stack than I ever have before (Golang and service meshes, reluctantly Python, you can rip my Zsh from my cold dead hands). I am excited to teach and speak and collaborate on all things Chaos Engineering, especially how it intersects with my musical proclivities. I will be speaking at Texas Linux Fest at the end of May about this very topic!

Most of all I’m excited to work with a stellar team. These are people I’ve changed up my life and taken great risks to work beside. Five years ago I discovered there was a close similarity between indeterminacy in distributed systems and the things I have studied in music. Verica is the next amalgamation of my love for musical improvisation and operating complex systems. It’s about time I wrote an opera, isn’t it?

MusiCirTechUs

Plenty of people who work in technology are musicians. Plenty of humans are musicians, and all of us are listening, even those who cannot hear. In my travels through datacenters and software, I have come across musicians like myself who have not only done double-duty with their computer science and creative lives but also relate and interweave them. If they’re like me, they cannot break them apart. There are hundreds, if not thousands, of us. We all have different stories, came into the worlds of music and of technology from different angles and different backgrounds.

I’ll hazard a guess that when you think about the words music and technology together, it’s likely you’re thinking about music technology. This isn’t surprising, a great deal of music technology surrounds the production, performance, and recording of music. Musical Instrument Digital Interface (MIDI), studio recording, guitar pedals, analog control voltage synthesizers, DJ decks, Ableton Live, Max/MSP, Joe Armstrong running Erlang on Sonic Pi… the list goes on and gets fractally esoteric. All great examples of technology facilitating the art of music, but I’m interested in what’s happening the other way around.

The original impetus for this post was a comment someone made that raised an eyebrow… it made me briefly question the ideas I have about the field of resilience in distributed systems somehow relating to music. But humans share such an indisputably organic relationship with music and musical training that I knew I wasn’t just some crazy person, the connection was there and it was real. How else do I have the job I have now? This pushed me to want to find others that share my views on the close relationship music has with software and operational reliability.

Take Chloe Condon (Dev Advocate at Microsoft), a self-described recovering musical theatre actress. Now, my musical theatre training and tastes are preeetty specific, but I can relate. She pulls her theatre training into her work unforgivingly, and that is awesome because I fear that. Inspiringly passionate about the connections between her performing arts expertise and working in tech.

Recently chatting with her at SCaLE, Chloe clued me in about someone close to my wheelhouse: “Opera Singer turned Software Engineer” Catherine Meyers. Her Mozart could’ve been an Engineer talk correlates music theory and counterpoint to building software. She emphasizes the similarities of pattern recognition between the two disciplines, especially as it relates to learning. Preparing a new piece or constructing a score interpretation is 100% relatable to the software engineer learning a new language or building in a new framework. I especially love how she emphasizes the importance of having different types of minds together in one group, in other words, don’t be afraid to hire a music major.

These folks that are trained in synchronizing common ground through the experience of playing in musical ensembles carry a distinct advantage. Diversity and communication are the trademarks of both successful musical groups and software teams. Establishing common ground through shared learning is crucial to the modern team building and operating distributed software. It is obvious to me that ensemble music not only relates to complex systems but also to highly coordinated teamwork, to wit:

I’ve conducted marching bands, wind ensembles, and jazz bands... They are complex, highly interdependent systems.

Tanner Lund (SRE at Azure)

This couldn’t be more true! I have been in an experimental vocal trio, a free improv quintet, a blues band, a jazz group, an early music ensemble, a 330 piece marching band, uncountable operas with full orchestra and cast, chorales of few to many, piano accompanists by the dozen. There is nothing in any of these situations that doesn’t require common ground and shared experience.

In her blog post, The Origins of Opera and the Future of Programming, Jessica Kerr (Lead Engineer at Atomist) introduces the concept of symmathesy, a state of collaboration that is integrated with learning, all as part of the system (even the tools are team members). She does so by taking us back to the foundations of modern opera in Italy, adeptly adopting the term ‘camerata‘ from the Florentine Camerata example she uses, applying it next to social circles of painters and artists in later centuries, and then to teams creating software. She considers how our mental models evolve and how common ground is reached through the shared experience of a varicolored team from all sorts of different backgrounds, ultimately making the team more powerful. “We are developing whole new ways of being human together” she states, and to me, that’s nowhere more clear than with multi-disciplinary collaboration.

The CS / Math / Music crossover is clearly one I have encountered a lot, from early on. While I was studying voice and music technology in undergrad, I vividly remember walking with a good friend in downtown Blacksburg, and her saying to me “I could never major in music, there’s too much math.” I used to laugh at this because there’s really only as much math as that you wish to encounter. Want to dive into tempered tuning and Kepler’s mathematical ratios for perfect fifths? Go for it. But you know, these days I look at that statement through different lenses. Yes, there is just enough math! And if you study music, you may run the risk of studying computer sciences. * Please be aware that some practitioners of both have reported sudden bouts of inspiration and lightheadedness due to euphoric discoveries that cross disciplines.

On the subject of cross-discipline study…

I find a lot of what of I learned in music performance transfers directly to public speaking.

Amy Tobey (SRE at GitHub)

As a student of opera, I couldn’t agree more. Long ago I had a theatre teacher tell me “you will never be happy not onstage.” I didn’t know what she meant until I myself experienced exactly the same kinds of challenges and rewards with giving talks and public speaking as I did singing Mozart or Fauré in front of large audiences. I discovered that my experiences onstage presenting music are exactly the same headspace as when I’m talking in front of a room of computer people about distributed systems.

When I gave my talk Measuring Distributed Databases Across the Globe at Southern California Linux Expo 13x (2015), a very enthusiastic artiste-cum-techie approached me afterward. They had decided to move into software engineering after art school and said my talk, which combines theories and ideas in music with those of observing and operating huge distributed databases, was inspiring. They had been trying to reconcile their internal ‘artistic identity and self’ with their passion for coding, and I had opened their eyes to a new way of considering it.

I will never forget this. It was exactly like having someone come up after one of my experimental music performances (which are also heavy with technology) being blown away and asking how things worked. Even if one mind out of a thousand is influenced by what I’m doing, whether it’s music or technology, I’m building the camerata through sharing.

Eureka! The audience contributes to symmathesy, too. There are tons of examples; the feedback of a crowd during a DJ set, participants of David Tudor’s Rainforest, someone opening a candy wrapper at the symphony. There’s no getting away from the audience, however small. John Cage preached throughout his career as an avant-garde composer: listeners are participants. Every performance is something new because there are different people in different places every single time. The title of this piece is a play on Cage’s Musicircus, an amalgamation of disciplines and performers occupying the same large space together with the audience.

So consider the sheer experience of music and how it feels oddly in-step with how we think about technology, and take it one step further. Many of us trained in the practice of music consider it so much a part of our bones that it seeps into our work, we don’t have a choice, the pattern recognition pathways seem to match up. They fit into our psyches in a comfortable way, share the same affinity for abstractions, incorporate complex and overlapping systems that stretch the human capacity for comprehension.

Do I have an aptitude for music because I’m successful in understanding the practical application of complex things? Or do I have an aptitude in technical operations because I have been trained in music and theatre arts? Doesn’t it make sense that the diversity of music is a human quality, not a disciplinary one? Why should our work as computer scientists, software engineers, database operators, and network gurus be any different?

Temple Grandin talks about the world needing all kinds of minds. In her view, different people fall amongst three “categories” of thought: Picture (common on the autistic spectrum), Pattern (code, math, music), and Verbal (writers). It resonates with me a great deal because I always feel like I’m thinking in ways others are not. More specifically, though, it underlines how different styles of thinking can bring entirely new perspectives on a particular question, and how crucial diversity is to human endeavor, if not evolution itself.

This post started out as me being passionate about a subject and wanting to search for others who were. I had some bullet points of my own to insert, but as I searched and connected with others in the field, I learned new things and this blog transformed from a list of Twitter handles to real human relationships. I have grown to understand what Kerr means when she says that ideas are for sharing and collaboration, not hoarding and heroism. I for one am claiming my spot in the camerata that pushes us forward through a New Renaissance.

Weirdness of Experiment

My job in ops has always been to keep things running. I never considered myself “working in software”, but have recently begun embracing the fact that I do. What I accomplish as an operations and infrastructure engineer is part of the system, it isn’t dislocated from its composition.

Relatedly, I have been considering the nature of the experiment in Chaos Engineering. How continuous verification is becoming a crucial part of the complex systems we build because there really is no end. Developing a software system isn’t just about writing it, it’s also every bit as much about running it. Unless there is some kind of evil catastrophic end-game planned from a volcano island hideout, most of us want to keep them running.

I’m big on experimental music. You probably know what I mean when I say that, but you might not because genres, in general, are horrible overgeneralizations. Similarly, after the composer John Cage had written his “silent piece” in 1952 (see also Living 4’33”), he seemed to have a struggle with the concept of calling the music composed by him and others he admired experimental.

In science, we often think of an experiment as a method to (dis)prove a hypothesis. We perform experiments to answer a question or assertion, often during the process of reaching an end goal. To Cage, this implied that calling something an “experiment” meant it was not complete, not finished. That there is a final state determined by products of the experimentation, and he thought that his (and others’) music was complete when performed. There was no “final state” that was decided as a result of an experiment either succeeding or failing, if it was itself called experimental.

Cage revised this view, however. He began embracing the term and actually ended up preferring it. The reason for this is the way he evolved to think about the context of sound. At the beginning of the decade, he experienced an anechoic chamber (an “echoless” room) and the non-presence of total silence, because he could hear both high and low sounds — explained to him by the engineer as his nervous system in operation and blood circulating, respectively. Whether or not that is physiologically probable, he had the now famous revelation that it is impossible to remove sound completely. 4’33” and an entire philosophy about the nature of sound and silence in music was not far behind.

To him then, the moniker experimental came to mean that which is undiscovered, because even if a piece of music requires certain sounds, environmental sounds are impossible to predict. This experimental music isn’t about the search for failure or success, but an experience of discovery, where questions become more interesting than answers. When applied to composition, each performance of a musical work is always new and different due to its context and sonic environment. Indeed, it is impossible to know ahead of time any structure of the interpenetrating sounds both intentional and not, themselves independent and unique (whether or not they are consonant). It is in fact in a total state of chaos, each and every time.

When complex systems run, they do so at the hand of indeterminacy and randomness. There certainly is a “steady state”, but it is continuously in need of verification. Just like Cage observing that no performance of a musical work is a repeat, the nature (structure and form) of distributed systems we operate cannot in truth be predicted with any kind of regularity.

So while it is useful to be very specific in defining and running our Chaos experiments, the nature of what we’re doing is more about asking questions and making discoveries, not testing for answers we already think we know or think we can guess. The “breaking things in production” mantra implies we are interested in failure when what we’re really interested in is what was determined and what questions arose, good or bad.

Appendix

Here are a couple of PDFs taken from Cage’s writing that highlight his viewpoint on the subject of “experimental music” as a title of what he did.

  • Experimental Music: Doctrine (1955) ::: This article, there titled Experimental Music, first appeared in The Score and I. M. A. Magazine, London, issue of June 1955. The inclusion of a dialogue between an uncompromising teacher and an unenlightened student, and the addition of the word ”doctrine” to the original title, are references to the Huang-Po Doctrine of Universal Mind.
  • Experimental Music (1957) ::: The following statement was given as an address to the convention of the Music Teachers National Association in Chicago in the winter of 1957. It was printed in the brochure accompanying George Avakian’s recording of my twenty-five-year retrospective concert at Town Hall, New York, in 1958.

The photo above is the album cover from a release by Craque called Meat Hacker.