Welcome to the Verified Machine

Imagine for a moment that you’re a software team, pushing new versions into production. You’re working on multiple pieces independently, so as you make changes automation pulls it all together in orderly Continuous Integration. At the right moment, your team’s work is pushed and Continuous Delivery does the job. Five deploys a day? No problem. Y’all have written, collaborated, tested, and published your glorious stuff. Job well done!!!

Just kidding, you’re not done. Does anyone write software that will suddenly just stop? Ok, maybe Y2K mitigation. But most of us in this industry want to see our stuff keep running into the unforeseeable future. That’s where Continuous Verification enters the picture. In operations, we’ve been given the ball to ensure the reliability of the software you write, but we want to be as invested in running it into the future as you are in writing it.

So, long story short (and trust me it’s long), that’s what I’m doing now at a company called Verica. We are a Continuous Verification company. It’s an area closely aligned with Human Factors Engineering, Resilience Engineering, Safety, and Chaos Engineering. In fact, Chaos Engineering is central to what we do. We are here to help the enterprise get a leg up on Continuous Verification for the complex distributed systems they run. Tested in production, like the glass says.

At Verica, I’m still doing ops. I am SRE, I am Sr. Infrastructure Engineer, I am IT, I am NetEng, I am Purchasing, I am UAT. When we get some, I’ll be DataOps. Nevertheless, I’m beginning to traverse a lot more of the software stack than I ever have before (Golang and service meshes, reluctantly Python, you can rip my Zsh from my cold dead hands). I am excited to teach and speak and collaborate on all things Chaos Engineering, especially how it intersects with my musical proclivities. I will be speaking at Texas Linux Fest at the end of May about this very topic!

Most of all I’m excited to work with a stellar team. These are people I’ve changed up my life and taken great risks to work beside. Five years ago I discovered there was a close similarity between indeterminacy in distributed systems and the things I have studied in music. Verica is the next amalgamation of my love for musical improvisation and operating complex systems. It’s about time I wrote an opera, isn’t it?

Pickles for Adam

(on the occasion of our meeting and bridge crossing)

Vinegar still rhythm
Sounds in parallel
Sticks shove and
Pucker.

Twiddling stone across
Steps to below regions
Glass smoothly to
Airwaves.

Pentatonic organum
Slides the rails aside
Bringing touches on
Reedlings.

Air patterns float around
Silent eyes and finds
Lowly tones with
Grains.

Fermented time brings
Succulent and sprung
Rocks melting or
Clicking.

Repeat and distracted
Silver round reflective
Cover glimpse for
Shakers.

Wooden spike and leather
Salient table top runner
Up down swideways for
Dripping.

Finally ice in clacks
Slides and carousels
Rotations wax and be
Poetic.

(March, New York City, 2019)

MusiCirTechUs

Plenty of people who work in technology are musicians. Plenty of humans are musicians, and all of us are listening, even those who cannot hear. In my travels through datacenters and software, I have come across musicians like myself who have not only done double-duty with their computer science and creative lives but also relate and interweave them. If they’re like me, they cannot break them apart. There are hundreds, if not thousands, of us. We all have different stories, came into the worlds of music and of technology from different angles and different backgrounds.

I’ll hazard a guess that when you think about the words music and technology together, it’s likely you’re thinking about music technology. This isn’t surprising, a great deal of music technology surrounds the production, performance, and recording of music. Musical Instrument Digital Interface (MIDI), studio recording, guitar pedals, analog control voltage synthesizers, DJ decks, Ableton Live, Max/MSP, Joe Armstrong running Erlang on Sonic Pi… the list goes on and gets fractally esoteric. All great examples of technology facilitating the art of music, but I’m interested in what’s happening the other way around.

The original impetus for this post was a comment someone made that raised an eyebrow… it made me briefly question the ideas I have about the field of resilience in distributed systems somehow relating to music. But humans share such an indisputably organic relationship with music and musical training that I knew I wasn’t just some crazy person, the connection was there and it was real. How else do I have the job I have now? This pushed me to want to find others that share my views on the close relationship music has with software and operational reliability.

Take Chloe Condon (Dev Advocate at Microsoft), a self-described recovering musical theatre actress. Now, my musical theatre training and tastes are preeetty specific, but I can relate. She pulls her theatre training into her work unforgivingly, and that is awesome because I fear that. Inspiringly passionate about the connections between her performing arts expertise and working in tech.

Recently chatting with her at SCaLE, Chloe clued me in about someone close to my wheelhouse: “Opera Singer turned Software Engineer” Catherine Meyers. Her Mozart could’ve been an Engineer talk correlates music theory and counterpoint to building software. She emphasizes the similarities of pattern recognition between the two disciplines, especially as it relates to learning. Preparing a new piece or constructing a score interpretation is 100% relatable to the software engineer learning a new language or building in a new framework. I especially love how she emphasizes the importance of having different types of minds together in one group, in other words, don’t be afraid to hire a music major.

These folks that are trained in synchronizing common ground through the experience of playing in musical ensembles carry a distinct advantage. Diversity and communication are the trademarks of both successful musical groups and software teams. Establishing common ground through shared learning is crucial to the modern team building and operating distributed software. It is obvious to me that ensemble music not only relates to complex systems but also to highly coordinated teamwork, to wit:

I’ve conducted marching bands, wind ensembles, and jazz bands... They are complex, highly interdependent systems.

Tanner Lund (SRE at Azure)

This couldn’t be more true! I have been in an experimental vocal trio, a free improv quintet, a blues band, a jazz group, an early music ensemble, a 330 piece marching band, uncountable operas with full orchestra and cast, chorales of few to many, piano accompanists by the dozen. There is nothing in any of these situations that doesn’t require common ground and shared experience.

In her blog post, The Origins of Opera and the Future of Programming, Jessica Kerr (Lead Engineer at Atomist) introduces the concept of symmathesy, a state of collaboration that is integrated with learning, all as part of the system (even the tools are team members). She does so by taking us back to the foundations of modern opera in Italy, adeptly adopting the term ‘camerata‘ from the Florentine Camerata example she uses, applying it next to social circles of painters and artists in later centuries, and then to teams creating software. She considers how our mental models evolve and how common ground is reached through the shared experience of a varicolored team from all sorts of different backgrounds, ultimately making the team more powerful. “We are developing whole new ways of being human together” she states, and to me, that’s nowhere more clear than with multi-disciplinary collaboration.

The CS / Math / Music crossover is clearly one I have encountered a lot, from early on. While I was studying voice and music technology in undergrad, I vividly remember walking with a good friend in downtown Blacksburg, and her saying to me “I could never major in music, there’s too much math.” I used to laugh at this because there’s really only as much math as that you wish to encounter. Want to dive into tempered tuning and Kepler’s mathematical ratios for perfect fifths? Go for it. But you know, these days I look at that statement through different lenses. Yes, there is just enough math! And if you study music, you may run the risk of studying computer sciences. * Please be aware that some practitioners of both have reported sudden bouts of inspiration and lightheadedness due to euphoric discoveries that cross disciplines.

On the subject of cross-discipline study…

I find a lot of what of I learned in music performance transfers directly to public speaking.

Amy Tobey (SRE at GitHub)

As a student of opera, I couldn’t agree more. Long ago I had a theatre teacher tell me “you will never be happy not onstage.” I didn’t know what she meant until I myself experienced exactly the same kinds of challenges and rewards with giving talks and public speaking as I did singing Mozart or Fauré in front of large audiences. I discovered that my experiences onstage presenting music are exactly the same headspace as when I’m talking in front of a room of computer people about distributed systems.

When I gave my talk Measuring Distributed Databases Across the Globe at Southern California Linux Expo 13x (2015), a very enthusiastic artiste-cum-techie approached me afterward. They had decided to move into software engineering after art school and said my talk, which combines theories and ideas in music with those of observing and operating huge distributed databases, was inspiring. They had been trying to reconcile their internal ‘artistic identity and self’ with their passion for coding, and I had opened their eyes to a new way of considering it.

I will never forget this. It was exactly like having someone come up after one of my experimental music performances (which are also heavy with technology) being blown away and asking how things worked. Even if one mind out of a thousand is influenced by what I’m doing, whether it’s music or technology, I’m building the camerata through sharing.

Eureka! The audience contributes to symmathesy, too. There are tons of examples; the feedback of a crowd during a DJ set, participants of David Tudor’s Rainforest, someone opening a candy wrapper at the symphony. There’s no getting away from the audience, however small. John Cage preached throughout his career as an avant-garde composer: listeners are participants. Every performance is something new because there are different people in different places every single time. The title of this piece is a play on Cage’s Musicircus, an amalgamation of disciplines and performers occupying the same large space together with the audience.

So consider the sheer experience of music and how it feels oddly in-step with how we think about technology, and take it one step further. Many of us trained in the practice of music consider it so much a part of our bones that it seeps into our work, we don’t have a choice, the pattern recognition pathways seem to match up. They fit into our psyches in a comfortable way, share the same affinity for abstractions, incorporate complex and overlapping systems that stretch the human capacity for comprehension.

Do I have an aptitude for music because I’m successful in understanding the practical application of complex things? Or do I have an aptitude in technical operations because I have been trained in music and theatre arts? Doesn’t it make sense that the diversity of music is a human quality, not a disciplinary one? Why should our work as computer scientists, software engineers, database operators, and network gurus be any different?

Temple Grandin talks about the world needing all kinds of minds. In her view, different people fall amongst three “categories” of thought: Picture (common on the autistic spectrum), Pattern (code, math, music), and Verbal (writers). It resonates with me a great deal because I always feel like I’m thinking in ways others are not. More specifically, though, it underlines how different styles of thinking can bring entirely new perspectives on a particular question, and how crucial diversity is to human endeavor, if not evolution itself.

This post started out as me being passionate about a subject and wanting to search for others who were. I had some bullet points of my own to insert, but as I searched and connected with others in the field, I learned new things and this blog transformed from a list of Twitter handles to real human relationships. I have grown to understand what Kerr means when she says that ideas are for sharing and collaboration, not hoarding and heroism. I for one am claiming my spot in the camerata that pushes us forward through a New Renaissance.

Weirdness of Experiment

My job in ops has always been to keep things running. I never considered myself “working in software”, but have recently begun embracing the fact that I do. What I accomplish as an operations and infrastructure engineer is part of the system, it isn’t dislocated from its composition.

Relatedly, I have been considering the nature of the experiment in Chaos Engineering. How continuous verification is becoming a crucial part of the complex systems we build because there really is no end. Developing a software system isn’t just about writing it, it’s also every bit as much about running it. Unless there is some kind of evil catastrophic end-game planned from a volcano island hideout, most of us want to keep them running.

I’m big on experimental music. You probably know what I mean when I say that, but you might not because genres, in general, are horrible overgeneralizations. Similarly, after the composer John Cage had written his “silent piece” in 1952 (see also Living 4’33”), he seemed to have a struggle with the concept of calling the music composed by him and others he admired experimental.

In science, we often think of an experiment as a method to (dis)prove a hypothesis. We perform experiments to answer a question or assertion, often during the process of reaching an end goal. To Cage, this implied that calling something an “experiment” meant it was not complete, not finished. That there is a final state determined by products of the experimentation, and he thought that his (and others’) music was complete when performed. There was no “final state” that was decided as a result of an experiment either succeeding or failing, if it was itself called experimental.

Cage revised this view, however. He began embracing the term and actually ended up preferring it. The reason for this is the way he evolved to think about the context of sound. At the beginning of the decade, he experienced an anechoic chamber (an “echoless” room) and the non-presence of total silence, because he could hear both high and low sounds — explained to him by the engineer as his nervous system in operation and blood circulating, respectively. Whether or not that is physiologically probable, he had the now famous revelation that it is impossible to remove sound completely. 4’33” and an entire philosophy about the nature of sound and silence in music was not far behind.

To him then, the moniker experimental came to mean that which is undiscovered, because even if a piece of music requires certain sounds, environmental sounds are impossible to predict. This experimental music isn’t about the search for failure or success, but an experience of discovery, where questions become more interesting than answers. When applied to composition, each performance of a musical work is always new and different due to its context and sonic environment. Indeed, it is impossible to know ahead of time any structure of the interpenetrating sounds both intentional and not, themselves independent and unique (whether or not they are consonant). It is in fact in a total state of chaos, each and every time.

When complex systems run, they do so at the hand of indeterminacy and randomness. There certainly is a “steady state”, but it is continuously in need of verification. Just like Cage observing that no performance of a musical work is a repeat, the nature (structure and form) of distributed systems we operate cannot in truth be predicted with any kind of regularity.

So while it is useful to be very specific in defining and running our Chaos experiments, the nature of what we’re doing is more about asking questions and making discoveries, not testing for answers we already think we know or think we can guess. The “breaking things in production” mantra implies we are interested in failure when what we’re really interested in is what was determined and what questions arose, good or bad.

Appendix

Here are a couple of PDFs taken from Cage’s writing that highlight his viewpoint on the subject of “experimental music” as a title of what he did.

  • Experimental Music: Doctrine (1955) ::: This article, there titled Experimental Music, first appeared in The Score and I. M. A. Magazine, London, issue of June 1955. The inclusion of a dialogue between an uncompromising teacher and an unenlightened student, and the addition of the word ”doctrine” to the original title, are references to the Huang-Po Doctrine of Universal Mind.
  • Experimental Music (1957) ::: The following statement was given as an address to the convention of the Music Teachers National Association in Chicago in the winter of 1957. It was printed in the brochure accompanying George Avakian’s recording of my twenty-five-year retrospective concert at Town Hall, New York, in 1958.

The photo above is the album cover from a release by Craque called Meat Hacker.

To Build or To Buy, That is the Contradiction.

It’s dead simple. Focus on your team and product.

Yes! An easy tweetable answer! Except that it’s loaded with questions and assumptions. Instead, what I will talk about here are technology decisions. Those that really matter are about things that elevate your ability to build what you are building at the velocity you need.

In Operations and SRE, we are not only tasked with evaluating and testing technologies, we are often requested (or sometimes instructed) to support the decisions others make. This isn’t an ideal situation, and in some cases, impractical because it feels like a policing. In fact, it might sound a little selfish to think that SRE needs the ability to say “yay” or “nay” to a decision. If you think that feels slightly siloed, you’d be right. I prefer working with all vested parties to come up with a solution that fits the problem, not just make blind leaps of faith on a particular platform. Flexibility and cooperative evaluation work really really well, believe it or not.

In my experience, it takes a minimum of six months for a team to work up a brand new (to them) technology or platform and have it be supportable in production. Even then, if the software team building it won’t be permanently oncall for it (which is something else I believe), they should at least be on the hook for a minimum of six months further as the kinks are worked out. This seems fairly straightforward for something like a development framework or a particular platform specific to operating the software system in question. For example, adopting Redis as a DBRE-supported platform in production. There are clear reasons why the software needs this kind of ultraspeedy k/v store, and building up widespread support in SRE for it means other software teams benefit. That is a highly opinionated decision and one that’s fairly easy to make especially because it’s free!

But is it? In contrast, what happens when you need to fulfill the needs of infrastructure? This is where Operations tires really hit the road. This doesn’t mean it’s a siloed decision, for example introspecting the system is as important to reliability engineering as it is to the folks developing the software. The Redis example above shows how technology decisions become a crucial part of the collaboration among everyone involved building your product, whether it be dev or sec or ops or everything in-between. However, it doesn’t quite reflect the reality of infrastructure requirements, particularly tooling that supports the needs of reliability. “We need monitoring and we need it yesterday!”

So, I posit: There is no “Build or Buy”. There is only Buy.

I build electronic instruments. My product is the music I make, and one the central philosophies in my approach to improvisation and performance is the concept of “original sound”. What music does a plucked cactus make when amplified, or how can a mercury tilt-switch and photoresistor create sound from electrons? The top picture here is of a breadboard, a prototype of a device that will eventually drive something called a “Voltage Controlled Oscillator”, or VCO. To the left is a photo of a professionally designed, tested, and built VCO. I have no need to design and build a VCO, even though fun, it is a distraction – both in time and money – from my main purpose: the creation of a new kind of analog synthesizer controller. I’d rather buy the VCO and the accompanying support of the experts who not only built it but are passionate about its success. Then I can focus resources on my creativity.

The technology decision to be made is heavily informed by what kinds of resources you want to spend on it. Building a platform in-house, using your own people, is sometimes only feasible with very large companies that have the ability to staff for this, and often have custom circumstances. It’s not hard to notice that a few very successful software platforms come out of organizations in this position. These teams also must deal with all the infrastructure costs, maintenance, reliability, and complex aggregations of data and transports. These problems have indeed been solved many times over by third party SaaS/PaaS/IaaS solutions, but sometimes the amount of customization required demands it. Or the development of these systems may be so clearly aligned with the company objectives that it is a non-question (e.g. Netflix pioneering Chaos Engineering or Google producing Borg/Kubernetes).

These days, such cases are the exceptions. Most of us are with companies where this luxury and concentration of engineering ability aren’t present. It may make a ton more sense to choose an expert in the field, whose company mission is specifically to provide this need. The argument that Ops Is A Cost Center is underlined, because the focus should really be on the product. “We’re good enough we can run a custom log aggregation stack at the same time we’re developing this completely dependent but orthogonally related product” is probably an approach that should be questioned, unless of course you are prepared to buy those resources through hiring and operational expenses.

Either way, you’re building something. Recall the decision to choose Redis, and how long it took to enable that system in production. The same will apply to any in-house platform or tooling. Nevertheless, it’s a tradeoff. To buy resources for building in-house or buy a third party SaaS, each have their own layers of complexity, complication, and frustration. Yep, it might be fun to build, but is embarking on such a project really the right choice for the team and the product?

Like many in Ops, I once considered the “Elastic Search / Logstash / Kabana” (aka ELK) stack for datacenter log aggregation. All its various pieces are a fun complex thing to put together, and it’s an extremely useful resource for gathering and displaying events. All these pieces are freely available things, but our SRE teams were already pretty busy with the task of running our own product and other custom bits of infrastructure. It would take at least a single SRE’s full time and attention to keep the stack running across multiple server farms of 10,000+ nodes. Not to mention training and documentation for others. Maintained, tested, resilience-hardened, budgeted, the list goes on. I’m definitely going to be “buying” a lot here, and it’s a messy looking BOM. ELK’s competitors at the time ranged from fairly entry-level cloud-based SaaS to “Enterprise” packages, but one vendor, in particular, would be a great solution. Was there a cost and the administrative toil of negotiating a contract and keeping a vendor relationship going? Yep. It’s also a single invoice, takes much less time from the SRE team to manage, and is much simpler to integrate with the farm.

Would you rather focus on building the instrumentation into your software and have a team of external experts guide, consult, and ultimately provide the intelligence platform to which they have dedicated their careers? Or would you rather split focus and build your own customizations into the infrastructure? It’s not black and white, either. One method may outweigh the other depending on whether you’re bare-metal or in the cloud. There may be security reasons to do one over the other. As the software product matures, these needs may blur. What needs to be bought may be simple and small or complex and large, and grow in either direction.

So let’s be clear: a third party platform isn’t automatically “more expensive” than creating an equally performant service or fulfill a particular infrastructure reliability need in-house. Do the research and compare, make the investment that makes the most sense for your team.

Don’t be fooled, though. Either way, you’re buying it.

Living 4’33”

I haven’t experienced Mute’s STUMM433 release yet, it’s not due out until May. The proceeds from its sale go to charities, so that’s a big huge plus for it already. Pulling big names like Depeche Mode and Moby will hopefully make for good sales.

This is also not the first time 4’33” has been “recorded” for the purpose of a record release and not as part of a live performance. Frank Zappa has done it, Pauline Oliveros and the Deep Listening Band plus others, and some even got sued for it. However, this tweet from the John Cage Trust, of all places, put me on guard:

Immediately, I think… “cover”? How? What is this photo?

So once I start looking into this, I learn about the accompanying videos. The Laibach one featured on the Mute website (which cleverly includes a shot of the Cramps Caged/Uncaged homage from 2000) is a kind of short silent film. I imagine many (if not all) of the other videos will be similar. So it’s telling that they refer to it as a “cover” and use words like “interpolation” to describe this collection.

The terminology now makes sense to me. These are not performances of the original score, but takes on Cage’s own expansion of the idea that it could be performed as anything, at any time, for any duration (like the versions of 0’00” from Song Books). Presenting an alternative action during the “silence” of a representative version of 4’33” is not so much a reading of the score as it is an interpenetration of events. Which is fine, even enjoyable. Nevertheless, the very idea of 4’33” in the popular eye is surrounded in jokes and doubt, so it is ironically funny to think that a recording imparts the same sort of wonder that a live experience of it does.

Spoiler: it doesn’t.

In addition to witnessing it several times, I performed 4’33” at my undergraduate recital (on classical guitar!). Let me tell you… it is much more than just the sounds around you and what leaks in. It is a visceral experience that isn’t captured in a recording, where the very best of intentions can only pay tribute to the surreal actuality of sitting there, enduring the seconds as nothing happens. The tension in the audience is very VERY real. Sometimes funny, sometimes raucous, but regardless the performer must stay focussed. There is simply nothing that gives the work the heft that Mute describes without experiencing it firsthand.

As a kind of funny postscript… when I was near the end of my time in grad school, where I rigorously studied voice and Cage’s music, I was asked to participate in a production of Theater Piece, a work of simultaneous but unrelated events. Somehow, CF Peters (the sole publishers of Cage’s scores) heard of this production, that I was involved, and assuming me responsible tracked me down to demand royalties be paid for staging the piece. Except – for once – I wasn’t staging it, I was merely performing, and explained as much. I wonder… had I said we weren’t performing Theater Piece, only doing a cover of it, if they would have left us alone. 😉

Finally, I recommend No Such Thing As Silence: John Cage’s 4’33” by Kyle Gann if you’re curious about the mythology around this composition. It is far from my “favorite” piece of Cage’s but is assuredly the most important in American musical culture.

So You Want 10 Albums?

Ok Bob. You asked for it. With one caveat.

In no particular order, post 10 of your favorite albums, one per day, which made an impact on you. Post cover, no explanation, nominate someone each day to do the challenge.

I think these games are fascinating windows into peoples’ aesthetic, but because of FB’s sorting and selection criteria, I see maybe TWO of a particular friend’s ten-albums-once-per-day series. So, if I am taking the time to compile, I want to make sure you see them all. Plus I’m all into the cross-platform sharing thing and doing this on my blog helps me spread the word of good music (and keeps me writing). And being a DJ there are plenty of previously posted lists, check out previous blogs for more of my listening habits and recommendations.

One final note: this was fucking hard, and I had to mostly stick with certain genres (I can’t even begin to describe the numbers of vocal music recordings that have influenced me, for example). To cull influential albums down to 10 is worth the challenge alone… indeed, mine goes to 11. It could be an entirely different list tomorrow. Nevertheless, every one of these has a story (and not necessarily musical ones), but in keeping with the guidelines of the challenge, they will remain untold… for now. 🙂

Pink Floyd ::: The Dark Side of the Moon
Jethro Tull ::: Stand Up
Sound Track ::: The Cooler
Miles Davis ::: Kind of Blue
Naked City ::: Grand Guignol
Autechre ::: LP5
Shpongle ::: Are You Shpongled?
Peter Gabriel ::: Us
Squarepusher ::: Selection Sixteen
Lusine ::: Serial Hodgepodge
John Cage ::: Sonatas & Interludes for Prepared Piano (Joshua Pierce, piano)

Musical Intuition meet Technology and Chaos

Today I read through the InfoQ eMag on Chaos Engineering, and was struck by John Allspaw’s (@allspaw) contribution because it reminded me of something I jotted down on a sticky at my desk a few days ago:

Intuition is valid because it is learned like jazz changes.

I’m pretty stubborn and refuse to accept that music is merely a hobby of mine. When people ask me if electronic music or singing is my “hobby”, I am wincing inside. So a question often on my mind is: how does the intuition I have when performing and composing music connect with the work I do as a technologist?

Some musicological background might help. One concept in learning how to improvise (jazz or otherwise) is that you have developed an intuition built around internalizing the materials and form of the piece (or genre) – like scales, chord changes, or rhythm structures. This is different from the more lizard-brainy concept of instinct. Think about a blues progression, the foundation of music you hear every day, everywhere. You know intuitively the chord progression and timing is “right”, even so much that anomalies and departures come across as emotionally significant. The rest is pop history.

But you, homo sapiens, do not have this chord sequence pre-programed in your DNA, it isn’t something that is instinctual. By the same token, great technology leaders develop good intuition (expertise over hundreds of interviews) when hiring engineers but never rely on instinct (oh I just have a good “gut feeling”). The best DBAs have an intuitive understanding of their platform (you want to do X, but did you think of Y+Z?), but there’s nothing instinctual about it.

It is not a stretch, then, to recognize that intuition in improvised music can be directly compared to how Allspaw writes about the “mental map” that engineers develop. They each have their own subjective view on relevant (but overlapping) parts of the system and are challenged when relating each substrate to theirs. For instance, a phenomenon known as “fundamental common-ground breakdown” (Woods & Klein: Common Ground and Coordination in Joint Activity) happens when what I describe as intuitions (accumulated individual learnings about the system) are assumed knowledge among participants, good or bad. Part of the game is learning how to harmonize these separate threads of experience, avoiding costly coordination surprise and re-synchronization… and trust me, I have been in plenty of rehearsals and narrowly saved performances that fit this description!

The important point here is that a system becomes more complex as it grows dimensions, shrinking the capacity of any one person to comprehend the whole thing. Therefore we rely on shared and discovered knowledge to fully grok these fascinating systems. Take any ensemble of musicians: as it grows in membership, individuals gradually lose the ability to contain its myriad relationships in their mental map, so coordination and integration become a matter of listening and rehearsal experience (both modes of communication). Oh and it characterizes the music, too. Building intuition about how to play a part in an opera is much different than in a free improv vocal trio. Orchestrating ten thousand linux containers in a cloud provider doesn’t compare to managing two rows of server racks at the datacenter downtown.

Technologists grapple with the task of building and sharing intuitions about a system because understanding an entire system contributes to what we know about making it more resilient. Communication is key in either musical or engineering teams, collaboration on understanding the whole is no exception. Our mental maps should be adaptable to constant updates, and practices like Chaos Engineering that make discoveries in complex system behavior are supported by this kind of cross-pollination and proliferation of our combined understanding.

A quote from Allspaw’s article highlights it well:

Maybe the process of designing a chaos experiment is just as valuable as the actual performance of the experiment.

– John Allspaw, Recalibrating Mental Models through Design of Chaos Experiments

The use of the term “performance” is apt. We’re familiar with this concept: practice makes perfect. Taken further, the experience of practice is necessary such that the result is merely an extension of practice. It takes meticulous work to understand a piece of music to the level of having an intuition about how it operates, and the same goes for building experimentation that challenges what you think you know about complex software. The results of the “performance” can be enhanced by a focus on understanding the system’s design and steady state (i.e. nominal condition), what we would call the language of the musical work. It is as if the performance of the event naturally evolves from learnings gained preparing for it.

Imagine you are a jazz musician, you have gone through years of studying scales and changes and charts and recordings of a particular artist, and have built a capability for understanding how the language of their music works. One evening at a local club, your dreams are fulfilled, you’re in the audience and invited up for a set with them. You intuitively know how this person plays their music, as it has been a guide for your own. But when you’re jamming together, they do something indeterminately that informs your intuition in a way you would have never discovered yourself. Not only has the process of designing your inevitable collaboration been valuable to understand what you thought you needed to know to play like your biggest influence, but it also served as the basis for learning something new and unexpected.

Whether it is free improvisation or interpreting a through-composed piece of music (and everything in between), there is a certain amount of experience and training informing the performance. Eventually, when we’ve practiced enough, the music itself steps out of the way and intuition takes over. I think this is where my musical performance connection with technology starts: once you understand the fundamentals of the system, let the presentation of the system get out of the way, and you’re in a better place to evolve your mental map and gain further intuition through disciplines like Chaos Engineering.

Successful Tech Teams and the Superb DJ Set

The term disc jockey was first coined in 1935, referring of course to the phonograph discs that had become popular for recorded music. Forty years and dozens of musical genre shifts later, the Technics SL-1200 direct drive paved the way for turntablists, still concerned with round artifacts of engraved sound, but doing more live manipulation than just selecting and playing tunes.

Since then, it has come to signify any presentation of recorded music mixed for a listening audience. Drum machines and synthesizers snuck their way into the mix as hip-hop and house music were invented. The use of vinyl record “discs” morphed into Compact Discs and even cassette tapes and reel-to-reel mixing. Eventually, this led the activity of “DJing” into the most widespread application today: digital files played by any variety of control surfaces, from none at all to touch surfaces to spinning metal platters to full vinyl replicants that signaled transport controls to the playback system storing the file.

Building a set is another responsibility of DJing that is done in all kinds of different ways, with all kinds of different music, in all kinds of environments and settings. Whether you’re a radio DJ performing to an unknown audience or at a club holding up the dancefloor, there is an elegant methodology of diversity involved. Even when it’s a prescribed style, a lot of preparation goes into the Superb DJ Set.

What makes the Superb Set? I divide “the work of a DJ” into four basic categories:

  1. The homework of selection. What do I like or want/need to include? (materials)
  2. The spectrum of choice. What tracks make sense with what other tracks? (methods)
  3. The environment of creation. What is the setting and acoustics? (form)
  4. The flow of performance. What happens controlled in real time? (structure)

Regardless of medium (disc or no disc), DJs accomplish each step in different ways depending on who they are. It always involves the collection strategies, stylistic tastes, requirements of the event, attention to form and/or/not structure, personal goals (musical or professional), and the audience itself. The DJ may be a completely free improviser, an experimental sound artist, a wedding MC, a techno junkie, an EDM opportunist, a desert party space explorer, a masterful breakbeat juggler, or an underground house magician. I believe they all include some version of each of these categories.

Ready for this? The same tenets apply to building a Successful Technology Team. Compare:

  1. Homework. Who do I want or need to include on this team?
  2. Spectrum. Who has the right balance of talent that makes sense on this team?
  3. Environment. What are the requirements of the business?
  4. Flow. How does the team interact, collaborate, and ultimately deliver?

Just like the diverse history of DJs and music, tech teams span a vast range of expertise and function, not to mention approaches to organization. I’m talking about everything from day-to-day (or should I say, sprint-to-sprint) Agile teams and purpose-built developer tiger teams to “T-shaped engineers” on multi-functional operations teams and necessarily heterogeneous IT departments. Each of these individuals are like specific tracks on a record, the pieces of music themselves, expertly selected and set into sympathetic vibration and choreographing their own results. The audience is not only the customer but also product managers and project coordinators (not unlike event producers and organizers), and many other parts of the technology business.

In fact, where the DJ and the Technology Leader (let’s call them a ‘TL’) really intersect is diversity in construction and release of control. For example, the homework of the DJ might be crate digging or further exploration of known things online. The TL has the homework of digging into the talent pool and recognizing what they want or need on the team. Both are fraught with mistakes and hidden gems. The DJ selects a diverse spectrum of tracks to mix, while the TL should espouse diversity among teammates mixed together. The DJ’s venue is analogous to the TL’s workplace, what is needed not by the music/team but what shapes the work they do. Flow from track to track and section to section is no different from the team’s ongoing efforts to work together, match each others’ strengths, and support each others’ weaknesses.

The resultant similarity between a Superb Set and a Successful Tech Team is not really a tenet as much as it is the desired effect: the DJ/TL steps back and lets the music/team combine and take control. Once the DJ has prepared their set, when it is in full swing and engaging whatever audience, specific control is lost. External forces – like the energy of the dance floor, requests made by listeners, anomalies and failures in the sound system itself, or even just a scratch in the record – indeterminately affect the materials, structure, and flow. The Superb DJ is ready for these challenges, to help guide both the rules under which they operate and the engagement of participants involved to a harmonious and fulfilling goal.

If that’s not like building and running a great team, I don’t know what is. We deal daily with interruptions and unknowns in running software on distributed networks. Humans are humans, and like scratches in records, they can cause mild-to-severe issues within the team. Requests made by product (or other) managers can interrupt, company events will necessarily intertwine, and unforeseen losses of momentum and energy can mean nobody is dancing anymore.

Sometimes as a DJ I will buy a record because I like the sound or the album art or even just the composer’s name, only to find that its BPM doesn’t match anything I own, or its sound is too harsh to really use in a public setting. It is why it is crucial not to only consider one of the four tenets, but to carry the desire for something great as a thread through each phase of the approach. Intuition is good, but also including close evaluation and clear decision making is better. In that sense, careful listening and observation is as important in musical interpretation as it is managing diverse human personalities and lifting them be successful technology leaders themselves. It might be true that everyone is a DJ because they can just “press play”, but the Superb Set – and the Successful Tech Team – is the reachable dream.

LA Phil graces Cage’s Europeras 1 & 2

John Cage liked to borrow. Whether in musical style or approach to theatre, the materials themselves are often not his. Musical scores evolved into graphic iconography or instruction governed by brackets of time and/or duration. Written pieces were amalgamations, mesostics “written through” other authors’ work, or more “cut-up” style constructions of various texts.

It applied to his philosophies, too. He is known to take bits and pieces of Eastern practice and weave their concepts into his worldview and compositional process. Concepts like the Huayan Buddhist “interpenetration” of all things were slightly bent by Cage, who found them useful for composing, but maybe slightly ignoring (or denying) any inherent interconnectedness.

Such synergy is often attributed to improvisation, like a jazz combo. On the contrary, Cage was very much about the accidental interpenetration of elements by downplaying any relationship importance between them at all. The idea of “playing off each other” as if playing jazz was not allowed, simply because it wasn’t scored that way.

These were simply concurrent events that coincided in layers against each other, necessarily connected by nothing but the experience, indeterminate and interpenetrating. It seems to imply chaos, and that is precisely how my wife described the LA Phil west coast premiere of Europeras 1 & 2 to me afterward: “too much going on for your brain to comprehend” and “the weirdest thing you’ve taken me to yet.”

Now that’s saying something! We’ve participated in Long Beach SoundWalk multiple years, we’ve seen some outrageous installations and concerts. Even my music is pretty damn strange. Whether or not this actually hit the tip of the Weirdometer, there was one thing we agreed upon: nothing was happening, and everything was happening.

My wife and I have never been to a movie studio lot, so it was a new experience just arriving and finding our way back through the closely stacked identical warehouses to Studio 23. The Sunday matinee crowd fascinated us. We guessed the throng held Cage fans, opera fans, music students, season ticket pass holders and supporters of LA Phil, friends of the cast and orchestra, and even scattered fashionistas.

Inside was a simple proscenium stage configuration, the audience rising up and back, fly loft and everything built into this huge soundstage. On each side of the main stage were three columns of orchestra, with most major instruments represented at least once. The stage itself was an 8×8 grid, marked with numbers 1-64 to represent hexagrams of the I Ching, which itself is used to drive the chance operations required for creating the performance based on Cage’s score.

The distinction of 1 & 2 is a programmatic one, they have always been performed together as a 90 + 45 minute show. So we settled in for a good bit of nothing we ever expected. Although I have never seen the Europera scores first hand, I have performed many other Cage pieces that revolve around the same concept: time brackets of performed material according to decisions arrived through chance operations. Many of his scores like this were performed simultaneously, so that not only did each individual composition interpenetrate with itself, it also did with the other piece.

The Europeras are a culmination of this approach by Cage, and not only because they are among the last works he composed before his death in 1992. Chance procedures determine every aspect of the production, from costuming and scenery to blocking and the placement of arias selected from the standard repertoire. In the midst, selections from Truckera (a tape of 101 layered European opera fragments) were broadcast stereoscopically across loudspeakers above the audience, giving the sonic illusion a “truck of opera” was rattling by the performance and drowning everything out.

The entire work becomes a brilliant collage of sight, sound, dimension, and movement. There are entirely mind-bending Fluxus moments of absurdity, subtle sequences of sublime beauty, and a good amount of unintended comedy.

The LA Phil performance mostly held true to Cage’s intent. There were dancers moving independently of both scrim- and prop-based scenery (also sequenced with chance operations), but also acting as stagehands and crew to move things around. Very seldom it starts to drift, as when these same dancers become engaged with the opera singers in their individual scenes.

I have to hand it to the singers in this production. It is not hyperbole to state that seeing this performed is at the top of my bucket list. These folks were charged with non-traditional blocking, ignoring every other musical cue they hear around them while staying in-tune as possible, having to watch the large count-up clocks posted on each precipice of the stage, navigate indeterminately moving scenery and other actors, all while performing a fully committed aria wearing costuming and performing blocking both separately derived by chance operations and completely unrelated to the entire way they were taught to interpret an aria.

Some did this better than others. One of the more successful sang “Oh du, mein holder Abendstern” from Wagner’s Tannhäuser, outfitted in full astronaut garb (minus helmet), the entire time both lying and moving around a hospital bed, while any number of other arias, scene changes, and lighting queues happened around him. I found it masterfully performed against a magnificent bed of chaos.

Many of the singers’ performances were like this. As was the orchestra. There was an incredible amount of commitment in this show that is absolutely vital to Cage’s music. The entire ensemble – from stage crew and props to performers and designers – dialed in on this aspect of performing Cage and it comes through.

Lighting queues and design were scored completely separately as well, often focussing on the audience, into the ceiling of the warehouse, across the back of the building behind the open stage area. Scrim backgrounds, some looking quite ancient, dropped at various elevations, frequently covering an entire “scene” behind it due to the chance operations involved. Sometimes individually numbered squares onstage were illuminated, amorphous areas of color appeared, and more than once a strange ladder descended made entirely of what appeared to be fluorescent rods.

Sopranos galloping on life-size fake horses carried by dancers, a baritone singing (Mozart?) while preparing a steak on a hot plate with chopped vegetables, a tenor in drag performing what I think was from Rake’s Progress (Stravinsky), the Toreador Song (from Carmen) staged to a commercial being filmed for hair products. Some scenes had no singing at all, like the baritone who sunbathed in 70s garb for what seemed like an eternity before he finally got up and sang a short aria. Or the girl backward in a running belt vibration machine drinking a coke with a straw in a full Wizard-of-Oz Dorothy outfit (complete with ruby slippers) and maybe sang, but maybe didn’t. Or the soprano auctioning off small statues to members of the orchestra, complete with gavel banging. Yoga and Queen of the Night, I think it was?

Except in one rare case where a tuba belted out Flight of the Valkyries, the orchestral parts were not as immediately recognizable as the arias, but equally enjoyable in the mix of it all. I especially loved interjections from percussion, like sudden cybals and tympani. The effect of this differentiated musical tissue was like an extended meditation on simultaneous sound against a landscape that was constantly in motion.

Overall, Europera 1 was more recognizable for me and felt like it breathed with long sequences, wonderful moments of silence where the HVAC in the huge studio warehouse took its solos, combinations of the orchestra that did not feel at all chance-derived, and I felt drawn in the entire time. Europera 2 felt more compressed, more frenetic, maybe more immediately interesting but definitely more chaotic. Nevertheless, it ultimately wasn’t as memorable as the first, and yet felt more voluminous and energetic. Even the chance-derived synopses (related to nothing) reflect this telescopic pattern:

1

He falls deeply in love with a beautiful streetsinger who staggers into the hut. He buys a love potion. Her candle goes out and impressed by his wealth she decides to marry him on the spot. The would reveals that after three years he will have himself crowned Emperor with the evil one’s help in exchange for his love. At first she flees; whereupon he gathers all his strength, she becomes passionately attached and begs a hermit’s refuge.

2

She sells his soul to her father with the aim of improving his impaired finances. Even her loving relatives are shocked. They rescue him. He retires. She agrees. Torn, they, in shame, pardon all conspirators. He agrees to marry her. She kills herself. He is chosen the victor.

After the final curtain dropped at precisely the correct time, we left the soundstage, the sun barely setting as we found our way out of Culver City. I reflected on how difficult it is to convey the real power in this category of Cage’s compositions without experiencing them firsthand, live and in person. In addition to the power of simply listening and seeing, his works of this kind express a sort of pandemonium that is at the same time masterfully controlled and undoubtedly a “work by Cage”: meticulously crafted experiences in anarchy.