The reciprocity of an interactive experience between humans depends on the absence of a script and on a mutual willingness towards the development of empathy. The mere existence of this empathy can elevate what would otherwise be a common encounter to the quality of an unforgettable moment. The relationship that we establish with machines is, on the contrary, one between masters and slaves: we expect them to be always available to promptly and precisely execute any task that we can formalize in their language. The only emotional responses they get from us is the anger and frustration when they stop responding or make mistakes (or even point out our own mistakes). The same happens in an interactive installation, where the machine turned into a performer is obliged to react to any stimulus in a diligent and submissive way, for the exclusive pleasure of its temporary master. The question seems obvious: can there be interaction without choice? Can we build true empathy between beings that are not free?
resono is an interactive installation that builds on the ideia that the cornerstone of autonomy is the capacity to say: no! It consists on an ecosystem of fifteen beings that wait, with different balances between anxiety and curiosity, for the arrival of a new visitor. It is this visitor that has the responsibility of taking the first step towards an interaction, conquered by the kindness of the approach and the affection in the singing towards a small group. Slowly, and if the visitor is patient, faint resonances start to emerge, as the beings learn, through imitation, how to communicate with the unknown stimulus. From this point on, the history of each interaction will be unique and dependent on the delicate balance between the states of mind of all performers. It is possible, in truly exceptional moments, that all the beings of the ecosystem will join a collective performance. Every successful interaction will live in the dreams of each of the beings of resono and will contribute for the development of their unique personalities. If reciprocity is in fact an essential element of empathy, the same will happen to the visitors.
Fig. 1: resono at MNAC · photo: José Paulo Ruas and Luísa Oliveira (DGPC/DDCI/DDF)
The 15 beings of resono are permanently listening to their surroundings. When they hear something that they do not recognize, they can react with fear, which is made visible by the acceleration of their breathing (i.e., the rhythm of the lights). When they hear something that they recognize, they may start to react timidly by singing their own specific note. If the visitor strives to establish a dialogue and to encourage an interaction, the confidence of the beings can increase and their contribution will become louder and more confident. This initial interaction can then expand to the neighboring beings and give birth to a collective performance. Each performance will be unique, since it is entirely dependent on the initial stimulus and on the evolving musical memory of each being.
Each day, however, there will be a predictable collective performance: at night, when the room is dark and silent, and after the batteries are fully charged, the sleeping beings will dream. These dreams consist on the random chopping of the daily experiences and the musical memory of each being. This process — witnessable only by humans that silently wait as in a field mission — is a fundamental part of the assimilation of the daily experiences and of the continuous development the beings' unique personalities.
The exploration of resonances has been a major part of my work for many years: in a more strict sense, via the exploration of the resonances of enclosed spaces in sound spatialization; in a broader manner, via the use of the resonances of musical instruments as the basis of my formal and harmonic language, following the steps of movements such as spectralism. The process that gave birth to many musical languages and instruments — the act of exploring the idiomatic response of a space or a given object by repeating a stimulus — is implicit in the interaction mode of resono.
Fig. 2: Helmholtz resonators · photo: Case Western Reserve University
The work of Hermann von Helmholtz is a cornerstone of the development of timbre (deeply connected to the resonant characteristics of musical instruments) as a musical parameter during the first half of the 20th century. Later on, is was the basilar knowledge behind the movements that emerged around a psychoacoustic attitude, with an emphasis on spectralism. Of particular importance is his 1863 book, titled "On the Sensations of Tone as a Physiological Basis for the Theory of Music". In this book, Helmholtz describes an equipment for the analysis of acoustical phenomena that became known as the Helmholtz Resonator. This model — which is still used today as a device for the acoustical treatment of spaces — was later built as a quasi-spherical object that resonates to a given frequency and permits the quasi-isolated audition of a given component of a sound with a complex spectrum. Sets of several objects with different tunings (figure 2) became, in the late 19th century, an instrument of particular importance for the work on musical acoustics and psychoacoustics.
Fig. 3: element form generation software (6th harmonic)
These objects became obsolete after a few decades of use and are nowadays museum artifacts, but the impact they had on the knowledge on musical acoustics is felt on today's music with particular emphasis. The external shapes of the beings of resono were directly inspired by the Helmholtz resonators and, as in these 19th century objects, each being is particularly sensitive to its specific frequency and is solely capable of reproducing it. The frequencies of the whole set are the ones that correspond to the first main harmonics of two different harmonic series — G and B flat, distinguishable by the color of the porcelain element (respectively, black and turquoise blue). The external shapes crystalize the vibration mode of the spherical harmonic that corresponds to each of the being's own harmonic (i.e., the more complex forms correspond to the higher pitched harmonics). The shapes and their several layers were generated using custom made software (figure 3).
Even though each being is capable of producing a single pure tone (i.e., a sinusoid, which is also the dominant shape of the corrugated cardboard), the amplifier and loudspeaker were purposely made to introduce some distortion. This distortion is mainly constituted by harmonics of the original pure tone and, consequently, the beings end up emitting the frequencies of some of their neighboring beings, who can then decide wether to respond or not to this stimulus. The relative position of the beings in the exhibition space is thus a major contributor to the type of performances that the ecosystem is capable of performing and a great deal of care was taken to make sure that the common notes between the two harmonic series would be strategically placed near to the center.
The interactive approach of resono builds on a fundamental definition by Robert Rowe, that defines musical interactive systems as the ones that are capable of listening to a human performance, interpret it and, in a complete cycle, produce a musical outcome. A significant part of the work brought by this definition, amongst which some of my own previous work, consist on interactive musical systems that try to interpret and respond to any stimulus that they are exposed to, without any previous qualitative judgment. In recent years, I have been searching for ways to develop interactive musical systems that, on the contrary, only respond to stimuli that they are capable of recognizing — in lack of a better term, I have been naming them robotic music snobs. These robots, as human musicians, have their own musical preferences and interpret external stimulus in the light of their own references, striving to create the best conditions for the integration of their own musical vocabulary in the performance. The first robotic music snob that I developed was capable of responding to slow modal melodies with tintinnabuli, a composition technique developed by Arvo Pärt. During the first experiments, and as expected, the system did not react when I played, e.g., Beethoven or Ligeti. Nonetheless, as soon as I played something that could be interpreted as a modal melody, the system would initiate a dialogue that would quickly become an engaging and immersive interactive experience. After a few moments, however, this experience would tend to become repetitive and predictable.
The pertinence of these explorations was confirmed later on, when I created the second system. This system listened to, and responded with, punctualism-type textures, in the manner of some vanguard-music from the mid-20th century. Whilst the original idea was to have two very different systems working at the same time but responding alternatively, at one point a strange thing happened: I was playing a modal melody, the tintinnabuli was reacting and, suddenly, the punctualism system reacted not to my stimulus, but to the one emerging from the tintinnabuli. I immediately stopped playing and the two systems played together for a tiny moment, in a language that was not entirely predictable for neither of them. It was at this moment that I decided to continue exploring the idea of beings that are capable of autonomously developing their own idiomatic musical vocabulary from the biased interpretation of an unknown external stimulus.
resono is the first public exhibition of this line of work. Each being is based on an artificial neural network that was trained by observing the behavior of its specific frequency in more than 44 hours of music — containing representatives of more than two millennia of musical history from the five main continents — selected as responses to a common question: how to build a musical discourse based on resonance(s)? Due to the limitations of the hardware inside each being and to the computational resources needed for the training of these artificial neural networks, the training of these networks was done beforehand on a more powerful computer. Despite that, the daily stimuli that the beings of resono receive are saved in their memories and come as a form of dreams during the night (i.e., when the batteries are being charged). These memories are then chopped and assembled in a stochastic process that also manipulates the weights and biases of the artificial neural network. This process contributes to the slow development of the unique personality of each being and, consequently, to the evolution of the musical discourse of the resono ecosystem.
In spite of their references and autonomy, the beings of resono are incapable of starting a performance on their own: they always need an external stimulus (except when, as previously mentioned, this external stimulus appears in their dreams). This external stimulus, for the visitors, almost always implies some kind of singing, that should emerge not as an exhibition of technical virtuosity, but as the consequence of a simple willingness to establish a dialogue in the most interactive type of experience that I know: performing music together. In western societies — and as a consequence of the division of labour, as identified by John Blacking — we are early on divided between "musical humans" — i.e., the ones that have some kind of "musical talent" and that are almost obliged to do something about it — and the "tone deaf", who will be forced to accept being confined to the role of listeners. A significant part of my work is based on the idea that if music is an integral part of all known human cultures and an important element on the life of each of us in particular, that means that we are all born as "musical beings": a fact that the musical daily life of some non-western cultures seems to prove. I asked for the painting Concerto de Amadores ("Concert of Amateurs") , by Columbano Bordalo Pinheiro, to be placed right by the beings of resono as a way to symbolize the will to sing and make music together as a group of both performers and listeners. This painting is part of the permanent collection of MNAC and, by happy coincidence, a contemporary of the Helmholtz resonators: a traveller from a time when at least a portion of a small and privileged social class still regarded active music making as a relevant part of their daily life.
Even though further exhibits of resono can be made without Bordalo Pinheiro's painting, their site-specific dialogue during the initial exhibition was so powerful that they certainly long for a new encounter!
Fig. 4: schematic of inside of each being
The beings of resono consist, on the outside, on a structure of laser-cut corrugated cardboard and a porcelain element. On the inside, there is a 3D printed structure that holds a custom made PCB that includes:
a 32 bit ARM Cortex-M4 microprocessor;
an audio microprocessor that manages the audio input and output;
a battery charge management controller;
a 2,8 W stereo amplifier;
a 2250 mAh lithium battery;
a ring of 24 RGB LEDs.
This structure, visible on figure 4, is finally wrapped in translucent tracing paper, the only visible element inside each being.
Composer and performer of live electroacoustic music, Rui Penha was born in Porto, in 1981. He completed his PhD in Music (Composition) at the University of Aveiro, where he worked under João Pedro Oliveira. His music is regularly recorded and played in festivals and concert halls around Europe and North America, by musicians such as Arditti Quartet, Peter Evans, Remix Ensemble or the Gulbenkian Orchestra. He was a funder and curator of Digitópia (Casa da Música) and has a deep interest on music technology. His recent production includes interfaces for musical expression, sound spatialization software, interactive installations, musical robots, autonomous improvisers and educational software. He taught at several Portuguese Institutions (DeCA - UA, ESMAE - IPP, ESART - IPCB, ULP), and is currently an assistant professor at FEUP and researcher at INESC TEC. More info here.