Nautilus digital score (Win & Mac)

Nautilus (2022) is composition for bass flute, neural network and Unity game environment (2022). Challenges/ research questions

can a games engine such as Unity be integrated into a digital score without detracting from the flow of musicking experience?

how can a neural network be trained with an aesthetic design to generate a series of digital scores?

how do we publish such a digital score so that others may engage with this composition?

how can narratives structures from game design be used to enhance the experience of a musician with a digital score?

what are the limitations of the dataset and neural net training? Does the jazz dataset shape/ distrort the aesthetic values or communicative potential of this digital score?

performer agency in the interpretation of the score – how can this approach develop or enhance (or restrict) that further? and what are the challenges that arise from this?

About Nautilus

On an artistic level this piece is inspired by an imaginary deep-sea journey of a nautilus mollusc, as it navigates deep water trenches across the oceans. The music describes this journey with the bass flute and generative sound-design highlighting the topography of the oceans and vast openness of the depths. The name ‘nautilus’ also works on a couple of meta levels too: on the one hand the word derives from the ancient Greek for ‘sailor’; here the musician is navigating around the uncharted ocean of sound that emerges through the generative processes of this piece. On the other hand, the word originally referred to the paper nautiluses of the genus Argonauta, and hints at Captain Nemo’s submarine journey that journeyed 20,000 leagues and explored new and wonderful under-sea worlds.

On a technical level, this piece was created using Deep Learning processes, and in performance uses a neural net to make in-the-flow decisions about how the music is to be shaped. The compositional process started with an improvisation by Carla on the idea of the nautilus’ journey (the core aesthetic of the piece). This improvisation then became the source material for machine learning processes and the sound-design manipulation that is heard during the performance. A neural network was trained using TensorFlow methods and a dataset of transcribed jazz improvisations. At the start of each iteration of the piece random notes from Carla’s original improv are passed through this neural net that in turn outputs a notated improvisation based on the input note choices. This notation forms part of the digital score for live interpretation. Another element of the digital score is the generative sound design, which uses the audio recording of the original improvisation as its source material, and responds to the live sound as a stimulus. The final part is the game environment made in Unity which creates an immersive world for the audience and musician to inhabit through the piece.

Core challenge

The challenges mentioned above centre on specific questions, but the overall challenge is to create a musicking experience that binds together, rather than is created from individual elements. The least successful version of this mixed-media approach would be to construct an experience for Carla that feels like a “bit of this”, stuck together with a “bit of that”. This unifying desire is immensely difficult as the creative musicians (coder, composer, performer), must work together to build something that has a purity and singularity of “message” (see post on Marshall McLuhan’s Media Theory), draws together and enhances the communicative value of a digital score, and has a unified aesthetic. This requires us to embrace a trans-disciplinary approach where we seek to find new, common principles and factors that contribute to a wholeness of experience, and this will normally go beyond/ distort/ transform/ enhance/ transcend our own training and ways-of-thinking.

Personnel

Carla Rees – Bass Flute, performer https://www.carlarees.co.uk/

Adam Stephenson – Unity developer https://www.adam-stephenson.co.uk/

Craig Vear – Python programmer, composer