embodiedMusickingDatasetMay2021-UK04

<p>The performance of music involves the physical expression of musical material in a complex and multimodal process. Furthermore, musical performance involves a sense of 'flow', or immersion in the creative act, that can be better understood through a careful and holistic examination of data captured from this complex physical activity. Flow is especially relevant in improvisatory performance contexts where musicians must make real-time decisions about content and its expression. The project detailed in this paper involves the design and creation of a low-cost protocol for collecting simultaneous streams of data from improvising human musicians that are performing from a common score. The protocol records and synchronises audio recording with body-, facial- and physiological response tracking with a ground-truth annotation through the reported flow of a performer. This association yields a robust dataset that serves to capture the complex and multi-model process of making music 'in the flow'. This dataset can be a useful tool for a range of applications, such as creative AI practices, music generation in game engines, music information retrieval and humanisation of static systems—e.g. MIDI file playback and sound processing parameters.</p>