The performance of music involves the physical expression of musical material in a complex and multimodal process. Furthermore, musical performance involves a sense of 'flow', or immersion in the creative act, that can be better understood through a careful and holistic examination of data captured from this complex physical activity. Flow is especially relevant in improvisatory performance contexts where musicians must make real-time decisions about content and its expression. The project detailed in this paper involves the design and creation of a low-cost protocol for collecting simultaneous streams of data from improvising human musicians that are performing from a common score. The protocol records and synchronises audio recording with body-, facial- and physiological response tracking with a ground-truth annotation through the reported flow of a performer. This association yields a robust dataset that serves to capture the complex and multi-model process of making music 'in the flow'. This dataset can be a useful tool for a range of applications, such as creative AI practices, music generation in game engines, music information retrieval and humanisation of static systems—e.g. MIDI file playback and sound processing parameters.
Human Data Interaction: Legibility, Agency, Negotiability
Engineering and Physical Sciences Research CouncilFind out more...