@article{di2023standardised, title={A standardised open science framework for sharing and re-analysing of neural data acquired to continuous sensory stimuli}, author={Di Liberto, Giovanni M and Nidiffer, Aaron and Crosse, Michael J and Zuk, Nathaniel and Haro, Stephanie and Cantisani, Giorgia and Winchester, Martin M and Igoe, Aoife and McCrann, Ross and Chandra, Satwik and others}, journal={Neurons, Behavior, Data analysis, and Theory (NBDT)}, year={2024}}

Neurophysiology research has demonstrated that it is possible and valuable to investigate sensory processing in scenarios involving continuous sensory streams, such as speech and music. Over the past 10 years or so, novel analytic frameworks combined with the growing participation in data sharing has led to a surge of publicly available datasets involving continuous sensory experiments. However, open science efforts in this domain of research remain scattered, lacking a cohesive set of guidelines. This paper presents an end-to-end open science framework for the storage, analysis, sharing, and re-analysis of neural data recorded during continuous sensory experiments. We propose a data structure that builds on existing custom structures (Continuous-event Neural Data), providing a precise naming convention and data types, as well as providing a workflow for storing and loading data in the general-purpose BIDS structure. The framework has been designed to interface easily with existing toolboxes, such as EelBrain, NapLib, MNE, and the mTRF-Toolbox. We present guidelines by taking both the user view (how to rapidly re-analyse existing data) and the experimenter view (how to store, analyse, and share), making the process as straightforward and accessible. Additionally, we introduce a web-based data browser that enables the effortless replication of published results and data re-analysis.

@article{di2023emergence, title={Emergence of the cortical encoding of phonetic features in the first year of life}, author={Di Liberto, Giovanni M and Attaheri, Adam and Cantisani, Giorgia and Reilly, Richard B and N{'\i} Choisdealbha, {'A}ine and Rocha, Sinead and Brusini, Perrine and Goswami, Usha}, journal={Nature Communications}, volume={14}, number={1}, pages={7789}, year={2023}, publisher={Nature Publishing Group UK London} }

Even prior to producing their first words, infants are developing a sophisticated speech processing system, with robust word recognition present by 4-6 months of age. These emergent linguistic skills, observed with behavioural investigations, are likely to rely on increasingly sophisticated neural underpinnings. The infant brain is known to robustly track the speech envelope, however previous cortical tracking studies were unable to demonstrate the presence of phonetic feature encoding. Here we utilise temporal response functions computed from electrophysiological responses to nursery rhymes to investigate the cortical encoding of phonetic features in a longitudinal cohort of infants when aged 4, 7 and 11 months, as well as adults. The analyses reveal an increasingly detailed and acoustically invariant phonetic encoding emerging over the first year of life, providing neurophysiological evidence that the pre-verbal human cortex learns phonetic categories. By contrast, we found no credible evidence for age-related increases in cortical tracking of the acoustic spectrogram.

@inproceedings{cantisani23_interspeech, author={Giorgia Cantisani and Amirhossein Chalehchaleh and Giovanni {Di Liberto} and Shihab Shamma}, title={{Investigating the cortical tracking of speech and music with sung speech}}, year=2023, booktitle={Proc. INTERSPEECH 2023}, pages={5157--5161}, doi={10.21437/Interspeech.2023-1949}}

The cortical tracking of speech and music has been primarily investigated separately. Here, we propose a novel paradigm involving sung speech to systematically compare the cortical encoding of sung speech with that of speech and music alone, offering a benchmark for using it in auditory research with ecologically-valid tasks. While this approach will ultimately lead to a variety of neural indices of speech and music processing at various levels of abstraction, the first step is to examine the envelope tracking of sung speech. EEG is recorded from subjects listening to a set of stimuli explicitly designed and built for the comparison: hummed melodies, speech monologues, and sung speech sharing the lyrics with the speech condition and the melody with the music condition. Preliminary analyses using encoding and decoding modeling show robust and consistent acoustic responses across conditions, with the only significant differences exclusively due to melody processing.

@article{cantisani2021user, title={User-guided one-shot deep model adaptation for music source separation}, author={Cantisani, Giorgia and Ozerov, Alexey and Essid, Slim and Richard, Ga{"e}l}, year={2021}}

Music source separation is the task of isolating individual instruments which are mixed in a musical piece. This task is particularly challenging, and even state-of-the-art models can hardly generalize to unseen test data. Nevertheless, prior knowledge about individual sources can be used to better adapt a generic source separation model to the observed signal. In this work, we propose to exploit a temporal segmentation provided by the user, that indicates when each instrument is active, in order to fine-tune a pre-trained deep model for source separation and adapt it to one specific mixture. This paradigm can be referred to as user-guided one-shot deep model adaptation for music source separation, as the adaptation acts on the target song instance only. Our results are promising and show that state-of-the-art source separation models have large margins of improvement especially for those instruments which are underrepresented in the training data.

@inproceedings{cantisani2021neuro, title={Neuro-steered music source separation with EEG-based auditory attention decoding and contrastive-NMF}, author={Cantisani, Giorgia and Essid, Slim and Richard, Gael}, booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages={36--40}, year={2021}, organization={IEEE}}

We propose a novel informed music source separation paradigm, which can be referred to as neuro-steered music source separation. More precisely, the source separation process is guided by the user's selective auditory attention decoded from his/her EEG response to the stimulus. This high-level prior information is used to select the desired instrument to isolate and to adapt the generic source separation model to the observed signal. To this aim, we leverage the fact that the attended instrument's neural encoding is substantially stronger than the one of the unattended sources left in the mixture. This "contrast" is extracted using an attention decoder and used to inform a source separation model based on non-negative matrix fac-torization named Contrastive-NMF. The results are promising and show that the EEG information can automatically select the desired source to enhance and improve the separation quality.

@inproceedings{cantisani2019eeg, title={EEG-Based Decoding of Auditory Attention to a Target Instrument in Polyphonic Music}, author={Cantisani, Giorgia and Essid, Slim and Richard, Ga{"e}l}, booktitle={IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)}, pages={80--84}, year={2019}, organization={IEEE}}

Auditory attention decoding aims at determining which sound source a subject is "focusing on". In this work, we address the problem of EEG-based decoding of auditory attention to a target instrument in realistic polyphonic music. To this end, exploit a stimulus reconstruction model which was proven to decode successfully the attention to speech in multi-speaker environments. To our knowledge, this model was never applied to musical stimuli for decoding attention. The task we consider ere is quite complex as the stimuli used are polyphonic, including duets and trios, and are reproduced using loudspeakers instead of headphones. We consider the decoding of three different audio representations and investigate the influence on the decoding performance of multiple variants of musical stimuli, such as the number and type of instruments in the mixture, the spatial rendering, the music genre and the melody/rhythmical pattern that is played. We obtain promising results, comparable to those obtained on speech data in previous works, and confirm that it is possible to correlate the human brain activity with musically relevant features of the attended source.

@inproceedings{Cantisani2019, author={Giorgia Cantisani and Gabriel Trégoat and Slim Essid and Gaël Richard}, title={{MAD-EEG: an EEG dataset for decoding auditory attention to a target instrument in polyphonic music}}, year=2019, booktitle={Proc. SMM19, Workshop on Speech, Music and Mind 2019}, pages={51--55}, doi={10.21437/SMM.2019-11}, url={http://dx.doi.org/10.21437/SMM.2019-11}}

We present MAD-EEG, a new, freely available dataset for studying EEG-based auditory attention decoding considering the challenging case of subjects attending to a target instrument in polyphonic music. The dataset represents the first music-related EEG dataset of its kind, enabling, in particular, studies on single-trial EEG-based attention decoding, while also opening the path for research on other EEG-based music analysis tasks. MAD-EEG has so far collected 20-channel EEG signals recorded from 8 subjects listening to solo, duo and trio music excerpts and attending to one pre-specified instrument. The proposed experimental setting differs from the ones previously considered as the stimuli are polyphonic and are played to the subject using speakers instead of headphones. The stimuli were designed considering variations in terms of number and type of instruments in the mixture, spatial rendering, music genre and melody that is played. Preliminary results obtained with a state-of-the-art stimulus reconstruction algorithm commonly used for speech stimuli show that the audio representation reconstructed from the EEG response is more correlated with that of the attended source than with the one of the unattended source, proving the dataset to be suitable for such kind of studies.

@inproceedings{grisan2015supervised, title={A supervised learning approach for the robust detection of heart beat in plethysmographic data}, author={Grisan, Enrico and Cantisani, Giorgia and Tarroni, Giacomo and Yoon, Seung Keun and Rossi, Michele}, booktitle={2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)}, pages={5825--5828}, year={2015}, organization={IEEE}}

Wearable devices equipped with photoplethysmography (PPG) sensors are gaining an increased interest in the context of biometric signal monitoring within clinical, e-health and fitness settings. When used in everyday life and during exercise, PPG traces are heavily affected by artifacts originating from motion and from a non constant positioning and contact of the PPG sensor with the skin. Many algorithms have been developed for the estimation of heart-rate from photoplethysmography signals. We remark that they were mainly conceived and tested in controlled settings and, in turn, do not provide robust performance, even during moderate exercise. Only a few of them have been designed for signals acquired at rest and during fitness. However, they provide the required resilience to motion artifacts at the cost of using computationally demanding signal processing tools. At variance with other methods from the literature, we propose a supervised learning approach, where a classifier is trained on a set of labelled data to detect the presence of heart beats at each position of a PPG signal, with only little preprocessing and postprocessing. We show that the results obtained on the TROIKA dataset using our approach are comparable with those shown in the original paper, providing a classification error of 14% in the detection of heart beat positions, that reduces to 2.86% on the heart-rate estimates after the postprocessing step.

Speech and music are highly structured auditory signals involving regularities at progressively longer time scales. Statistical processing of such structures in our brains has a central role in perception as it allows us to generate expectations about the incoming auditory input for efficient sensory processing and learning. Top-down predictions are confronted with bottom-up sensory inputs to adapt our internal model embodying language knowledge or, for music, musical culture and expertise, generating brain responses whose strength is modulated by expectations and surprises. Recent advances in neurophysiology demonstrated that prediction mechanisms can be probed during natural speech and music listening. Yet, the neural mechanisms supporting the engagement of speech and music predictions remain unclear. Measuring such neural signatures for both speech and music with a comparative framework would allow for interesting insights into non-primary auditory processing and provide evidence for a shared fundamental neural mechanism. To achieve this goal while minimising possible confounding factors in the comparison (e.g., differences in response strengths and patterns due to acoustical differences), we propose to use sung speech as a connecting link, a case study that implies simultaneous processing of linguistic and melodic components, and compare its processing to the one of speech and music only. We recorded electroencephalographic data from N=20 adult participants as they listened to continuous hummed melodies, speech, and sung speech, where the spoken and sung versions share linguistic content while the sung and music-only versions share the melody. The data was then analyzed with a framework based on linear modelling techniques, namely Temporal Response Functions (TRFs), for studying the cortical tracking of natural stimuli and measuring the contribution of phonemic and melodic expectations simultaneously. Our results show a robust and consistent cortical encoding of both phonetic and melodic expectations in sung speech as for speech and music only. Measuring such simultaneous and distinct encoding of high-level linguistic and melodic expectations represents an important result that opens the door for studying neural mechanisms specific to lyrics-tunes integration and how the two types of information are weighted for parallel processing. Interestingly, we observed a lateralized spatial reweighting of the expectations encoding strength when the melody and lyrics have to be processed in parallel, i.e., in sung speech, as opposed to speech and music only. To the best of our knowledge, this study represents the first direct comparison of the statistical processing of speech and music in ecologically valid scenarios, providing a benchmark for using sung speech in both auditory and translational research and asses linguistic and music processing in one experiment. Our results provide further evidence for speech and music being processed by the brain under the same fundamental statistical processive principles and give interesting insights into non-primary auditory processing.

The cortical tracking of speech and music has been primarily investigated separately. Here, we propose a novel paradigm involving sung speech to systematically compare the cortical encoding of sung speech with that of speech and music alone, offering a benchmark for using it in auditory research with ecologically-valid tasks. While this approach will ultimately lead to a variety of neural indices of speech and music processing at various levels of abstraction, the first step is to examine the envelope tracking of sung speech. EEG is recorded from subjects listening to a set of stimuli explicitly designed and built for the comparison: hummed melodies, speech monologues, and sung speech sharing the lyrics with the speech condition and the melody with the music condition. Preliminary analyses using encoding and decoding modeling show robust and consistent acoustic responses across conditions, with the only significant differences exclusively due to melody processing.

Music source separation is the task of isolating individual instruments which are mixed in a musical piece. This task is particularly challenging, and even state-of-the-art models can hardly generalize to unseen test data with significant timbral variation compared to training. Nevertheless, prior knowledge about individual sources can be used to better adapt a generic source separation model to the observed signal. In this work, we propose to exploit a temporal segmentation provided by the user that indicates when each instrument is active to fine-tune a pre-trained deep model for source separation and adapt it to one specific mixture. This approach can be referred to as a one-shot adaptation, as it acts on the target song instance only using the side information available only at test time. Our results are promising and show that state-of-the-art source separation models have large margins of improvement, especially for instruments underrepresented in the training data.

In this project, we address the challenge of integrating Brain-Computer Interfaces (BCI) and music technologies on the specific application of music source separation, which is the task of isolating individual sound sources that are mixed in the audio recording of a musical piece. This problem has been investigated for decades but never considering BCI as a possible way to guide and inform separation systems. Specifically, we explored how the neural activity characterized by electroencephalographic signals (EEG) reflects information about the attended instrument and how we can use it to inform a source separation system. First, we studied the problem of EEG-based auditory attention decoding of a target instrument in polyphonic music, showing that the EEG tracks musically relevant features which are highly correlated with the time-frequency representation of the attended source and only weakly correlated with the unattended one. Second, we leveraged this ``contrast'' to inform an unsupervised source separation model based on a novel non-negative matrix factorisation (NMF) variant, named contrastive-NMF (C-NMF) and automatically separate the attended source. Unsupervised NMF represents a powerful approach in such applications with no or limited amounts of training data as when neural recording is involved. Indeed, the available music-related EEG datasets are still costly and time-consuming to acquire, precluding the possibility of tackling the problem with fully supervised deep learning approaches. Thus, in the last part of the thesis, we explored alternative learning strategies to alleviate this problem. Specifically, we propose to adapt a state-of-the-art music source separation model to a specific mixture using the time activations of the sources derived from the user's neural activity. This paradigm can be referred to as one-shot adaptation, as it acts on the target song instance only. We conducted an extensive evaluation of both the proposed system on the MAD-EEG dataset which was specifically assembled for this study obtaining encouraging results, especially in difficult cases where non-informed models struggle.

Music source separation is the task of isolating individual instruments which are mixed in a musical piece. This task is particularly challenging, and even state-of-the-art models can hardly generalize to unseen test data. Nevertheless, prior knowledge about individual sources can be used to better adapt a generic source separation model to the observed signal. In this work, we propose to exploit a temporal segmentation provided by the user, that indicates when each instrument is active, in order to fine-tune a pre-trained deep model for source separation and adapt it to one specific mixture. This paradigm can be referred to as user-guided one-shot deep model adaptation for music source separation, as the adaptation acts on the target song instance only. Our results are promising and show that state-of-the-art source separation models have large margins of improvement especially for those instruments which are underrepresented in the training data.

We propose a novel informed music source separation paradigm, which can be referred to as neuro-steered music source separation. More precisely, the source separation process is guided by the user's selective auditory attention decoded from his/her EEG response to the stimulus. This high-level prior information is used to select the desired instrument to isolate and to adapt the generic source separation model to the observed signal. To this aim, we leverage the fact that the attended instrument's neural encoding is substantially stronger than the one of the unattended sources left in the mixture. This "contrast" is extracted using an attention decoder and used to inform a source separation model based on non-negative matrix fac-torization named Contrastive-NMF. The results are promising and show that the EEG information can automatically select the desired source to enhance and improve the separation quality.

Singing voice separation aims to separate the vocals from the accompaniment, which are mixed in the audio recording of a musical piece. It is a fundamental prepossessing step for many other downstream tasks as lyrics transcription, pitch tracking, singer identification, and more user-related application as remixing and karaoke. However, separating singing voice can be more complex than separating speech due to the vocals' nature which presents a complex harmonic structure and a considerable variance in timbre, register, and expressiveness. Moreover, singing voice can be highly correlated with the background music as they play together in an ensemble. Nowadays, deep-learning models trained in a supervised fashion are dominating the state-of-the-art with excellent performances. For training such models, one needs access to large datasets containing mixtures and corresponding vocals targets. However, often the vocal targets are ill-defined. For instance, there might be music segments with only the leading voice, some with multiple voices and others with only silence. Moreover, when there are multiple voices, those can sing the same or a different lyric and/or melody. In this work, we investigated how differently behaves a deep-learning source separation model for different definitions of the vocals target and if a better definition of the target can improve the separation quality.

Auditory attention decoding aims at determining which sound source a subject is "focusing on". In this work, we address the problem of EEG-based decoding of auditory attention to a target instrument in realistic polyphonic music. To this end, exploit a stimulus reconstruction model which was proven to decode successfully the attention to speech in multi-speaker environments. To our knowledge, this model was never applied to musical stimuli for decoding attention. The task we consider ere is quite complex as the stimuli used are polyphonic, including duets and trios, and are reproduced using loudspeakers instead of headphones. We consider the decoding of three different audio representations and investigate the influence on the decoding performance of multiple variants of musical stimuli, such as the number and type of instruments in the mixture, the spatial rendering, the music genre and the melody/rhythmical pattern that is played. We obtain promising results, comparable to those obtained on speech data in previous works, and confirm that it is possible to correlate the human brain activity with musically relevant features of the attended source.

We present MAD-EEG, a new, freely available dataset for studying EEG-based auditory attention decoding considering the challenging case of subjects attending to a target instrument in polyphonic music. The dataset represents the first music-related EEG dataset of its kind, enabling, in particular, studies on single-trial EEG-based attention decoding, while also opening the path for research on other EEG-based music analysis tasks. MAD-EEG has so far collected 20-channel EEG signals recorded from 8 subjects listening to solo, duo and trio music excerpts and attending to one pre-specified instrument. The proposed experimental setting differs from the ones previously considered as the stimuli are polyphonic and are played to the subject using speakers instead of headphones. The stimuli were designed considering variations in terms of number and type of instruments in the mixture, spatial rendering, music genre and melody that is played. Preliminary results obtained with a state-of-the-art stimulus reconstruction algorithm commonly used for speech stimuli show that the audio representation reconstructed from the EEG response is more correlated with that of the attended source than with the one of the unattended source, proving the dataset to be suitable for such kind of studies.

In this work, we address the problem of EEG-based decoding of auditory attention to a target instrument in realistic polyphonic music, i.e. recorded music, featuring two or three instruments played concurrently. To this end, we exploit a stimulus reconstruction model able to reconstruct an audio representation of the attended instrument from single-trial multi-channel EEG recordings. This model was proven to decode successfully the attention to speech in multi-speaker environments but, to our knowledge, was never applied to musical stimuli. For this purpose, we acquired a new dataset, named MAD-EEG, which consists of 20-channel EEG signals recorded from 8 subjects listening to solo, duo and trio music excerpts and attending to one pre-specified instrument in the mixture. It is worth noting that these stimuli were not specifically designed to elicit ERPs, and were played to the subjects using loudspeakers instead of headphones, rendering different spatial configurations. We were thus able to investigate how the decoding performance is influenced by properties of the musical stimuli, such as number and type of instruments, spatial rendering, music genre, and the melody/rhythmical pattern that is played. The obtained results are comparable to those obtained on speech data in previous works, and confirm that it is thus possible to correlate the human brain’s activity with musically relevant features of the attended source.

Our aim is to develop methods for a user-centered remastering of music performance recordings for giving the user an interactive multimedia experience. The idea is to guide audio source separation/enhancement using the user's attention as a high-level control/feedback to select which, for him/her, is the desired source to enhance. In the case of music performances, the source to enhance is represented by a particular instrument in the ensemble, thus we have a polyphonic music source separation problem.