Machine listening intelligence

C.E. Cella

DOI: https://doi.org/10.48550/arXiv.1706.09557

2017-06-29

Abstract:This manifesto paper will introduce machine listening intelligence, an integrated research framework for acoustic and musical signals modelling, based on signal processing, deep learning and computational musicology.

Sound,Machine Learning

What problem does this paper attempt to address?

Machine Intelligence, New Interfaces, and the Art of the Soluble

Michael J. Lyons

DOI: https://doi.org/10.48550/arXiv.1707.08011

IF: 6.4588

2017-07-25

Human-Computer Interaction

Abstract:Position: (1) Partial solutions to machine intelligence can lead to systems which may be useful creating interesting and expressive musical works. (2) An appropriate general goal for this field is augmenting human expression. (3) The study of the aesthetics of human augmentation in musical performance is in its infancy.
Guest Editorial: AI for Computational Audition—sound and Music Processing

Zijin Li,Wenwu Wang,Kejun Zhang,Mengyao Zhu

DOI: https://doi.org/10.1186/s13636-024-00353-7

2024-01-01

Abstract:Nowadays, the application of artificial intelligence (AI) algorithms and techniques is ubiquitous and transversal. Fields that take advantage of AI advances include sound and music processing. The advances in interdisciplinary research potentially yield new insights that may further advance the AI methods in this field. This special issue aims to report recent progress and spur new research lines in AI-driven sound and music processing, especially within interdisciplinary research scenarios.
Machine learning in acoustics: theory and applications

Michael J. Bianco,Peter Gerstoft,James Traer,Emma Ozanich,Marie A. Roch,Sharon Gannot,Charles-Alban Deledalle

DOI: https://doi.org/10.1121/1.5133944

2019-12-01

Abstract:Acoustic data provide scientific and engineering insights in fields ranging from biology and communications to ocean and Earth science. We survey the recent advances and transformative potential of machine learning (ML), including deep learning, in the field of acoustics. ML is a broad family of techniques, which are often based in statistics, for automatically detecting and utilizing patterns in data. Relative to conventional acoustics and signal processing, ML is data-driven. Given sufficient training data, ML can discover complex relationships between features and desired labels or actions, or between features themselves. With large volumes of training data, ML can discover models describing complex acoustic phenomena such as human speech and reverberation. ML in acoustics is rapidly developing with compelling results and significant future promise. We first introduce ML, then highlight ML developments in four acoustics research areas: source localization in speech processing, source localization in ocean acoustics, bioacoustics, and environmental sounds in everyday scenes.

Signal Processing,Machine Learning,Sound,Audio and Speech Processing,Applied Physics
Artificial Intelligence and Musicking

Adam Eric Berkowitz

DOI: https://doi.org/10.1525/mp.2024.41.5.393

2024-06-01

Music Perception: An Interdisciplinary Journal

Abstract:Artificial intelligence (AI) deployed for customer relationship management (CRM), digital rights management (DRM), content recommendation, and content generation challenge longstanding truths about listening to and making music. CRM uses music to surveil audiences, removes decision-making responsibilities from consumers, and alters relationships among listeners, artists, and music. DRM overprotects copyrighted content by subverting Fair Use Doctrine and privatizing the Public Domain thereby restricting human creativity. Generative AI, often trained on music misappropriated by developers, renders novel music that seemingly represents neither the artistry present in the training data nor the handiwork of the AI’s user. AI music, as such, appears to be produced through AI cognition, resulting in what some have called “machine folk” and contributing to a “culture in code.” A philosophical analysis of these relationships is required to fully understand how AI impacts music, artists, and audiences. Using metasynthesis and grounded theory, this study considers physical reductionism, metaphysical nihilism, existentialism, and modernity to describe the quiddity of AI’s role in the music ecosystem. Concluding thoughts call researchers and educators to act on philosophical and ethical discussions of AI and promote continued research, public education, and democratic/laymen intervention to ensure ethical outcomes in the AI music space.
ML-ASPA: A Contemplation of Machine Learning-based Acoustic Signal Processing Analysis for Sounds, & Strains Emerging Technology

Ratul Ali,Aktarul Islam,Md. Shohel Rana,Saila Nasrin,Sohel Afzal Shajol,Professor Dr. A.H.M. Saifullah Sadi

2023-12-18

Abstract:Acoustic data serves as a fundamental cornerstone in advancing scientific and engineering understanding across diverse disciplines, spanning biology, communications, and ocean and Earth science. This inquiry meticulously explores recent advancements and transformative potential within the domain of acoustics, specifically focusing on machine learning (ML) and deep learning. ML, comprising an extensive array of statistical techniques, proves indispensable for autonomously discerning and leveraging patterns within data. In contrast to traditional acoustics and signal processing, ML adopts a data-driven approach, unveiling intricate relationships between features and desired labels or actions, as well as among features themselves, given ample training data. The application of ML to expansive sets of training data facilitates the discovery of models elucidating complex acoustic phenomena such as human speech and reverberation. The dynamic evolution of ML in acoustics yields compelling results and holds substantial promise for the future. The advent of electronic stethoscopes and analogous recording and data logging devices has expanded the application of acoustic signal processing concepts to the analysis of bowel sounds. This paper critically reviews existing literature on acoustic signal processing for bowel sound analysis, outlining fundamental approaches and applicable machine learning principles. It chronicles historical progress in signal processing techniques that have facilitated the extraction of valuable information from bowel sounds, emphasizing advancements in noise reduction, segmentation, signal enhancement, feature extraction, sound localization, and machine learning techniques...

Sound,Artificial Intelligence,Machine Learning,Audio and Speech Processing
A Data-Centric Framework for Machine Listening Projects: Addressing Large-Scale Data Acquisition and Labeling through Active Learning

Javier Naranjo-Alcazar,Jordi Grau-Haro,Ruben Ribes-Serrano,Pedro Zuccarello

2024-10-08

Abstract:Machine Listening focuses on developing technologies to extract relevant information from audio signals. A critical aspect of these projects is the acquisition and labeling of contextualized data, which is inherently complex and requires specific resources and strategies. Despite the availability of some audio datasets, many are unsuitable for commercial applications. The paper emphasizes the importance of Active Learning (AL) using expert labelers over crowdsourcing, which often lacks detailed insights into dataset structures. AL is an iterative process combining human labelers and AI models to optimize the labeling budget by intelligently selecting samples for human review. This approach addresses the challenge of handling large, constantly growing datasets that exceed available computational resources and memory. The paper presents a comprehensive data-centric framework for Machine Listening projects, detailing the configuration of recording nodes, database structure, and labeling budget optimization in resource-constrained scenarios. Applied to an industrial port in Valencia, Spain, the framework successfully labeled 6540 ten-second audio samples over five months with a small team, demonstrating its effectiveness and adaptability to various resource availability situations. Acknowledgments: The participation of Javier Naranjo-Alcazar, Jordi Grau-Haro and Pedro Zuccarello in this research was funded by the Valencian Institute for Business Competitiveness (IVACE) and the FEDER funds by means of project Soroll-IA2 (IMDEEA/2023/91). The research carried out for this publication has been partially funded by the project STARRING-NEURO (PID2022-137048OA-C44) funded by the Ministry of Science, Innovation and Universities of Spain and the European Union.

Sound,Machine Learning,Audio and Speech Processing
Machine listening in a neonatal intensive care unit

Modan Tailleur,Vincent Lostanlen,Jean-Philippe Rivière,Pierre Aumond

2024-10-07

Abstract:Oxygenators, alarm devices, and footsteps are some of the most common sound sources in a hospital. Detecting them has scientific value for environmental psychology but comes with challenges of its own: namely, privacy preservation and limited labeled data. In this paper, we address these two challenges via a combination of edge computing and cloud computing. For privacy preservation, we have designed an acoustic sensor which computes third-octave spectrograms on the fly instead of recording audio waveforms. For sample-efficient machine learning, we have repurposed a pretrained audio neural network (PANN) via spectral transcoding and label space adaptation. A small-scale study in a neonatological intensive care unit (NICU) confirms that the time series of detected events align with another modality of measurement: i.e., electronic badges for parents and healthcare professionals. Hence, this paper demonstrates the feasibility of polyphonic machine listening in a hospital ward while guaranteeing privacy by design.

Sound,Artificial Intelligence,Machine Learning,Audio and Speech Processing
Harmonizing minds and machines: survey on transformative power of machine learning in music

Jing Liang

DOI: https://doi.org/10.3389/fnbot.2023.1267561

2023-11-10

Abstract:This survey explores the symbiotic relationship between Machine Learning (ML) and music, focusing on the transformative role of Artificial Intelligence (AI) in the musical sphere. Beginning with a historical contextualization of the intertwined trajectories of music and technology, the paper discusses the progressive use of ML in music analysis and creation. Emphasis is placed on present applications and future potential. A detailed examination of music information retrieval, automatic music transcription, music recommendation, and algorithmic composition presents state-of-the-art algorithms and their respective functionalities. The paper underscores recent advancements, including ML-assisted music production and emotion-driven music generation. The survey concludes with a prospective contemplation of future directions of ML within music, highlighting the ongoing growth, novel applications, and anticipation of deeper integration of ML across musical domains. This comprehensive study asserts the profound potential of ML to revolutionize the musical landscape and encourages further exploration and advancement in this emerging interdisciplinary field.
Living Together: Mind and Machine Intelligence

Neil D. Lawrence

DOI: https://doi.org/10.48550/arXiv.1705.07996

2017-05-23

Abstract:In this paper we consider the nature of the machine intelligences we have created in the context of our human intelligence. We suggest that the fundamental difference between human and machine intelligence comes down to \emph{embodiment factors}. We define embodiment factors as the ratio between an entity's ability to communicate information vs compute information. We speculate on the role of embodiment factors in driving our own intelligence and consciousness. We briefly review dual process models of cognition and cast machine intelligence within that framework, characterising it as a dominant System Zero, which can drive behaviour through interfacing with us subconsciously. Driven by concerns about the consequence of such a system we suggest prophylactic courses of action that could be considered. Our main conclusion is that it is \emph{not} sentient intelligence we should fear but \emph{non-sentient} intelligence.

Artificial Intelligence
VoiceListener

Lei Wang,Meng Chen,Li Lü,Zhongjie Ba,Feng Lin,Kui Ren

DOI: https://doi.org/10.1145/3580789

2023-01-01

Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies

Abstract:Recently, voice leakage gradually raises more significant concerns of users, due to its underlying sensitive and private information when providing intelligent services. Existing studies demonstrate the feasibility of applying learning-based solutions on built-in sensor measurements to recover voices. However, due to the privacy concerns, large-scale voices-sensor measurements samples for model training are not publicly available, leading to significant efforts in data collection for such an attack. In this paper, we propose a training-free and universal eavesdropping attack on built-in speakers, VoiceListener, which releases the data collection efforts and is able to adapt to various voices, platforms, and domains. In particular, VoiceListener develops an aliasing-corrected super resolution mechanism, including an aliasing-based pitch estimation and an aliasing-corrected voice recovering, to convert the undersampled narrow-band sensor measurements to wide-band voices. Extensive experiments demonstrate that our proposed VoiceListener could accurately recover the voices from undersampled sensor measurements and is robust to different voices, platforms and domains, realizing the universal eavesdropping attack.
A machine-intelligent world

DOI: https://doi.org/10.1126/science.adj4843

IF: 56.9

2023-07-14

Science

Abstract:Finding a place for AI in science and society

multidisciplinary sciences
Artificial Intelligence in Music and Performance: A Subjective Art-Research Inquiry

Baptiste Caramiaux,Marco Donnarumma

DOI: https://doi.org/10.48550/arXiv.2007.15843

2020-07-31

Abstract:This article presents a five-year collaboration situated at the intersection of Art practice and Scientific research in Human-Computer Interaction (HCI). At the core of our collaborative work is a hybrid, Art and Science methodology that combines computational learning technology -- Machine Learning (ML) and Artificial Intelligence (AI) -- with interactive music performance and choreography. This article first exposes our thoughts on combining art, science, movement and sound research. We then describe two of our artistic works \textit{Corpus Nil} and \textit{Humane Methods} -- created five years apart from each other -- that crystallize our collaborative research process. We present the scientific and artistic motivations, framed through our research interests and cultural environment of the time. We conclude by reflecting on the methodology we developed during the collaboration and on the conceptual shift of computational learning technologies, from ML to AI, and its impact on Music performance.

Human-Computer Interaction
Applying machine learning to primate bioacoustics: Review and perspectives

Jules Cauzinille,Benoit Favre,Ricard Marxer,Arnaud Rey

DOI: https://doi.org/10.1002/ajp.23666

2024-08-09

Abstract:This paper provides a comprehensive review of the use of computational bioacoustics as well as signal and speech processing techniques in the analysis of primate vocal communication. We explore the potential implications of machine learning and deep learning methods, from the use of simple supervised algorithms to more recent self-supervised models, for processing and analyzing large data sets obtained within the emergence of passive acoustic monitoring approaches. In addition, we discuss the importance of automated primate vocalization analysis in tackling essential questions on animal communication and highlighting the role of comparative linguistics in bioacoustic research. We also examine the challenges associated with data collection and annotation and provide insights into potential solutions. Overall, this review paper runs through a set of common or innovative perspectives and applications of machine learning for primate vocal communication analysis and outlines opportunities for future research in this rapidly developing field.
Machine Intelligence in Africa: a survey

Allahsera Auguste Tapo,Ali Traore,Sidy Danioko,Hamidou Tembine

2024-02-04

Abstract:In the last 5 years, the availability of large audio datasets in African countries has opened unlimited opportunities to build machine intelligence (MI) technologies that are closer to the people and speak, learn, understand, and do businesses in local languages, including for those who cannot read and write. Unfortunately, these audio datasets are not fully exploited by current MI tools, leaving several Africans out of MI business opportunities. Additionally, many state-of-the-art MI models are not culture-aware, and the ethics of their adoption indexes are questionable. The lack thereof is a major drawback in many applications in Africa. This paper summarizes recent developments in machine intelligence in Africa from a multi-layer multiscale and culture-aware ethics perspective, showcasing MI use cases in 54 African countries through 400 articles on MI research, industry, government actions, as well as uses in art, music, the informal economy, and small businesses in Africa. The survey also opens discussions on the reliability of MI rankings and indexes in the African continent as well as algorithmic definitions of unclear terms used in MI.

Computers and Society,Artificial Intelligence
The Machine Learning Algorithm as Creative Musical Tool

Rebecca Fiebrink,Baptiste Caramiaux

DOI: https://doi.org/10.48550/arXiv.1611.00379

2016-11-02

Abstract:Machine learning is the capacity of a computational system to learn structures from datasets in order to make predictions on newly seen data. Such an approach offers a significant advantage in music scenarios in which musicians can teach the system to learn an idiosyncratic style, or can break the rules to explore the system's capacity in unexpected ways. In this chapter we draw on music, machine learning, and human-computer interaction to elucidate an understanding of machine learning algorithms as creative tools for music and the sonic arts. We motivate a new understanding of learning algorithms as human-computer interfaces. We show that, like other interfaces, learning algorithms can be characterised by the ways their affordances intersect with goals of human users. We also argue that the nature of interaction between users and algorithms impacts the usability and usefulness of those algorithms in profound ways. This human-centred view of machine learning motivates our concluding discussion of what it means to employ machine learning as a creative tool.

Human-Computer Interaction,Machine Learning
Toward smart acoustic spaces: embedded machine learning for sound event detection and classification in the built environment

Tre DiPassio,Michael C. Heilemann,Benjamin R. Thompson,Jenna Rutowski,Mark Bocko

DOI: https://doi.org/10.1121/10.0027510

2024-03-01

The Journal of the Acoustical Society of America

Abstract:Edge computing integrated into the built environment will enable the development of smart acoustic spaces in which the flexural vibrations of the walls and screens that define a space may be employed to generate and monitor sound and serve as unobtrusive and convenient audio interfaces. Integration of data processing and neural decision making on compact, low- power embedded hardware-software systems will make it possible to mediate real-time, personalized acoustic interactions of a user with their environment without the need to connect to the cloud, thus providing privacy and security. In this presentation we show how embedded machine learning (ML) models combined with vibroacoustic control and monitoring of elastic panels may be employed to perform various tasks such as speech recognition, sound source localization, or the detection of acoustic signatures of specific events such as a fall or other health emergencies. The selection of vibroacoustic features, the associated signal processing requirements, and the computational resources utilized by the ML models for various acoustic tasks will be discussed. We also highlight how the distributed modal response of extended flat panels can simplify the sensing and signal processing requirements in such systems.

acoustics,audiology & speech-language pathology
AI (r)evolution -- where are we heading? Thoughts about the future of music and sound technologies in the era of deep learning

Giovanni Bindi,Nils Demerlé,Rodrigo Diaz,David Genova,Aliénor Golvet,Ben Hayes,Jiawen Huang,Lele Liu,Vincent Martos,Sarah Nabi,Teresa Pelinski,Lenny Renault,Saurjya Sarkar,Pedro Sarmento,Cyrus Vahidi,Lewis Wolstanholme,Yixiao Zhang,Axel Roebel,Nick Bryan-Kinns,Jean-Louis Giavitto,Mathieu Barthet

2023-09-20

Abstract:Artificial Intelligence (AI) technologies such as deep learning are evolving very quickly bringing many changes to our everyday lives. To explore the future impact and potential of AI in the field of music and sound technologies a doctoral day was held between Queen Mary University of London (QMUL, UK) and Sciences et Technologies de la Musique et du Son (STMS, France). Prompt questions about current trends in AI and music were generated by academics from QMUL and STMS. Students from the two institutions then debated these questions. This report presents a summary of the student debates on the topics of: Data, Impact, and the Environment; Responsible Innovation and Creative Practice; Creativity and Bias; and From Tools to the Singularity. The students represent the future generation of AI and music researchers. The academics represent the incumbent establishment. The student debates reported here capture visions, dreams, concerns, uncertainties, and contentious issues for the future of AI and music as the establishment is rightfully challenged by the next generation.

Computers and Society,Artificial Intelligence
Research in music and artificial intelligence

Curtis Roads

DOI: https://doi.org/10.1145/4468.4469

IF: 16.6

1985-06-01

ACM Computing Surveys

Abstract:Although the boundaries of artificial intelligence (AI) remain elusive, computers can now perform musical tasks that were formerly associated exclusively with naturally intelligent musicians. After a historical note, this paper sermonizes on the need for AI techniques in four areas of musical research: composition, performance, music theory, and digital sound processing. The next part surveys recent work involving AI and music. The discussion concentrates on applications in the four areas of research just mentioned. The final part examines how AI techniques of planning and learning could be used to expand the knowledge base and enrich the behavior of musically intelligent systems.

computer science, theory & methods
Generative Machine Listener

Guanxin Jiang,Lars Villemoes,Arijit Biswas

2023-08-18

Abstract:We show how a neural network can be trained on individual intrusive listening test scores to predict a distribution of scores for each pair of reference and coded input stereo or binaural signals. We nickname this method the Generative Machine Listener (GML), as it is capable of generating an arbitrary amount of simulated listening test data. Compared to a baseline system using regression over mean scores, we observe lower outlier ratios (OR) for the mean score predictions, and obtain easy access to the prediction of confidence intervals (CI). The introduction of data augmentation techniques from the image domain results in a significant increase in CI prediction accuracy as well as Pearson and Spearman rank correlation of mean scores.

Audio and Speech Processing,Sound
Our Practice Of Using Machine Learning To Recognize Species By Voice

Siddhardha Balemarthy,Atul Sajjanhar,James Xi Zheng

DOI: https://doi.org/10.48550/arXiv.1810.09078

2018-10-22

Abstract:As the technology is advancing, audio recognition in machine learning is improved as well. Research in audio recognition has traditionally focused on speech. Living creatures (especially the small ones) are part of the whole ecosystem, monitoring as well as maintaining them are important tasks. Species such as animals and birds are tending to change their activities as well as their habitats due to the adverse effects on the environment or due to other natural or man-made calamities. For those in far deserted areas, we will not have any idea about their existence until we can continuously monitor them. Continuous monitoring will take a lot of hard work and labor. If there is no continuous monitoring, then there might be instances where endangered species may encounter dangerous situations. The best way to monitor those species are through audio recognition. Classifying sound can be a difficult task even for humans. Powerful audio signals and their processing techniques make it possible to detect audio of various species. There might be many ways wherein audio recognition can be done. We can train machines either by pre-recorded audio files or by recording them live and detecting them. The audio of species can be detected by removing all the background noise and echoes. Smallest sound is considered as a syllable. Extracting various syllables is the process we are focusing on which is known as audio recognition in terms of Machine Learning (ML).

Sound,Machine Learning,Audio and Speech Processing

Machine listening intelligence

Machine Intelligence, New Interfaces, and the Art of the Soluble

Guest Editorial: AI for Computational Audition—sound and Music Processing

Machine learning in acoustics: theory and applications

Artificial Intelligence and Musicking

ML-ASPA: A Contemplation of Machine Learning-based Acoustic Signal Processing Analysis for Sounds, & Strains Emerging Technology

A Data-Centric Framework for Machine Listening Projects: Addressing Large-Scale Data Acquisition and Labeling through Active Learning

Machine listening in a neonatal intensive care unit

Harmonizing minds and machines: survey on transformative power of machine learning in music

Living Together: Mind and Machine Intelligence

VoiceListener

A machine-intelligent world

Artificial Intelligence in Music and Performance: A Subjective Art-Research Inquiry

Applying machine learning to primate bioacoustics: Review and perspectives

Machine Intelligence in Africa: a survey

The Machine Learning Algorithm as Creative Musical Tool

Toward smart acoustic spaces: embedded machine learning for sound event detection and classification in the built environment

AI (r)evolution -- where are we heading? Thoughts about the future of music and sound technologies in the era of deep learning

Research in music and artificial intelligence

Generative Machine Listener

Our Practice Of Using Machine Learning To Recognize Species By Voice