Music Puzzle: an Application for Hearing Training on Android Smart Devices

Zheng Liu,Hua Wang,Kjetil Falkenberg Hansen

2012-01-01

Abstract:The Music Puzzle is part of an ongoing project called Ljudparken/The Soundpark – Using modern smartphones to create interactive listening experiences for hearing impaired. The aim is to create inte ...

What problem does this paper attempt to address?

Mind Band: A Crossmedia AI Music Composing Platform

Zhaolin Qiu,Yufan Ren,Canchen Li,Hongfu Liu,Yifan Huang,Yiheng Yang,Songruoyao Wu,Hanjia Zheng,Juntao Ji,Jianjia Yu,Kejun Zhang

DOI: https://doi.org/10.1145/3343031.3350610

2019-01-01

Abstract:Various media information in life can have an important impact on our understanding of music. In this paper, we present a demo, Mind Band, which is a Cross-Media artificial intelligent composing platform using our life elements such as emoji, image and humming. In practice, we base our system on the valence-arousal model. We use emotion analysis of life elements to map them to music pieces, which are generated by a Variational Autoencoder - Generative Adversarial Networks model. We provide users with immersive experience by uploading emoji/image/humming and retrieving emotionally related music pieces back. With this platform, everyone can be a composer.
Soundoku: A sound puzzle game

Hufschmitt Aline,Adrien Dombrowsky,Adam J. Sporka

DOI: https://doi.org/10.1109/cog52621.2021.9618984

2021-08-17

Abstract:We present Soundoku, a sound-based puzzle game designed for sighted people. The goal of this game is to determine if it is possible to transform a popular puzzle game like Blendoku into a sound-based puzzle, without making it a musical game, and to obtain a fun, interesting and eventually instructive game. Initial evaluations of our game by players between the ages of 17 and 26 indicate that the game is considered fun if not addictive. These results pave the way for the creation of a game using procedural generation of sound puzzles.
MusicCollage: A Music Composition Tool for Children Based on Synesthesia and a Genetic Algorithm

Ge Yan,Cheng Yao,Chao Zhang,Jiadi Wang,Yuqi Hu,Fangtian Ying

DOI: https://doi.org/10.1007/978-3-031-05431-0_15

2022-01-01

Abstract:Creating music can be challenging for children aged 4-6. Making the creation procedure more interesting has been proved to stimulate and build children's interests in music. Based on the design concept of synesthesia from vision to hearing, we propose an application bywhich children aged 4-6 can create collages by matching different images of scenes and animals. The information contained in the created collages is then used as input parameters for generating music so as to create melodies corresponding to the collages. To evaluate our approach, we conducted a quantitative and qualitative user study with children (n = 6) by using our initial prototype. We concluded that the proposed application helps cultivate children's willingness to create melodies, and helps children better perceive and understand music by combining scenes and animal images in the collages.
Bubble Beats

Yajing Hu,Yijun Zhao,Yaping Shao,Can Zhu,Jiayan Chen,Tianlei Shi,Zili Zhou,Fangtian Ying,Guanyun Wang,Cheng Yao

DOI: https://doi.org/10.1145/3459990.3465207

2021-01-01

Abstract:We designed and built Bubble Beats, a breathing exercise game based on music rhythm, which maps breathing into musical gameplay mechanics using sensors-based technology. Bubble Beats aims to explore the effectiveness of combining music rhythm and breathing exercises. In the game, children blow bubbles to the music rhythm with visual guidance. Initial user tests show that Bubble Beats can effectively motivate children and improve their adherence to breathing exercises. We will continue to iterate and upgrade the system to attract and match a more diverse population, for example, patients recovering from pneumonia and other respiratory diseases.
DrumGenius: bridging learning-gap with interactive musical instruments.

Ye Tao,Guanyun Wang,Yujie Hong,Qi Wang,Cheng Yao,Fangtian Ying

DOI: https://doi.org/10.1145/2556420.2556474

2014-01-01

Abstract:ABSTRACTWhen independent learning is conducted in musical instruments learning, the learning process can be easily interrupted due to expectation gap between children and parents. Traditional acoustic musical instruments are challenging for young children because they require high levels of coordination and precision. We found that children learnt rhythm and cadence through physical activity in the early music learning by instinct. DrumGenius, as a new controller for interactive musical games, was designed to combine exploration and play by providing real-time dynamic feedback depending on children's movements in the learning process. The study aims to bridge the gap by a dynamic learning pattern in initial learning stage with respect to the existing static way.
Mixing Levels -- A Rock Music Spirit Level App

Tim Ziemer

DOI: https://doi.org/10.48550/arXiv.2302.08249

2023-02-16

Abstract:To date, sonification apps are rare. Music apps on the other hand are widely used. Smartphone users like to play with music. In this manuscript, we present Mixing Levels, a spirit level sonification based on music mixing. Tilting the smartphone adjusts the volumes of 5 musical instruments in a rock music loop. Only when perfectly leveled, all instruments in the mix are well-audible. The app is supposed to be useful and fun. Since the app appears like a music mixing console, people have fun to interact with Mixing Levels, so that learning the sonification is a playful experience.

Multimedia,Human-Computer Interaction
OrchestraBox: RFID Music Box for Musical Education at Schools

Andrey Borisov,Jürgen Sieck,Elisabeth Thielen,Jurgen Sieck

DOI: https://doi.org/10.1109/idaacs53288.2021.9660878

2021-09-22

Abstract:There are many different educational applications for schools that use technologies like augmented reality (AR) and virtual reality (VR) in order to make the learning process more intuitive and to motivate the students. However, most of these applications are designed to be used by only one person, making it difficult for teachers to use such applications during their lessons. This paper presents a multiuser AR application for musical education at schools that allows the teachers to involve the whole class in the learning process. The application compounds of a music box with RFID antennas and 3D printed figurines of musicians with an RFID chip. The music box can recognize if some figurine is currently placed on the music box and starts playing the corresponding music part.
Addressing the Issue of Musical Education through Gamified Tuning Analysis

Adelle Kang,Jeffrey Fan,Aminjin Battulga,Hyunwook Choi

DOI: https://doi.org/10.47611/jsrhs.v13i1.6235

2024-02-29

Journal of Student Research

Abstract:Technologies are rapidly evolving, and they are impacting human lives profoundly. And this also applies to the music industry, especially musical instrument training. There is the traditional option of receiving private expensive lessons including how to accurately tune your own instrument from a music teacher. However, it is not always viable for a teacher to be present, let alone the high cost. Furthermore, musical instrument training could overall be enhanced through a more technical approach without being constrained by the cost and human resources. With this goal in mind, we Noteblasters decided to investigate whether technology can address and resolve this common problem facing any musically inclined instrument players. Our Noteblast app is designed to detect pitches and give real-time feedback to the musicians on whether the correct note is being played or not. Through our multiple versions of prototypes, we were able to test our Noteblast app’s functionalities such as the tuner, providing real-time accurate feedback on the user’s playing, creating an easy and convenient tuning experience. The data shows aforementioned musical tuning problems can be resolved significantly more with a technology integrated device, which is our app.
ArchiTone: A LEGO-Inspired Gamified System for Visualized Music Education

Jiaxing Yu,Tieyao Zhang,Songruoyao Wu,Xinda Wu,Tingxiao Wu,Yanjun Chen,Kejun Zhang

2024-10-20

Abstract:Participation in music activities has many benefits, but often requires music theory knowledge and aural skills, which can be challenging for beginners. To help them engage more easily, it's crucial to adopt teaching strategies that lower these barriers. Informed by formative investigation and inspired by LEGO, we introduce ArchiTone, a gamified system that employs constructivism by visualizing music theory concepts as musical blocks and buildings for music education. This system includes two modes: Learning Mode, which involves recognizing and learning common musical blocks through familiar musical works; Creation Mode, which allows learners to freely create and combine musical blocks to produce new musical works. User studies demonstrate that our gamified system is not only more engaging than traditional music education methods but also more effective in helping learners understand abstract music theory and apply it to music praxis. Additionally, learners demonstrate superior performance on music theory tasks after using ArchiTone.

Human-Computer Interaction
Generating Music Medleys via Playing Music Puzzle Games

Yu-Siang Huang,Szu-Yu Chou,Yi-Hsuan Yang

DOI: https://doi.org/10.1609/aaai.v32i1.11843

2018-04-26

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Generating music medleys is about finding an optimal permutation of a given set of music clips. Toward this goal, we propose a self-supervised learning task, called the music puzzle game, to train neural network models to learn the sequential patterns in music. In essence, such a game requires machines to correctly sort a few multisecond music fragments. In the training stage, we learn the model by sampling multiple non-overlapping fragment pairs from the same songs and seeking to predict whether a given pair is consecutive and is in the correct chronological order. For testing, we design a number of puzzle games with different difficulty levels, the most difficult one being music medley, which requiring sorting fragments from different songs. On the basis of state-of-the-art Siamese convolutional network, we propose an improved architecture that learns to embed frame-level similarity scores computed from the input fragment pairs to a common space, where fragment pairs in the correct order can be more easily identified. Our result shows that the resulting model, dubbed as the similarity embedding network (SEN), performs better than competing models across different games, including music jigsaw puzzle, music sequencing, and music medley. Example results can be found at our project website, https://remyhuang.github.io/DJnet.
Smartphone Apps in the Context of Tinnitus: Systematic Review

Muntazir Mehdi,Constanze Riha,Patrick Neff,Albi Dode,Rüdiger Pryss,Winfried Schlee,Manfred Reichert,Franz J Hauck

DOI: https://doi.org/10.3390/s20061725

2020-03-19

Abstract:Smartphones containing sophisticated high-end hardware and offering high computational capabilities at extremely manageable costs have become mainstream and an integral part of users' lives. Widespread adoption of smartphone devices has encouraged the development of many smartphone applications, resulting in a well-established ecosystem, which is easily discoverable and accessible via respective marketplaces of differing mobile platforms. These smartphone applications are no longer exclusively limited to entertainment purposes but are increasingly established in the scientific and medical field. In the context of tinnitus, the ringing in the ear, these smartphone apps range from relief, management, self-help, all the way to interfacing external sensors to better understand the phenomenon. In this paper, we aim to bring forth the smartphone applications in and around tinnitus. Based on the PRISMA guidelines, we systematically analyze and investigate the current state of smartphone apps, that are directly applied in the context of tinnitus. In particular, we explore Google Scholar, CiteSeerX, Microsoft Academics, Semantic Scholar for the identification of scientific contributions. Additionally, we search and explore Google's Play and Apple's App Stores to identify relevant smartphone apps and their respective properties. This review work gives (1) an up-to-date overview of existing apps, and (2) lists and discusses scientific literature pertaining to the smartphone apps used within the context of tinnitus.
Smartphone and Mobile Health Apps for Tinnitus: Systematic Identification, Analysis, and Assessment

Muntazir Mehdi,Michael Stach,Constanze Riha,Patrick Neff,Albi Dode,Rüdiger Pryss,Winfried Schlee,Manfred Reichert,Franz J Hauck

DOI: https://doi.org/10.2196/21767

2020-08-18

Abstract:Background: Modern smartphones contain sophisticated high-end hardware features, offering high computational capabilities at extremely manageable costs and have undoubtedly become an integral part in users' daily life. Additionally, smartphones offer a well-established ecosystem that is easily discoverable and accessible via the marketplaces of differing mobile platforms, thus encouraging the development of many smartphone apps. Such apps are not exclusively used for entertainment purposes but are also commonplace in health care and medical use. A variety of those health and medical apps exist within the context of tinnitus, a phantom sound perception in the absence of any physical external source. Objective: In this paper, we shed light on existing smartphone apps addressing tinnitus by providing an up-to-date overview. Methods: Based on PRISMA guidelines, we systematically searched and identified existing smartphone apps on the most prominent app markets, namely Google Play Store and Apple App Store. In addition, we applied the Mobile App Rating Scale (MARS) to evaluate and assess the apps in terms of their general quality and in-depth user experience. Results: Our systematic search and screening of smartphone apps yielded a total of 34 apps (34 Android apps, 26 iOS apps). The mean MARS scores (out of 5) ranged between 2.65-4.60. The Tinnitus Peace smartphone app had the lowest score (mean 2.65, SD 0.20), and Sanvello-Stress and Anxiety Help had the highest MARS score (mean 4.60, SD 0.10). The interrater agreement was substantial (Fleiss κ=0.74), the internal consistency was excellent (Cronbach α=.95), and the interrater reliability was found to be both high and excellent-Guttman λ6=0.94 and intraclass correlation, ICC(2,k) 0.94 (95% CI 0.91-0.97), respectively. Conclusions: This work demonstrated that there exists a plethora of smartphone apps for tinnitus. All of the apps received MARS scores higher than 2, suggesting that they all have some technical functional value. However, nearly all identified apps were lacking in terms of scientific evidence, suggesting the need for stringent clinical validation of smartphone apps in future. To the best of our knowledge, this work is the first to systematically identify and evaluate smartphone apps within the context of tinnitus.
Electroacoustic Evaluation of Smartphone-Based Hearing Aid Applications

Willy Nguyen,Miseung Koo,Jun Ho Lee,Seung-Ha Oh,Moo Kyun Park

DOI: https://doi.org/10.21053/ceo.2021.01004

Abstract:Objectives: This study evaluated the electroacoustic characteristics of smartphone-based hearing aid applications (apps). Methods: We investigated hearing aid apps based on processing delay measurements, hearing instrument testing, simulated real ear measurements, and a head-and-torso simulator. Results: Many apps exceeded the recommended level for processing delay. Hearing instrument testing showed the highest amplification characteristics and the best sound quality when a hearing aid was used, followed by the high-end apps and then the low-end apps. The simulated real ear measurements results showed that the high-end apps had a better ability to match the amplification targets than the low-end apps, but there was no consistent pattern among apps when controlling the output. Only a few apps could improve the signal-to-noise ratio in the head-and-torso simulator. Conclusion: Most of the apps showed relatively poor electroacoustic performance in comparison with hearing aids. Generalizing access to hearing care through hearing aid apps induces a wide diversity of hearing performance with no fixed standard for reliability. However, we expect their overall quality to improve over the next few years.
Gamifying Music Education: An Appealing and Engaging Method of Practice for Teenagers

Vincent Park,Alan Cheng

DOI: https://doi.org/10.47611/jsrhs.v12i1.3913

2023-02-28

Journal of Student Research

Abstract:Music interventions have been proven to have positive effects on educational processes and development of children. But music education has been in jeopardy for the past 3 years due to the Covid-19 pandemic and overall has been inaccessible due to exorbitant costs, social stigma, and lack of engagement. Under these conditions, this paper seeks to answer the question of whether gamification makes music practice more effective and appealing. To accomplish this, a mobile game is designed to utilize a fantasy narrative with interactions triggered by the sound from an instrument This game, named Octavion, features a captured princess who escapes with movement dictated by notes played on a musical instrument. Music notes appear on screen at certain times, which emulates sight reading sheet music. Qualitative and quantitative user experiments are conducted to understand the impact of the game on users’ instrument playing aptitude and engagement. The results from playtests show that playing the game does improve the testers’ sight-reading speed. In a larger context, this experiment indicates that a game of this structure can be appealing and effective to help teenagers read and play music.
World of sounds (Seslerin Dünyası): A mobile auditory training game for children with cochlear implants

Merve Meral Çetinkaya,Özlem Konukseven,Ali Efe İrali

DOI: https://doi.org/10.1016/j.ijporl.2024.111908

IF: 1.626

2024-03-08

International Journal of Pediatric Otorhinolaryngology

Abstract:Objectives The aim of this study is to develop a mobile auditory training application based on gaming for children aged 3–5 years using cochlear implants and to evaluate its usability. Methods 4 games were developed in the scope of the application World of Sounds; the crucible sound for auditory awareness, mole hunting for auditory discrimination, find the sound for auditory recognition, and choo-choo for auditory comprehension. The prototype was applied to 20 children with normal hearing and 20 children with cochlear implants, all of whom were aged 3–5. The participants were asked to fill out the Game Evaluation Form for Children. Moreover, 40 parents were included in the study, and the Evaluation Form for the Application was applied. Results According to the form, at least 80% of children using cochlear implants, and all children in the healthy group, responded well to the usability factors. All factors were obtained as highly useable by parents of the children using cochlear implants. The results indicated that in the healthy group, the usefulness and motivation factors were above moderate, and the other factors were highly useable. In the mole-hunting game, there was no significant difference between the groups in the easy level of the first sub-section (p > 0.05). There was a significant difference between the groups in terms of the other sub-sections of the mole-hunting game and all sub-sections of the crucible sound, find the sound, and the choo-choo games (p 0.05); a correlation was found in the other sub-sections of crucible sound and all sub-sections of the mole hunting, find the sound, and Choo-Choo games (p < 0.05). Conclusion It is thought that the application World of Sounds can serve as an accessible option to support traditional auditory rehabilitation for children with cochlear implants.

pediatrics,otorhinolaryngology
An Interactive Smart Music Toy Design for Children

Shijian Luo,Yun Wang,Na Xiong,Ping Shan,Yexing Zhou

DOI: https://doi.org/10.1007/978-3-319-91125-0_31

2018-01-01

Abstract:With the improvement of social material conditions, children’s education has drawn increasing attention. Children’s toys enter the field of vision as an important medium for children’s education, while the advent of a variety of scientific and technological achievements gave birth to the intelligent trend of children’s toys. At the same time, traditional children’s music education has many inconveniences, application of scientific and technological achievements is also needed in this education process. It provides a perfect opportunity for children’s musical education to be the perfect application scenarios for children’s smart toys.With related market research and targeted user research, a new design idea of interactive smart music toy for children is put forward. The musical instrument type music toy is selected for product design, and designed from three parts: design research, product scheme design and program test. Using the hand wearable product solution, an interactive way was designed that conforms to the natural behavior and psychological cognition of the children, to let children feel the charm of music and enhance children’s visual perception and coordination of hand movements ability.This article uses the methods of desktop research, user interviews and tangible user interfaces design principles to sort out the design characteristics of children’s music toys in the market, and draw our design opportunities after analyzing its advantages and disadvantages. Then the target user’s needs and product design were taken. Under the theoretical guidance of the tangible user interface, the product’s design ideas are improved and the system design of the product is carried out. Finally, the feasibility of the design scheme is verified through user test. The analysis and experimental results show that the design of the interactive children’s smart music toy scheme is reasonable and feasible, which can stimulate children’s interest in use and enhance the user experience to a certain degree when they carry out music education.
Design and Implementation of Music Teaching Platform in College Based on Android Mobile Technology

Juan Tong

DOI: https://doi.org/10.3991/ijet.v11i05.5686

2016-05-23

International Journal of Emerging Technologies in Learning (iJET)

Abstract:multi-media teaching is the trend for improving teaching approach, yet what is currently prevalent in university education is its low interaction and inefficient knowledge absorption level. According to the characteristics of multimedia teaching, this paper discussed application of informational teaching resources in university music teaching and explored ways to improve the quality of music teaching. This paper took chorus conduction in university music course as experiment subject, through analysis of teaching resources based on Android system and its design principle, analysis of teaching machine based on android system covered various aspects which included teaching process, resources interface, interactive approaches, and effect assessment. The paper used methods like interview, statistics and comparing the efficiency difference before and after its application. Through the research, we found that optimized android teaching system can improve students’ scores and overall quality level, which provides ideas and data backup applications of modern teaching resources in university music course.
SoundScape: A Human-AI Co-Creation System Making Your Memories Heard

Chongjun Zhong,Jiaxing Yu,Yingping Cao,Songruoyao Wu,Wenqi Wu,Kejun Zhang

2024-10-11

Abstract:Sound plays a significant role in human memory, yet it is often overlooked by mainstream life-recording methods. Most current UGC (User-Generated Content) creation tools emphasize visual content while lacking user-friendly sound design features. This paper introduces SoundScape, a human-AI co-creation system that allows users to easily create sound memories on mobile devices through innovative interaction. By integrating sound effects and music with visual scenes, SoundScape encourages users to enrich their creations with immersive sound elements, enhancing the atmosphere of their works. To support public creation, SoundScape incorporates a conversational agent and AI music generation technology. User studies indicate that our approach is effective for sound memory creation, with SoundScape outperforming existing tools in terms of user experience and the perceived quality of produced works.

Human-Computer Interaction
A Study on the Interactive "HOPSCOTCH" Game for the Children Using Computer Music Techniques

Shing-Kwei Tzeng,Chih-Fang Huang

DOI: https://doi.org/10.48550/arXiv.1006.0866

2010-06-04

Sound

Abstract:"Hopscotch" is a world-wide game for children to play since the times in the ancient Roman Empire and China. Here we present a study mainly focused on the research and discussion of the application on the children's well-know edutainment via the physical interactive design to provide the sensing of the times for the conventional hopscotch, which is a new type of experiment for the technology aided edutainment. The innovated hopscotch music game involves the sound samples of various animals and the characters of cartoon, and the algorithmic composition via the development of the music technology based interactive game, to gradually make the children perceive the world of digits, sound, and music. It can guide the growing children's personality and character from disorder into clarity. Furthermore, the traditional teaching materials can be improved via the implementation of the electrical sensing devices, electrical I/O module, and the computer music program Max/MSP, to integrate the interactive computer music with the interactive and immersive soundscapes composition, and the teaching tool with educational gaming is completely accomplished eventually.
An Automatic Method to Develop Music With Music Segment and Long Short Term Memory for Tinnitus Music Therapy

Jiemei Chen,Fan Pan,Ping Zhong,Tiantian He,Leiyu Qi,Jingzhe Lu,Peiyu He,Yun Zheng

DOI: https://doi.org/10.1109/access.2020.3013339

IF: 3.9

2020-01-01

IEEE Access

Abstract:Tinnitus is a perception of sound when no external sound is present. It has seriously affected patients' life. Music is an option to relieve tinnitus in clinic, as it can bring enjoyment to listeners. However, existing music used in tinnitus therapies has limited duration, it is usually repetitively played during the long-term treatment and may not be helpful for relaxation. Moreover, individualized preferences of patients are ignored in most cases. Both of them may hinder tinnitus relief. Although existing methods can synthesize specific music that has unlimited duration and is not repetitively played, the synthesized music has defects with pitch mutations and long pitch durations. Moreover, characteristics of these synthesized music have not been confirmed by tinnitus patients. Therefore, this study presents an automatic method to develop the specific music based on music segments from existing music and long short term memory (LSTM). Numerical results indicate that specific music developed in this study not only retains characteristics of original music, but also overcome the defects above. Besides, a total of 30 tinnitus patients and 10 tinnitus-free volunteers participated the auditory experiment. Auditory results are consistent with numerical results and also suggest that tinnitus patients can perceive feelings that are conducive to tinnitus relief after listening preferred music. Therefore, the developed music presents a possible complement to tinnitus treatment in clinic.

computer science, information systems,telecommunications,engineering, electrical & electronic

Music Puzzle: an Application for Hearing Training on Android Smart Devices

Mind Band: A Crossmedia AI Music Composing Platform

Soundoku: A sound puzzle game

MusicCollage: A Music Composition Tool for Children Based on Synesthesia and a Genetic Algorithm

Bubble Beats

DrumGenius: bridging learning-gap with interactive musical instruments.

Mixing Levels -- A Rock Music Spirit Level App

OrchestraBox: RFID Music Box for Musical Education at Schools

Addressing the Issue of Musical Education through Gamified Tuning Analysis

ArchiTone: A LEGO-Inspired Gamified System for Visualized Music Education

Generating Music Medleys via Playing Music Puzzle Games

Smartphone Apps in the Context of Tinnitus: Systematic Review

Smartphone and Mobile Health Apps for Tinnitus: Systematic Identification, Analysis, and Assessment

Electroacoustic Evaluation of Smartphone-Based Hearing Aid Applications

Gamifying Music Education: An Appealing and Engaging Method of Practice for Teenagers

World of sounds (Seslerin Dünyası): A mobile auditory training game for children with cochlear implants

An Interactive Smart Music Toy Design for Children

Design and Implementation of Music Teaching Platform in College Based on Android Mobile Technology

SoundScape: A Human-AI Co-Creation System Making Your Memories Heard

A Study on the Interactive "HOPSCOTCH" Game for the Children Using Computer Music Techniques

An Automatic Method to Develop Music With Music Segment and Long Short Term Memory for Tinnitus Music Therapy