Virtual Agents in Live Coding: A Short Review

Anna Xambó

DOI: https://doi.org/10.48550/arXiv.2106.14835

IF: 6.4588

2021-06-28

Human-Computer Interaction

Abstract:AI and live coding has been little explored. This article contributes with a short review of different perspectives of using virtual agents in the practice of live coding looking at past and present as well as pointing to future directions.

What problem does this paper attempt to address?

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante,Qiuyuan Huang,Naoki Wake,Ran Gong,Jae Sung Park,Bidipta Sarkar,Rohan Taori,Yusuke Noda,Demetri Terzopoulos,Yejin Choi,Katsushi Ikeuchi,Hoi Vo,Li Fei-Fei,Jianfeng Gao

2024-01-26

Abstract:Multi-modal AI systems will likely become a ubiquitous presence in our everyday lives. A promising approach to making these systems more interactive is to embody them as agents within physical and virtual environments. At present, systems leverage existing foundation models as the basic building blocks for the creation of embodied agents. Embedding agents within such environments facilitates the ability of models to process and interpret visual and contextual data, which is critical for the creation of more sophisticated and context-aware AI systems. For example, a system that can perceive user actions, human behavior, environmental objects, audio expressions, and the collective sentiment of a scene can be used to inform and direct agent responses within the given environment. To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions. In particular, we explore systems that aim to improve agents based on next-embodied action prediction by incorporating external knowledge, multi-sensory inputs, and human feedback. We argue that by developing agentic AI systems in grounded environments, one can also mitigate the hallucinations of large foundation models and their tendency to generate environmentally incorrect outputs. The emerging field of Agent AI subsumes the broader embodied and agentic aspects of multimodal interactions. Beyond agents acting and interacting in the physical world, we envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.

Artificial Intelligence,Human-Computer Interaction,Machine Learning
Live Coding: A Review of the Literature

Ana Selvaraj,Eda Zhang,Leo Porter,Adalbert Gerald Soosai Raj

DOI: https://doi.org/10.1145/3430665.3456382

2021-06-26

Abstract:One of the goals of computing education research is to document the potential strengths and weaknesses of contemporary teaching methods in computing. Live coding has recently gained attention as one of the best practices for teaching programming. To offer a more comprehensive understanding of the existing body of research about live coding, we reviewed papers in computing education research that investigated the value of live coding in an educational setting. We categorized each paper based on (1) how it defines live coding, (2) whether its version of live coding could be considered active learning, (3) the type of study conducted, (4) types of data collected and the data analysis methods used, (5) evidence provided for the effectiveness of live coding, (6) reported benefits and drawbacks of live coding, and (7) reported theoretical frameworks used to explain the basis, effects or goals of live coding. We found that although live coding has been recommended as one of the best practices for teaching programming, there is a lack of empirical evidence to support claims about the effectiveness of live coding on student learning. Finally, we discuss the implications of our findings and suggest future research directions that could develop a more holistic understanding of this pedagogical technique.
Virtual-Agent-Based Language Learning: A Scoping Review of Journal Publications from 2012 to 2022

Xinyan Gu,Taxue Yu,Jun Huang,Feng Wang,Xiaoli Zheng,Mengxiang Sun,Zihao Ye,Qi Li

DOI: https://doi.org/10.3390/su151813479

IF: 3.9

2023-09-09

Sustainability

Abstract:Researchers have pointed out the importance of virtual agents in technology-supported language learning; however, how to effectively combine the two remains a challenge for educators and educational technologists. To this end, this study reviewed publications in the field of virtual-agent-based language learning research from 2012 to 2022 in the Web of Science SSCI Core Collection database and explored the dimensions of publication trends, country and regional distribution, participants, research methodology, research platforms, role of virtual agents, language proficiency, research hot topics, theoretical foundations, and hot issues and trends in the field of virtual-agent-based language learning research. Cluster and co-occurrence analysis using VOSviewer software was used to analyze the links among country and region distribution, keywords, and terms. It was found that (1) the top four regions in terms of the number of citations for authors were, in descending order, Iran, Japan, South Korea, and Brazil; (2) the learner characteristics that scholars were most concerned about were learning effectiveness, memory performance, social presence, learning experience, and motivation; and (3) the results of co-occurrence analyses classified virtual-agent-based language learning research into eight clusters, namely, anthropomorphic virtual agents, the effects produced by virtual agents, the social interaction of virtual agents, animated virtual agents and language achievement, the gestures of virtual agents, the effects of virtual agents on learner characteristics, computer-assisted learning, and the design of virtual agents. The lack of the systematic application of virtual agents in language learning prevented previous studies from revealing the language learning process in virtual-agent-based learning environments. Therefore, this study made appropriate recommendations for future investigations on how virtual agents can improve language learning for researchers, teachers, and decision makers.

environmental sciences,environmental studies,green & sustainable science & technology
A concise review of intelligent game agent

Hui Li,Xinyi Pang,Bixia Sun,Kexin Liu

DOI: https://doi.org/10.1016/j.entcom.2024.100894

IF: 2.072

2024-11-01

Entertainment Computing

Abstract:Intelligent game agents are crafted using AI technologies to mimic player behavior and make decisions autonomously. Over the past decades, the scope of intelligent agents has broadened from chess to encompass content generation, player modeling, and result prediction, reflecting the field's evolving and multifaceted nature. In this paper, we conduct a systematic review of recent literature on intelligent methods and applications of game agents, along with general game agent frameworks. Our findings suggest that creating general intelligent agents remains a significant challenge, yet it is worthwhile to explore methods that better integrate the strengths of different techniques to build more robust and adaptable intelligent game agents.

computer science, cybernetics, interdisciplinary applications, software engineering
A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions

Avyay Casheekar,Archit Lahiri,Kanishk Rath,Kaushik Sanjay Prabhakar,Kathiravan Srinivasan

DOI: https://doi.org/10.1016/j.cosrev.2024.100632

IF: 8.757

2024-04-11

Computer Science Review

Abstract:This review paper offers an in-depth analysis of AI-powered virtual conversational agents, specifically focusing on OpenAI's ChatGPT. The main contributions of this paper are threefold: (i) an exhaustive review of prior literature on chatbots, (ii) a background of chatbots including existing chatbots/conversational agents like ChatGPT, and (iii) a UI/UX design analysis of prominent chatbots. Another contribution of this review is the comprehensive exploration of ChatGPT's applications across a multitude of sectors, including education, business, public health, and more. This review highlights the transformative potential of ChatGPT, despite the challenges it faces such as hallucination, biases in training data, jailbreaks, and anonymous data collection. The review paper then presents a comprehensive survey of prior literature reviews on chatbots, identifying gaps in the prior work and highlighting the need for further research in areas such as chatbot evaluation, user experience, and ethical considerations. The paper also provides a detailed analysis of the UI/UX design of prominent chatbots, including their conversational flow, visual design, and user engagement. The paper also identifies key future research directions, including mitigating language bias, enhancing ethical decision-making capabilities, improving user interaction and personalization, and developing robust governance frameworks. By solving these issues, we can ensure that AI chatbots like ChatGPT are used responsibly and effectively across a broad variety of applications. This review will be a valuable resource for researchers and practitioners in understanding the current state and future potential of AI chatbots like ChatGPT.

computer science, information systems, theory & methods, software engineering
Creative Agents: Simulating the Systems Model of Creativity with Generative Agents

Naomi Imasato,Kazuki Miyazawa,Takayuki Nagai,Takato Horii

2024-11-26

Abstract:With the growing popularity of generative AI for images, video, and music, we witnessed models rapidly improve in quality and performance. However, not much attention is paid towards enabling AI's ability to "be creative". In this study, we implemented and simulated the systems model of creativity (proposed by Csikszentmihalyi) using virtual agents utilizing large language models (LLMs) and text prompts. For comparison, the simulations were conducted with the "virtual artists" being: 1)isolated and 2)placed in a multi-agent system. Both scenarios were compared by analyzing the variations and overall "creativity" in the generated artifacts (measured via a user study and LLM). Our results suggest that the generative agents may perform better in the framework of the systems model of creativity.

Multiagent Systems,Artificial Intelligence
A Systematic Review of Voice-based Intelligent Virtual Agents in EFL Education

Eirene Katsarou,Fridolin Wild,Areti-Maria Sougari,Paraskevi Chatzipanagiotou

DOI: https://doi.org/10.3991/ijet.v18i10.37723

2023-05-23

International Journal of Emerging Technologies in Learning (iJET)

Abstract:Since its debut in the field of education nearly three decades ago, Artificial Intelligence (AI) has been considered as a powerful tool to facilitate new paradigms for instructional design and innovative educational practice in the form of intelligent tutoring systems, chatbots, teaching robots and adaptive learning systems among others. Recent technological advances in the adjacent areas of natural language processing, machine learning and computer graphics focusing primarily on design features that can improve their human-like qualities of naturalness and believability as human interlocutors have also amplified new application opportunities for Intelligent Virtual Agents (IVAs) or Animated Pedagogical Agents (APAs) within the area of Intelligent Computer-Assisted Language Learning (ICALL). Although AI-powered IVAs hold the potential to enhance the learning process in nearly any knowledge domain and personalize automation in teaching by embodying different roles in the learning environment, strikingly few studies have empirically attempted to assess IVAs impact on L2 learners’ academic achievement when learning English as a Foreign (EFL) so far. This study addresses this issue via a systematic review of relevant interventionist IVA studies that were conducted in EFL settings and published within the 2015-2020 timeframe examining IVAs key affordances, major barriers in their adoption for language learning purposes, and the CALL research trends currently prevalent on the topic. Pedagogical implications for the effective implementation of IVA technology in L2 contexts are discussed and future research avenues in the area are highlighted.
Contribution of Artificial Intelligence and Virtual Worlds towards development of Super Intelligent AI Agents

Simran Koul,,

DOI: https://doi.org/10.35940/ijeat.e9923.069520

2020-06-30

International Journal of Engineering and Advanced Technology

Abstract:Artificial Intelligence (AI) has always been a goal as well as a challenge in the scientific field with a whole new world being accessible to people by accomplishing various achievements in the past decades. However, in the last many years, tremendous popularity has been received by the interactive computer games involving multi-user virtual environment which has been the prime attraction for millions of users around the world. This paper focuses on AI agents along with exploring the progress towards super intelligent AI and the roles played by artificial intelligence and virtual worlds in its development. It follows a unique approach to give an unbiased view on the various problems and the relation between them. There are namely two benefits from this research: firstly, evolution in science and technology requires a profound understanding on the advanced AI agents in the virtual worlds, and secondly, the virtual worlds have always been a magnificent platform for research on the challenging problems faced in the domain of Artificial Intelligence.
A Review of Agent-Based Programming for Multi-Agent Systems

Rafael C. Cardoso,Angelo Ferrando

DOI: https://doi.org/10.3390/computers10020016

2021-01-27

Computers

Abstract:Intelligent and autonomous agents is a subarea of symbolic artificial intelligence where these agents decide, either reactively or proactively, upon a course of action by reasoning about the information that is available about the world (including the environment, the agent itself, and other agents). It encompasses a multitude of techniques, such as negotiation protocols, agent simulation, multi-agent argumentation, multi-agent planning, and many others. In this paper, we focus on agent programming and we provide a systematic review of the literature in agent-based programming for multi-agent systems. In particular, we discuss both veteran (still maintained) and novel agent programming languages, their extensions, work on comparing some of these languages, and applications found in the literature that make use of agent programming.

English Else
AI-Based Virtual Assistant Using Python: A Systematic Review

Patil Kavita Manojkumar,Aditi Patil,Sakshi Shinde,Shaktiprasad Patra,Saloni Patil

DOI: https://doi.org/10.22214/ijraset.2023.49519

2023-03-31

International Journal for Research in Applied Science and Engineering Technology

Abstract:Abstract: A software agent that will carry out tasks or provide services in response to a user's privately supported instructions or inquiries is known as an intelligent virtual assistant (IVA) or intelligent personal assistant (IPA). A virtual assistant capable of being accessed via web chat is sometimes called a "chatbot." Online chat systems can occasionally only be used for amusement. Some virtual assistants are equipped to comprehend spoken language and answer with synthetic voices. Users can use voice commands to manage other basic chores like email, to-do lists, and calendars in addition to asking their assistants questions, controlling home automation devices, and controlling media playing. One of the best applications of artificial intelligence is the virtual personal assistant (VPA), which offers a new way for people to delegate tasks to machines. To create a Virtual Personal Assistant (VPA) and use it in various software applications, certain approaches and principles are used. To enable users to communicate with virtual assistants, speech recognition systems—also known as Automatic Speech Recognition, or ASR—play a crucial role.
Live Coding Machine Learning: Finding the Moments of Intervention in Autonomous Processes

Iván Paz,Shelly Knotts

DOI: https://doi.org/10.1162/comj_a_00663

IF: 0.688

2024-02-06

Computer Music Journal

Abstract:Abstract Machine learning (ML) deals with algorithms able to learn from data, with the primary aim of finding optimum solutions to perform tasks autonomously. In recent years there has been development in integrating ML algorithms with live coding practices, raising questions about what to optimize or automate, the agency of the algorithms, and in which parts of the ML processes one might intervene midperformance. Live coding performance practices typically involve conversational interaction with algorithmic processes in real time. In analyzing systems integrating live coding and ML, we consider the musical and performative implications of the “moment of intervention” in the ML model and workflow, and the channels for real-time intervention. We propose a framework for analysis, through which we reflect on the domain-specific algorithms and practices being developed that combine these two practices.

computer science, interdisciplinary applications
CodeAgent: Autonomous Communicative Agents for Code Review

Xunzhu Tang,Kisub Kim,Yewei Song,Cedric Lothritz,Bei Li,Saad Ezzini,Haoye Tian,Jacques Klein,Tegawende F. Bissyande

2024-09-25

Abstract:Code review, which aims at ensuring the overall quality and reliability of software, is a cornerstone of software development. Unfortunately, while crucial, Code review is a labor-intensive process that the research community is looking to automate. Existing automated methods rely on single input-output generative models and thus generally struggle to emulate the collaborative nature of code review. This work introduces \tool{}, a novel multi-agent Large Language Model (LLM) system for code review automation. CodeAgent incorporates a supervisory agent, QA-Checker, to ensure that all the agents' contributions address the initial review question. We evaluated CodeAgent on critical code review tasks: (1) detect inconsistencies between code changes and commit messages, (2) identify vulnerability introductions, (3) validate code style adherence, and (4) suggest code revision. The results demonstrate CodeAgent's effectiveness, contributing to a new state-of-the-art in code review automation. Our data and code are publicly available (\url{<a class="link-external link-https" href="https://github.com/Code4Agent/codeagent" rel="external noopener nofollow">this https URL</a>}).

Software Engineering
Conversational Agents: Goals, Technologies, Vision and Challenges

Merav Allouch,Amos Azaria,Rina Azoulay

DOI: https://doi.org/10.3390/s21248448

IF: 3.9

2021-12-17

Sensors

Abstract:In recent years, conversational agents (CAs) have become ubiquitous and are a presence in our daily routines. It seems that the technology has finally ripened to advance the use of CAs in various domains, including commercial, healthcare, educational, political, industrial, and personal domains. In this study, the main areas in which CAs are successful are described along with the main technologies that enable the creation of CAs. Capable of conducting ongoing communication with humans, CAs are encountered in natural-language processing, deep learning, and technologies that integrate emotional aspects. The technologies used for the evaluation of CAs and publicly available datasets are outlined. In addition, several areas for future research are identified to address moral and security issues, given the current state of CA-related technological developments. The uniqueness of our review is that an overview of the concepts and building blocks of CAs is provided, and CAs are categorized according to their abilities and main application domains. In addition, the primary tools and datasets that may be useful for the development and evaluation of CAs of different categories are described. Finally, some thoughts and directions for future research are provided, and domains that may benefit from conversational agents are introduced.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
AI-Assisted Programming Tasks Using Code Embeddings and Transformers

Sotiris Kotsiantis,Vassilios Verykios,Manolis Tzagarakis

DOI: https://doi.org/10.3390/electronics13040767

IF: 2.9

2024-02-16

Electronics

Abstract:This review article provides an in-depth analysis of the growing field of AI-assisted programming tasks, specifically focusing on the use of code embeddings and transformers. With the increasing complexity and scale of software development, traditional programming methods are becoming more time-consuming and error-prone. As a result, researchers have turned to the application of artificial intelligence to assist with various programming tasks, including code completion, bug detection, and code summarization. The utilization of artificial intelligence for programming tasks has garnered significant attention in recent times, with numerous approaches adopting code embeddings or transformer technologies as their foundation. While these technologies are popular in this field today, a rigorous discussion, analysis, and comparison of their abilities to cover AI-assisted programming tasks is still lacking. This article discusses the role of code embeddings and transformers in enhancing the performance of AI-assisted programming tasks, highlighting their capabilities, limitations, and future potential in an attempt to outline a future roadmap for these specific technologies.

engineering, electrical & electronic,computer science, information systems,physics, applied
Visualization of AI Systems in Virtual Reality: A Comprehensive Review

Medet Inkarbekov,Rosemary Monahan,Barak A. Pearlmutter

DOI: https://doi.org/10.14569/IJACSA.2023.0140805

2023-06-27

Abstract:This study provides a comprehensive review of the utilization of Virtual Reality (VR) for visualizing Artificial Intelligence (AI) systems, drawing on 18 selected studies. The results illuminate a complex interplay of tools, methods, and approaches, notably the prominence of VR engines like Unreal Engine and Unity. However, despite these tools, a universal solution for effective AI visualization remains elusive, reflecting the unique strengths and limitations of each technique. We observed the application of VR for AI visualization across multiple domains, despite challenges such as high data complexity and cognitive load. Moreover, it briefly discusses the emerging ethical considerations pertaining to the broad integration of these technologies. Despite these challenges, the field shows significant potential, emphasizing the need for dedicated research efforts to unlock the full potential of these immersive technologies. This review, therefore, outlines a roadmap for future research, encouraging innovation in visualization techniques, addressing identified challenges, and considering the ethical implications of VR and AI convergence.

Human-Computer Interaction
Asynchronous Tool Usage for Real-Time Agents

Antonio A. Ginart,Naveen Kodali,Jason Lee,Caiming Xiong,Silvio Savarese,John Emmons

2024-10-29

Abstract:While frontier large language models (LLMs) are capable tool-using agents, current AI systems still operate in a strict turn-based fashion, oblivious to passage of time. This synchronous design forces user queries and tool-use to occur sequentially, preventing the systems from multitasking and reducing interactivity. To address this limitation, we introduce asynchronous AI agents capable of parallel processing and real-time tool-use. Our key contribution is an event-driven finite-state machine architecture for agent execution and prompting, integrated with automatic speech recognition and text-to-speech. Drawing inspiration from the concepts originally developed for real-time operating systems, this work presents both a conceptual framework and practical tools for creating AI agents capable of fluid, multitasking interactions.

Artificial Intelligence
Let's Give a Voice to Conversational Agents in Virtual Reality

Michele Yin,Gabriel Roccabruna,Abhinav Azad,Giuseppe Riccardi

2023-08-05

Abstract:The dialogue experience with conversational agents can be greatly enhanced with multimodal and immersive interactions in virtual reality. In this work, we present an open-source architecture with the goal of simplifying the development of conversational agents operating in virtual environments. The architecture offers the possibility of plugging in conversational agents of different domains and adding custom or cloud-based Speech-To-Text and Text-To-Speech models to make the interaction voice-based. Using this architecture, we present two conversational prototypes operating in the digital health domain developed in Unity for both non-immersive displays and VR headsets.

Artificial Intelligence
Directing Avatars in Live Performances -- an Autonomy Simulacrum of Virtual Entities

Georges Gagneré

2024-10-31

Abstract:We propose to review the main stages in the computer history of virtual actors, with a view to the exploration of virtual reality and discussion on different approaches to human simulation. The notion of autonomy emerges as a key issue for the virtual entities. We then explore one way of building elements of autonomy and conclude with an example of avatar stage direction leading to a simulacrum of autonomy in a live performance.

Human-Computer Interaction,Graphics
Review of Artificial Intelligence Applications and Modelling AI Framework in Education System

Patel Karika Digesh

DOI: https://doi.org/10.32628/cseit2390542

2023-10-01

Abstract:The potential of using artificial intelligence in education to enhance learning, assist teachers and fuel more effective individualized learning is exciting, but also a bit challenging. To even have an intelligent conversation about AI in education, one must first push past imaginary science-fiction scenarios of computers and robots teaching our children, replacing teachers and reducing the human element from what is a fundamentally human activity. AI can automate grading so that the tutor can have more time to teach. AI chatbot can communicate with students as a teaching assistant. This research paper focuses on modelling of AI ingredients in framework of education. AI in future can work as a personal virtual tutor for students, which will be easily accessible at any time and any place.
A Survey on Applications of Digital Human Avatars toward Virtual Co-presence

Matthew Korban,Xin Li

DOI: https://doi.org/10.48550/arXiv.2201.04168

2022-01-12

Abstract:This paper investigates different approaches to build and use digital human avatars toward interactive Virtual Co-presence (VCP) environments. We evaluate the evolution of technologies for creating VCP environments and how the advancement in Artificial Intelligence (AI) and Computer Graphics affect the quality of VCP environments. We categorize different methods in the literature based on their applications and methodology and compare various groups and strategies based on their applications, contributions, and limitations. We also have a brief discussion about the approaches that other forms of human representation, rather than digital human avatars, have been utilized in VCP environments. Our goal is to fill the gap in the research domain where there is a lack of literature review investigating different approaches for creating avatar-based VCP environments. We hope this study will be useful for future research involving human representation in VCP or Virtual Reality (VR) environments. To the best of our knowledge, it is the first survey research that investigates avatar-based VCP environments. Specifically, the categorization methodology suggested in this paper for avatar-based methods is new.

Human-Computer Interaction,Artificial Intelligence

Virtual Agents in Live Coding: A Short Review

Agent AI: Surveying the Horizons of Multimodal Interaction

Live Coding: A Review of the Literature

Virtual-Agent-Based Language Learning: A Scoping Review of Journal Publications from 2012 to 2022

A concise review of intelligent game agent

A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions

Creative Agents: Simulating the Systems Model of Creativity with Generative Agents

A Systematic Review of Voice-based Intelligent Virtual Agents in EFL Education

Contribution of Artificial Intelligence and Virtual Worlds towards development of Super Intelligent AI Agents

A Review of Agent-Based Programming for Multi-Agent Systems

AI-Based Virtual Assistant Using Python: A Systematic Review

Live Coding Machine Learning: Finding the Moments of Intervention in Autonomous Processes

CodeAgent: Autonomous Communicative Agents for Code Review

Conversational Agents: Goals, Technologies, Vision and Challenges

AI-Assisted Programming Tasks Using Code Embeddings and Transformers

Visualization of AI Systems in Virtual Reality: A Comprehensive Review

Asynchronous Tool Usage for Real-Time Agents

Let's Give a Voice to Conversational Agents in Virtual Reality

Directing Avatars in Live Performances -- an Autonomy Simulacrum of Virtual Entities

Review of Artificial Intelligence Applications and Modelling AI Framework in Education System

A Survey on Applications of Digital Human Avatars toward Virtual Co-presence