Abstract: In this technical report, we target generating anthropomorphized personas for LLM-based characters in an online manner, including visual appearance, personality and tones, with only text descriptions. To achieve this, we first leverage the in-context learning capability of LLMs for personality generation by carefully designing a set of system prompts. We then propose two novel concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for diverse voice and appearance generation. For MoV, we utilize the text-to-speech (TTS) algorithms with a variety of pre-defined tones and select the most matching one based on the user-provided text description automatically. For MoD, we combine the recent popular text-to-image generation techniques and talking head algorithms to streamline the process of generating talking objects. We termed the whole framework as ChatAnything. With it, users could be able to animate anything with any personas that are anthropomorphic using just a few text inputs. However, we have observed that the anthropomorphic objects produced by current generative models are often undetectable by pre-trained face landmark detectors, leading to failure of the face motion generation, even if these faces possess human-like appearances because those images are nearly seen during the training (e.g., OOD samples). To address this issue, we incorporate pixel-level guidance to infuse human face landmarks during the image generation phase. To benchmark these metrics, we have built an evaluation dataset. Based on it, we verify that the detection rate of the face landmark is significantly increased from 57.0% to 92.5% thus allowing automatic face animation based on generated speech content. The code and more results can be found at https://chatanything.github.io/.

Can Large Language Models Be Good Companions? An LLM-Based Eyewear System with Conversational Common Ground

Hello Again! LLM-powered Personalized Agent for Long-term Dialogue

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Building Emotional Support Chatbots in the Era of LLMs

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responses

Ain't Misbehavin' -- Using LLMs to Generate Expressive Robot Behavior in Conversations with the Tabletop Robot Haru

A Reliable Common-Sense Reasoning Socialbot Built Using LLMs and Goal-Directed ASP

Enabling Conversational Interaction with Mobile UI using Large Language Models

User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants

LLMs as Academic Reading Companions: Extending HCI Through Synthetic Personae

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey

Mplug-Octopus: the Versatile Assistant Empowered by A Modularized End-to-End Multimodal LLM

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations

SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions

More than Chit-Chat: Developing Robots for Small-Talk Interactions

ChatAnything: Facetime Chat with LLM-Enhanced Personas.

"Shall I Be Your Chat Companion?"