From Speech to Data: Unraveling Google's Use of Voice Data for User Profiling

Xinhang Ma,Sirui Chen

2024-03-04

Abstract:Smart home voice assistants enable users to conveniently interact with IoT devices and perform Internet searches; however, they also collect the voice input that can carry sensitive personal information about users. Previous papers investigated how information inferred from the contents of users' voice commands are shared or leaked for tracking and advertising purposes. In this paper, we systematically evaluate how voice itself is used for user profiling in the Google ecosystem. To do so, we simulate various user personas by engaging with specific categories of websites. We then use \textit{neutral voice commands}, which we define as voice commands that neither reveal personal interests nor require Google smart speakers to use the search APIs, to interact with these speakers. We also explore the effects of the non-neutral voice commands for user profiling. Notably, we employ voices that typically would not match the predefined personas. We then iteratively improve our experiments based on observations of profile changes to better simulate real-world user interactions with smart speakers. We find that Google uses these voice recordings for user profiling, and in some cases, up to 5 out of the 8 categories reported by Google for customizing advertisements are altered following the collection of the voice commands.

Human-Computer Interaction

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to explore how voice data in the Google ecosystem is used for user profiling. Specifically, by simulating different user roles to interact with Google smart speakers and using neutral voice commands (these commands do not contain personal interest information), researchers evaluate whether and how Google uses this voice data to build detailed user profiles. In addition, the study also explores the impact of non - neutral voice commands on user profiles to understand how, in actual use, user interactions with smart speakers affect their ad customization and personalized services.

### Main research questions

1. **Is voice data used for user profiling in the Google ecosystem?**
- Researchers use neutral voice commands of the opposite gender to observe changes in user profiles and verify whether Google uses voice data for user profiling.

2. **How much information about the user can Google infer from a neutral voice command?**
- By creating new user accounts and using only one specific neutral voice command, researchers attempt to determine how much user information Google can infer from a single voice command.

3. **For user profiles that already contain voice data, will different voice features change the existing profiles?**
- Researchers introduce voice commands with different gender features into established user profiles to evaluate the impact of these changes on user profiles.

4. **How are non - neutral voice commands used for user profiling?**
- Researchers use non - neutral voice commands containing personal preference information to evaluate the impact of these commands on user profiles, especially in different types of user profiles (user profiles based only on web - browsing behavior and user profiles containing voice data).

### Experimental methods

1. **Establish baseline user profiles**:
- Use automated browsing bots to simulate the web - browsing behaviors of different users and create multiple user profiles with different interests and behavioral characteristics.

2. **Interact with smart speakers**:
- Use neutral voice commands to interact with Google smart speakers and observe changes in user profiles. These commands are designed not to contain personal interest information but may contain voice features (such as gender, age, etc.).

3. **Iterative experiments**:
- Based on the preliminary experimental results, continuously adjust the experimental methods to more accurately simulate the interaction between real users and smart speakers.

### Experimental results

1. **Google does use voice data for user profiling**:
- Even when using neutral voice commands, Google's user profiles change, indicating that voice features themselves have a significant impact on user profiles.

2. **The impact of a single neutral voice command is limited**:
- The cumulative effect of multiple voice commands can cause significant changes in user profiles, indicating that Google's algorithms require a certain amount of voice data to update user profiles.

3. **Different voice features have a relatively small impact on established user profiles**:
- For user profiles that already contain a large amount of voice data, new voice commands with different gender features have a limited impact on them.

4. **Non - neutral voice commands have a significant impact on user profiles**:
- Non - neutral voice commands containing personal preference information can quickly change user profiles, especially in categories such as income and occupation.

### Conclusions

This study reveals that Google smart speakers use not only the content of voice commands but also voice features (such as gender, age, etc.) for user profiling. This finding poses new challenges to user privacy protection, especially in the context of the increasing popularity of smart devices. Researchers recommend that users should pay more attention to the privacy of their voice data and call on relevant institutions to strengthen the supervision of data processing in smart devices.