Semantics in Robotics: Environmental Data Can't Yield Conventions of Human Behaviour

Jamie Milton Freestone

2024-10-25

Abstract:The word semantics, in robotics and AI, has no canonical definition. It usually serves to denote additional data provided to autonomous agents to aid HRI. Most researchers seem, implicitly, to understand that such data cannot simply be extracted from environmental data. I try to make explicit why this is so and argue that so-called semantics are best understood as data comprised of conventions of human behaviour. This includes labels, most obviously, but also places, ontologies, and affordances. Object affordances are especially problematic because they require not only semantics that are not in the environmental data (conventions of object use) but also an understanding of physics and object combinations that would, if achieved, constitute artificial superintelligence.

Robotics,Artificial Intelligence

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is: **Can robots and artificial intelligence systems autonomously extract semantic information (i.e., semantics) of human behavior from environmental data?** The author points out that although many researchers hope that future robots can directly extract these high - level abstract information (such as labels, categories, functions, places, etc.) from the environment, in fact, this is impossible because these semantic information does not exist directly in the environmental data.

### Specific problems include:

1. **Definition and classification of semantic information**:
- The author distinguishes two definitions of semantics:
1. **Semantics as Conventional Semantics**: It refers to high - level, abstract data that depend on human behavior or usage conventions. For example, labeling an object is based on human language - use conventions.
2. **Semantics as Contentful Semantics**: It refers to certain types of data that have inherent or intrinsic properties, and their meanings can be directly extracted from the data itself without additional context or historical information.

2. **The relationship between environmental data and semantic information**:
- Environmental data refers to data that can be directly obtained from the environment, such as visual, auditory and other information captured by sensors. However, much of the so - called "semantic information" is not included in these raw data, but depends on human behavior conventions and historical experiences. For example, a decorative cup may look no different from other cups, but its way of use (not for drinking) is based on specific human behavior conventions, not the physical characteristics of the cup itself.

3. **The impossibility of autonomously extracting semantic information**:
- The author emphasizes that even humans cannot directly extract this semantic information solely from environmental data. This is because semantic information is essentially about the history and conventions of human behavior, not just data features. Therefore, robots cannot understand these high - level semantic information through pure environmental perception.

4. **The impact on future research**:
- Many researchers hope that future robots can autonomously learn and understand these semantic information from the environment, but the author believes that this expectation is unrealistic. Instead, researchers should focus on how to effectively associate human behavior conventions with environmental data, rather than trying to let robots autonomously "discover" this information.

### Conclusion:
The main contribution of this paper is to clearly point out that semantic information cannot be directly extracted from the environment, but depends on human behavior conventions and historical experiences. Therefore, robots and AI systems need to rely on pre - provided semantic data or learn human behavior patterns to obtain this information, rather than attempting to autonomously extract from the original environmental data.