Towards vision-based robotic skins: a data-driven, multi-camera tactile sensor

Camill Trueeb,Carmelo Sferrazza,Raffaello D'Andrea

DOI: https://doi.org/10.1109/RoboSoft48309.2020.9116060

2020-06-23

Abstract:This paper describes the design of a multi-camera optical tactile sensor that provides information about the contact force distribution applied to its soft surface. This information is contained in the motion of spherical particles spread within the surface, which deforms when subject to force. The small embedded cameras capture images of the different particle patterns that are then mapped to the three-dimensional contact force distribution through a machine learning architecture. The design proposed in this paper exhibits a larger contact surface and a thinner structure than most of the existing camera-based tactile sensors, without the use of additional reflecting components such as mirrors. A modular implementation of the learning architecture is discussed that facilitates the scalability to larger surfaces such as robotic skins.

Robotics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to design a multi - camera optical tactile sensor to provide information about the contact force distribution applied on its soft surface. Specifically, this paper aims to:

1. **Reduce sensor thickness**: Compared with the existing camera - based tactile sensors, this design achieves a thinner overall structure (about 17.5 mm) while maintaining a large contact surface.

2. **Improve sensor scalability**: By using four embedded cameras and a deep neural network (DNN), the sensor can cover a larger surface area and has a modular design, which is convenient for expanding to larger - scale applications, such as robot skin.

3. **Achieve real - time processing**: Utilizing the GPU capabilities provided by the embedded computing module (such as NVIDIA Jetson Nano), the sensor is able to predict the contact force distribution 40 times per second under real - time conditions.

4. **Solve the limitations of traditional camera tactile sensors**: Traditional camera tactile sensors are usually large in size, and due to the minimum focal length limitation of the lens, additional space is required to place the monitoring pattern between the camera and the soft gel, which increases the overall size. This design reduces these limitations by using a fisheye lens and an optimized hardware layout.

### Specific problems and solutions

- **Problem**: Traditional camera tactile sensors are large in volume and high in thickness.
- **Solution**: Adopt four closely - arranged embedded cameras and use a fisheye lens to increase the field of view, thereby reducing the overall thickness of the sensor.

- **Problem**: How to accurately extract contact force distribution information from images.
- **Solution**: Through a deep - learning architecture, directly reconstruct the three - dimensional contact force distribution from the image pixel intensity. Use the real - label data generated by the finite element method (FEM) simulation during the training process.

- **Problem**: How to achieve large - area expansion of the sensor.
- **Solution**: Propose a modular learning architecture, so that the sensor can adapt to newly added cameras by only re - training some layers without re - training the entire network, thereby achieving large - area expansion.

In summary, this paper solves the challenges of existing tactile sensors in terms of thickness, real - time performance and scalability through innovative multi - camera design and deep - learning technology.