Smart Shelf System for Customer Behavior Tracking in Supermarkets

John Anthony C. Jose,Christopher John B. Bertumen,Marianne Therese C. Roque,Allan Emmanuel B. Umali,Jillian Clara T. Villanueva,Richard Josiah TanAi,Edwin Sybingco,Jayne San Juan,Erwin Carlo Gonzales

DOI: https://doi.org/10.3390/s24020367

IF: 3.9

2024-01-09

Sensors

Abstract:Transactional data from point-of-sales systems may not consider customer behavior before purchasing decisions are finalized. A smart shelf system would be able to provide additional data for retail analytics. In previous works, the conventional approach has involved customers standing directly in front of products on a shelf. Data from instances where customers deviated from this convention, referred to as "cross-location", were typically omitted. However, recognizing instances of cross-location is crucial when contextualizing multi-person and multi-product tracking for real-world scenarios. The monitoring of product association with customer keypoints through RANSAC modeling and particle filtering (PACK-RMPF) is a system that addresses cross-location, consisting of twelve load cell pairs for product tracking and a single camera for customer tracking. In this study, the time series vision data underwent further processing with R-CNN and StrongSORT. An NTP server enabled the synchronization of timestamps between the weight and vision subsystems. Multiple particle filtering predicted the trajectory of each customer's centroid and wrist keypoints relative to the location of each product. RANSAC modeling was implemented on the particles to associate a customer with each event. Comparing system-generated customer–product interaction history with the shopping lists given to each participant, the system had a general average recall rate of 76.33% and 79% for cross-location instances over five runs.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation

What problem does this paper attempt to address?

The paper attempts to address the problem of tracking customer behavior in supermarkets through an intelligent shelf system to obtain additional data for retail analysis. Specifically, the paper aims to solve the following issues:

1. **Limitations of existing systems**: Traditional Point of Sale (POS) systems can track sales volume but cannot provide data on customer behavior before making a purchase decision. Therefore, a method is needed to capture more detailed data to better understand customer behavior.

2. **Cross-Location Issue**: When customers are not directly standing in front of the shelf, existing intelligent shelf systems usually ignore this data. However, this "cross-location" data is crucial for tracking multiple customers and multiple products. The method proposed in this paper can handle such cross-location situations.

3. **Complexity of multi-customer and multi-product interactions**: In an environment with multiple customers and multiple products, the interaction between customers and products becomes more complex. The paper proposes a new method that combines visual data and weight sensors, using particle filtering and Random Sample Consensus (RANSAC) modeling to associate customer and product interaction events.

In summary, the main goal of this paper is to develop an intelligent shelf system that can accurately track customer behavior in a supermarket environment and address cross-location issues, thereby providing retailers with more comprehensive and detailed data support.