Abstract:Online content is filled with logos, from ads and social media posts to website branding and product placements. Consequently, these logos are prevalent in the extensive web-scraped datasets used to pretrain Vision-Language Models, which are used for a wide array of tasks (content moderation, object classification). While these models have been shown to learn harmful correlations in various tasks, whether these correlations include logos remains understudied. Understanding this is especially important due to logos often being used by public-facing entities like brands and government agencies. To that end, we develop SLANT: A Spurious Logo ANalysis Toolkit. Our key finding is that some logos indeed lead to spurious incorrect predictions, for example, adding the Adidas logo to a photo of a person causes a model classify the person as greedy. SLANT contains a semi-automatic mechanism for mining such "spurious" logos. The mechanism consists of a comprehensive logo bank, CC12M-LogoBank, and an algorithm that searches the bank for logos that VLMs spuriously correlate with a user-provided downstream recognition target. We uncover various seemingly harmless logos that VL models correlate 1) with negative human adjectives 2) with the concept of `harmlessness'; causing models to misclassify harmful online content as harmless, and 3) with user-provided object concepts; causing lower recognition accuracy on ImageNet zero-shot classification. Furthermore, SLANT's logos can be seen as effective attacks against foundational models; an attacker could place a spurious logo on harmful content, causing the model to misclassify it as harmless. This threat is alarming considering the simplicity of logo attacks, increasing the attack surface of VL models. As a defense, we include in our Toolkit two effective mitigation strategies that seamlessly integrate with zero-shot inference of foundation models.

Design of a Modular Framework for Noisy Logo Classification in Fraud Detection.

Logo Detection and Recognition Based on Classification

An Effective and Robust Detector for Logo Detection

A two-stage deep learning framework for counterfeit luxury handbag detection in logo images

Logo Classification and Data Augmentation Techniques for PCB Assurance and Counterfeit Detection

Automatic video logo detection and removal

Robust Logo Detection in E-Commerce Images by Data Augmentation

Automatic location and removal of video logos

Automatic Graphic Logo Detection via Fast Region-based Convolutional Networks

Logo Detection Based on Spatial-Spectral Saliency and Partial Spatial Context

A numeric-based machine learning design for detecting organized retail fraud in digital marketplaces

An adaptive representation algorithm for Multi-scale logo detection

A Robust Image Forensic Framework Utilizing Multi-Colorspace Enriched Vision Transformer for Distinguishing Natural and Computer-Generated Images

Scalable Logo Recognition Based on Compact Sparse Dictionary for Mobile Devices.

Logo Detection in Document Images Based on Boundary Extension of Feature Rectangles

A Secure Fingerprinting Framework for Distributed Image Classification

LOGO-Net: Large-scale Deep Logo Detection and Brand Recognition with Deep Region-based Convolutional Networks

Logo Information Recognition in Large-Scale Social Media Data

SLANT: Spurious Logo ANalysis Toolkit

Multimodal Large Language Models for Phishing Webpage Detection and Identification

Letting Logos Speak: Leveraging Multiview Representation Learning for Data-Driven Logo Design