Abstract:Online user generated content games (UGCGs) are increasingly popular among children and adolescents for social interaction and more creative online entertainment. However, they pose a heightened risk of exposure to explicit content, raising growing concerns for the online safety of children and adolescents. Despite these concerns, few studies have addressed the issue of illicit image-based promotions of unsafe UGCGs on social media, which can inadvertently attract young users. This challenge arises from the difficulty of obtaining comprehensive training data for UGCG images and the unique nature of these images, which differ from traditional unsafe content. In this work, we take the first step towards studying the threat of illicit promotions of unsafe UGCGs. We collect a real-world dataset comprising 2,924 images that display diverse sexually explicit and violent content used to promote UGCGs by their game creators. Our in-depth studies reveal a new understanding of this problem and the urgent need for automatically flagging illicit UGCG promotions. We additionally create a cutting-edge system, UGCG-Guard, designed to aid social media platforms in effectively identifying images used for illicit UGCG promotions. This system leverages recently introduced large vision-language models (VLMs) and employs a novel conditional prompting strategy for zero-shot domain adaptation, along with chain-of-thought (CoT) reasoning for contextual identification. UGCG-Guard achieves outstanding results, with an accuracy rate of 94% in detecting these images used for the illicit promotion of such games in real-world scenarios.

What problem does this paper attempt to address?

This paper primarily addresses the issue of illegal image promotion in User-Generated Content Games (UGCGs). With the increasing popularity of UGCGs among children and adolescents, the unsafe content (such as sexually suggestive and violent images) contained in these games poses a threat to the online safety of young users. However, there is currently relatively little research and countermeasures for such illegal image promotion. ### Problems Addressed by the Paper 1. **Data Collection and Understanding**: First, the paper identifies keywords related to unsafe UGCG by analyzing self-reported stories shared by users on the Common Sense Media platform about UGCGs. Based on these keywords, it collects potential UGCG promotion images from social platforms (such as X, i.e., Twitter). 2. **Evaluation of Existing Tools**: Researchers evaluated the performance of some existing unsafe image detection systems (such as Google Cloud Vision API, Clarifai, etc.) in detecting UGCG promotion images and found that these systems have limitations when facing UGCG promotion images. 3. **Design of a New Framework**: To address the above issues, the paper designs a new system called UGCG-GUARD to label images that illegally promote unsafe UGCG. This system utilizes large visual language models (VLMs) and a novel conditional prompting strategy to achieve zero-shot adaptation, and it also employs a chain-of-thought (CoT) reasoning mechanism to identify contextual information in images. ### Main Contributions 1. **New Dataset**: Constructed a real-world image dataset containing 2,924 images used by actual game creators on the social platform X for unsafe UGCG promotion. 2. **New Understanding of Unsafe UGCG and Its Illegal Promotion**: Through the study of these illegal online image promotions, it reveals that such promotions often use inappropriate images captured from UGCG as a means of advertisement. 3. **New Framework**: Proposed an advanced framework, UGCG-GUARD, which can effectively identify images that illegally promote unsafe UGCG. This system can achieve efficient detection without requiring extensive specific data training. 4. **System Performance Evaluation**: UGCG-GUARD achieved an average accuracy of 94% in labeling such illegal promotion images, significantly outperforming existing baseline detectors. In summary, this paper aims to address the issue of illegal promotion of unsafe UGCG images by developing new technologies and methods to enhance the online safety of children and adolescents.

Moderating Illicit Online Image Promotion for Unsafe User-Generated Content Games Using Large Vision-Language Models

Towards Understanding Unsafe Video Generation

Image Safeguarding: Reasoning with Conditional Vision Language Model and Obfuscating Unsafe Content Counterfactually

XAI-Driven Explainable Multi-view Game Cheating Detection

Unsupervised Representation Learning of Player Behavioral Data with Confidence Guided Masking

Explainable AI for Cheating Detection and Churn Prediction in Online Games

GuardT2I: Defending Text-to-Image Models from Adversarial Prompts

UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images

Advancing Content Moderation: Evaluating Large Language Models for Detecting Sensitive Content Across Text, Images, and Videos

Perceptual Quality Assessment of UGC Gaming Videos

The Potential of Vision-Language Models for Content Moderation of Children's Videos

SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models

Image Matters: Scalable Detection of Offensive and Non-Compliant Content / Logo in Product Images

T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognition

Evaluating and Mitigating IP Infringement in Visual Generative AI

Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models

Subjective and Objective Analysis of Streamed Gaming Videos

Combating the Elsagate Phenomenon: Deep Learning Architectures for Disturbing Cartoons

VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data

ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users

VGMShield: Mitigating Misuse of Video Generative Models