Abstract:Data-centric AI has emerged as a new research area to systematically engineer the data to land AI models for real-world applications. As a core method for data-centric AI, data programming helps experts inject domain knowledge into data and label data at scale using carefully designed labeling functions (e.g., heuristic rules, logistics). Though data programming has shown great success in the NLP domain, it is challenging to program image data because of a) the challenge to describe images using visual vocabulary without human annotations and b) lacking efficient tools for data programming of images. We present Visual Concept Programming, a first-of-its-kind visual analytics approach of using visual concepts to program image data at scale while requiring a few human efforts. Our approach is built upon three unique components. It first uses a self-supervised learning approach to learn visual representation at the pixel level and extract a dictionary of visual concepts from images without using any human annotations. The visual concepts serve as building blocks of labeling functions for experts to inject their domain knowledge. We then design interactive visualizations to explore and understand visual concepts and compose labeling functions with concepts without writing code. Finally, with the composed labeling functions, users can label the image data at scale and use the labeled data to refine the pixel-wise visual representation and concept quality. We evaluate the learned pixel-wise visual representation for the downstream task of semantic segmentation to show the effectiveness and usefulness of our approach. In addition, we demonstrate how our approach tackles real-world problems of image retrieval for autonomous driving.

DeepVisual: a visual programming tool for deep learning systems

Towards A Visual Programming Tool to Create Deep Learning Models

A Visual Programming Paradigm for Abstract Deep Learning Model Development

NeuralVis: Visualizing and Interpreting Deep Learning Models

VisuaLizations As Intermediate Representations (VLAIR): an Approach for Applying Deep Learning-Based Computer Vision to Non-Image-based Data

Understanding Neural Networks Through Deep Visualization

An In-Situ Visual Analytics Framework for Deep Neural Networks

Deep Learning Development Environment in Virtual Reality

Deep Learning Approaches in Flow Visualization

DPVis: Automatic Visual Encoding Based on Deep Learning

Deep Learning, Machine Learning -- Digital Signal and Image Processing: From Theory to Application

De-fine: Decomposing and Refining Visual Programs with Auto-Feedback

Visual Concept Programming: A Visual Analytics Approach to Injecting Human Intelligence at Scale

Evolutionary Programming Based Deep Learning Feature Selection and Network Construction for Visual Data Classification

Exploring Multiple Application Scenarios of Visual Communication Course Using Deep Learning Under the Digital Twins

JustDeepIt: Software tool with graphical and character user interfaces for deep learning-based object detection and segmentation in image analysis

Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding

Study on the Application of Visual Communication Design in APP Interface Design in the Context of Deep Learning

DeepEye: Towards Automatic Data Visualization

A comprehensive study on challenges in deploying deep learning based software

Towards interpretable-by-design deep learning algorithms