Abstract:In Marius von Senden's Space and Sight, a newly sighted blind patient describes the experience of a corner as lemon-like, because corners "prick" sight like lemons prick the tongue. Prickliness, here, is a dimension in the feature space of sensory experience, an effect of the perceived on the perceiver that arises where the two interact. In the account of the newly sighted, an effect familiar from one interaction translates to a novel context. Perception serves as the vehicle for generalization, in that an effect shared across different experiences produces a concrete abstraction grounded in those experiences. Cezanne and the post-impressionists, fluent in the language of experience translation, realized that the way to paint a concrete form that best reflected reality was to paint not what they saw, but what it was like to see. We envision a future of creation using AI where what it is like to see is replicable, transferrable, manipulable - part of the artist's palette that is both grounded in a particular context, and generalizable beyond it. An active line of research maps human-interpretable features onto directions in GAN latent space. Supervised and self-supervised approaches that search for anticipated directions or use off-the-shelf classifiers to drive image manipulation in embedding space are limited in the variety of features they can uncover. Unsupervised approaches that discover useful new directions show that the space of perceptually meaningful directions is nowhere close to being fully mapped. As this space is broad and full of creative potential, we want tools for direction discovery that capture the richness and generalizability of human perception. Our approach puts creators in the discovery loop during real-time tool use, in order to identify directions that are perceptually meaningful to them, and generate interpretable image translations along those directions.

Perspective (In)consistency of Paint by Text

Lighting (In)consistency of Paint by Text

Creating Word Paintings Jointly Considering Semantics, Attention, and Aesthetics.

Chasing Consistency in Text-to-3D Generation from a Single Image.

Everyone Can Be Picasso? A Computational Framework into the Myth of Human versus AI Painting

Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing

Novel 3D-Aware Composition Images Synthesis for Object Display with Diffusion Model.

Mind Artist: Creating Artistic Snapshots with Human Thought

PixelSynth: Generating a 3D-Consistent Experience from a Single Image

Art2Real: Unfolding the Reality of Artworks via Semantically-Aware Image-to-Image Translation

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

PaintScene4D: Consistent 4D Scene Generation from Text Prompts

Learning to Synthesize Graphics Programs for Geometric Artworks

Language Does More Than Describe: On The Lack Of Figurative Speech in Text-To-Image Models

Latent Compass: Creation by Navigation

Generative artificial intelligence, human creativity, and art

Inverse Painting: Reconstructing The Painting Process

Exploring the Space of Abstract Textures by Principles and Random Sampling

Intelli-Paint: Towards Developing Human-like Painting Agents

Do DALL-E and Flamingo Understand Each Other?

Robot Synesthesia: A Sound and Emotion Guided AI Painter