Abstract:In Marius von Senden's Space and Sight, a newly sighted blind patient describes the experience of a corner as lemon-like, because corners "prick" sight like lemons prick the tongue. Prickliness, here, is a dimension in the feature space of sensory experience, an effect of the perceived on the perceiver that arises where the two interact. In the account of the newly sighted, an effect familiar from one interaction translates to a novel context. Perception serves as the vehicle for generalization, in that an effect shared across different experiences produces a concrete abstraction grounded in those experiences. Cezanne and the post-impressionists, fluent in the language of experience translation, realized that the way to paint a concrete form that best reflected reality was to paint not what they saw, but what it was like to see. We envision a future of creation using AI where what it is like to see is replicable, transferrable, manipulable - part of the artist's palette that is both grounded in a particular context, and generalizable beyond it. An active line of research maps human-interpretable features onto directions in GAN latent space. Supervised and self-supervised approaches that search for anticipated directions or use off-the-shelf classifiers to drive image manipulation in embedding space are limited in the variety of features they can uncover. Unsupervised approaches that discover useful new directions show that the space of perceptually meaningful directions is nowhere close to being fully mapped. As this space is broad and full of creative potential, we want tools for direction discovery that capture the richness and generalizability of human perception. Our approach puts creators in the discovery loop during real-time tool use, in order to identify directions that are perceptually meaningful to them, and generate interpretable image translations along those directions.

Deep Meditations: Controlled navigation of latent space

Latent Compass: Creation by Navigation

Towards Kinetic Manipulation of the Latent Space

Automatic Camera Trajectory Control with Enhanced Immersion for Virtual Cinematography

Motion Prompting: Controlling Video Generation with Motion Trajectories

3DLatNav: Navigating Generative Latent Spaces for Semantic-Aware 3D Object Manipulation

VideoMap: Supporting Video Editing Exploration, Brainstorming, and Prototyping in the Latent Space

A Mapping Strategy for Interacting with Latent Audio Synthesis Using Artistic Materials

Latent Traversals in Generative Models as Potential Flows

Navigating Chemical Space with Latent Flows

Semantics‐Guided Latent Space Exploration for Shape Generation

VividDream: Generating 3D Scene with Ambient Dynamics

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

The Wanderings of Odysseus in 3D Scenes

Compositional 3D-aware Video Generation with LLM Director

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation

Deep Generative Modelling of Human Reach-and-Place Action

Generative Human Motion Stylization in Latent Space

Playable Environments: Video Manipulation in Space and Time

Graph-based Latent Space Traversal for New Molecules Discovery.