Abstract:This dissertation attempts to drive innovation in the field of generative modeling for computer vision, by exploring novel formulations of conditional generative models, and innovative applications in images, 3D animations, and video. Our research focuses on architectures that offer reversible transformations of noise and visual data, and the application of encoder-decoder architectures for generative tasks and 3D content manipulation. In all instances, we incorporate conditional information to enhance the synthesis of visual data, improving the efficiency of the generation process as well as the generated content. We introduce the use of Neural ODEs to model video dynamics using an encoder-decoder architecture, demonstrating their ability to predict future video frames despite being trained solely to reconstruct current frames. Next, we propose a conditional variant of continuous normalizing flows that enables higher-resolution image generation based on lower-resolution input, achieving comparable image quality while reducing parameters and training time. Our next contribution presents a pipeline that takes human images as input, automatically aligns a user-specified 3D character with the pose of the human, and facilitates pose editing based on partial inputs. Next, we derive the relevant mathematical details for denoising diffusion models that use non-isotropic Gaussian processes, and show comparable generation quality. Finally, we devise a novel denoising diffusion framework capable of solving all three video tasks of prediction, generation, and interpolation. We perform ablation studies, and show SOTA results on multiple datasets. Our contributions are published articles at peer-reviewed venues. Overall, our research aims to make a meaningful contribution to the pursuit of more efficient and flexible generative models, with the potential to shape the future of computer vision.

Generative Modeling with Conditional Autoencoders: Building an Integrated Cell

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

Expression Conditional Gan for Facial Expression-to-Expression Translation.

Cellcyclegan: Spatiotemporal Microscopy Image Synthesis Of Cell Populations Using Statistical Shape Models And Conditional Gans

Generative modeling and latent space arithmetics predict single-cell perturbation response across cell types, studies and species

An Organism Starts with a Single Pix-Cell: A Neural Cellular Diffusion for High-Resolution Image Synthesis

CellGAN: Conditional Cervical Cell Synthesis for Augmenting Cytopathological Image Classification

Capturing Single-Cell Phenotypic Variation via Unsupervised Representation Learning

Generative modeling of living cells with SO(3)-equivariant implicit neural representations

Learning Consistent Subcellular Landmarks to Quantify Changes in Multiplexed Protein Maps.

A one-to-many conditional generative adversarial network framework for multiple image-to-image translations

Predicting cell morphological responses to perturbations using generative modeling

Neural Cellular Automata Manifold

Virtual organelle self-coding for fluorescence imaging via adversarial learning

Implicit Neural Representations for Generative Modeling of Living Cell Shapes

Generative Spatiotemporal Modeling Of Neutrophil Behavior

Conditional Generative Modeling for Images, 3D Animations, and Video

SubCell: Vision foundation models for microscopy capture single-cell biology

Unsupervised Learning for Cell-level Visual Representation in Histopathology Images with Generative Adversarial Networks.

High-Resolution In Silico Painting with Generative Models

Super-Resolution Reconstruction of Cell Images Based on Generative Adversarial Networks