Analysis and Synthesis of Textured Motion

WangYizhou,ZhuSong-Chun

IF: 23.6

2004-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Natural scenes contain a wide range of textured motion phenomena which are characterized by the movement of a large amount of particle and wave elements, such as falling snow, wavy water, and danci...

What problem does this paper attempt to address?

A Generative Method for Textured Motion: Analysis and Synthesis

Yizhou Wang,Song Chun Zhu

DOI: https://doi.org/10.1007/3-540-47969-4_39

2002-01-01

Abstract:Natural scenes contain rich stochastic motion patterns which are characterized by the movement of a large number of small elements, such as falling snow, raining, flying birds, firework and waterfall. In this paper, we call these motion patterns textured motion and present a generative method that combines statistical models and algorithms from both texture and motion analysis. The generative method includes the following three aspects. 1). Photometrically, an image is represented as a superposition of linear bases in atomic decomposition using an over-complete dictionary, such as Gabor or Laplacian. Such base representation is known to be generic for natural images, and it is low dimensional as the number of bases is often 100 times smaller than the number of pixels. 2). Geometrically, each moving element (called moveton), such as the individual snowflake and bird, is represented by a deformable template which is a group of several spatially adjacent bases. Such templates are learned through clustering. 3). Dynamically, the movetons are tracked through the image sequence by a stochastic algorithm maximizing a posterior probability. A classic second order Markov chain model is adopted for the motion dynamics. The sources and sinks of the movetons are modeled by birth and death maps. We adopt an EM-like stochastic gradient algorithm for inference of the hidden variables: bases, movetons, birth/death maps, parameters of the dynamics. The learned models are also verified through synthesizing random textured motion sequences which bear similar visual appearance with the observed sequences.
Analysis and Synthesis of Textured Motion: Particle, Wave and Cartoon Sketch

Yizhou Wang,Song-Chun Zhu

IF: 23.6

2003-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Natural scenes contain a wide range of textured motion patterns which are characterized by the move- ment of a large amount of particle and wave elements, such as falling snow, water waves, dancing grass, etc. In this paper, we present a generative method for modeling these motion phenomena and our method consists of four components: (1). A photometric model which represents an image as a linear superposition of image bases selected from a generic and over-complete dictionary. The dictionary contains Gabor and LoG bases for point/particle-elements and Fourier bases for wave-elements. These bases compete to explain the input images and transfer a raw image to a base (token) representation with O(102)-fold of dimension reduction. (2). A geometric model which groups adjacent bases and their motion trajectories into a number of basic moving elements { called \motons". A moton is a deformable template in space-time representing a moving element, such as a snow∞ake. (3). A dynamic model which characterizes the motion of particles, waves, and their interactions, e.g., balls/leaves ∞oating on water. The trajectories of these elements are coupled Markov chains. Given an input video sequence, a statistical learning algorithm computes a set of motons with their trajectories as hidden variables. It also learns the parameters that govern the geo- metric deformations and motion dynamics by maximum likelihood estimation (MLE). Consequently, novel sequences are synthesized easily from the learnt models. (4). A sketch model replaces the dictionary of Gabor/LoG and Fourier bases with symbolic sketches (token symbols). With the same generative model, the sketch model can render realistic and stylish cartoon animation. In this paper, cartoon sketch is viewed as a symbolic visualization and simpliflcation of the inner representation in visual perception. The success of the cartoon animation, in turn, suggests that the generative model captures the essence of visual perception of textured motion.
Analysis and synthesis of textured motion: particles and waves.

Yizhou Wang,Song-Chun Zhu

DOI: https://doi.org/10.1109/TPAMI.2004.76

IF: 23.6

2004-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Natural scenes contain a wide range of textured motion phenomena which are characterized by the movement of a large amount of particle and wave elements, such as falling snow, wavy water, and dancing grass. In this paper, we present a generative model for representing these motion patterns and study a Markov chain Monte Carlo algorithm for inferring the generative representation from observed video sequences. Our generative model consists of three components. The first is a photometric model which represents an image as a linear superposition of image bases selected from a generic and overcomplete dictionary. The dictionary contains Gabor and LoG bases for point/particle elements and Fourier bases for wave elements. These bases compete to explain the input images and transfer them to a token (base) representation with an O(10(2))-fold dimension reduction. The second component is a geometric model which groups spatially adjacent tokens (bases) and their motion trajectories into a number of moving elements--called "motons." A moton is a deformable template in time-space representing a moving element, such as a falling snowflake or a flying bird. The third component is a dynamic model which characterizes the motion of particles, waves, and their interactions. For example, the motion of particle objects floating in a river, such as leaves and balls, should be coupled with the motion of waves. The trajectories of these moving elements are represented by coupled Markov chains. The dynamic model also includes probabilistic representations for the birth/death (source/sink) of the motons. We adopt a stochastic gradient algorithm for learning and inference. Given an input video sequence, the algorithm iterates two steps: 1) computing the motons and their trajectories by a number of reversible Markov chain jumps, and 2) learning the parameters that govern the geometric deformations and motion dynamics. Novel video sequences are synthesized from the learned models and, by editing the model parameters, we demonstrate the controllability of the generative model.
The Technology of Sampled-based Texture Synthesis

朱文浩,魏宝刚

2008-01-01

Abstract:Texture synthesis has been well studied in recent years.It has become an active research topic of computer graphics,vision,and image processing.Aiming at the current progress of texture synthesis,this work we set forth the algorithms based on synthesis sampling,character matching,constrained synthesis and multi-objects synthesis.Further more,we analyze typical algorithms' advantage and the disadvantage on the point of synthesis quality and efficiency.At last,we discuss the orientation of texture synthesis and emphasize that the constrained synthesis techniques would speed up the study development and become the key point of synthesis application.
Using Texture Synthesis in Fractal Pattern Design

Wei Bao-gang,Li Jian-ping,Pang Xiang-bin

DOI: https://doi.org/10.1631/jzus.2006.a0289

2006-01-01

Abstract:Traditional fractal pattern design has some disadvantages such as inability to effectively reflect the characteristics of real scenery and texture. We propose a novel pattern design technique combining fractal geometry and image texture synthesis to solve these problems. We have improved Wei and Levoy (2000)’s texture synthesis algorithm by first using two-dimensional autocorrelation function to analyze the structure and distribution of textures, and then determining the size of L neighborhood. Several special fractal sets were adopted and HSL (Hue, Saturation, and Light) color space was chosen. The fractal structure was used to manipulate the texture synthesis in HSL color space where the pattern’s color can be adjusted conveniently. Experiments showed that patterns with different styles and different color characteristics can be more efficiently generated using the new technique.
Modeling Textured Motion : Particle, Wave and Sketch.

YZ Wang,SC Zhu

DOI: https://doi.org/10.1109/iccv.2003.1238343

2003-01-01

Abstract:We present a generative model for textured motion phenomena, such as falling snow, wavy river and dancing grass, etc. Firstly, we represent an image as a linear superposition of image bases selected from a generic and over-complete dictionary. The dictionary contains Gabor bases for point/particle elements and Fourier bases for wave-elements. These bases compete to explain the input images. The transform from a raw image to a base or a token representation leads to large dimension reduction. Secondly, we introduce a unified motion equation to characterize the motion of these bases and the interactions between waves and particles, e.g. a ball floating on water. We use statistical learning algorithm to identify the structure of moving objects and their trajectories automatically. Then novel sequences can be synthesized easily from the motion and image models. Thirdly, we replace the dictionary of Gabor and Fourier bases with symbolic sketches (also bases). With the same image and motion model, we can render realistic and stylish cartoon animation. In our view, cartoon and sketch are symbolic visualization of the inner representation for visual perception. The success of the cartoon animation, in turn, suggests that our image and motion models capture the essence of visual perception of textured motion.
A Novel Method of Dynamic Textures Analysis and Synthesis.

Chuanzhen Li,Jingling Wang,Long Ye,Hui Wang

DOI: https://doi.org/10.1109/cso.2009.381

2009-01-01

Abstract:Dynamic textures (or temporal textures) are a kind of visual motion pattern, and their dynamics are not trackable. Dynamic textures such as flames, smoke and water are statistical stationary and exhibit spatial self-similarity and temporal self-similarity. Recently, description of dynamic textures has attracted growing attention. In this paper, a novel method for analyzing and synthesizing dynamic textures is proposed First, the dynamic textures are modeled by combination of dynamic texture units. Second, the segment parameters and dynamic texture units' parameters are learnt at the same time. To reduce computational complexity, we discard Fourier coefficients that are under contribution ratio threshold. Our algorithm is general and automatic, and it works well on various types of textures. Experimental results demonstrate that our approach can reconstruct dynamic texture sequences with promising visual quality and fewer coefficients.
Motion Field Texture Synthesis.

Chongyang Ma,Li-Yi Wei,Baining Guo,Kun Zhou

DOI: https://doi.org/10.1145/1661412.1618456

IF: 6.2

2009-01-01

ACM Transactions on Graphics

Abstract:A variety of animation effects such as herds and fluids contain detailed motion fields characterized by repetitive structures. Such detailed motion fields are often visually important, but tedious to specify manually or expensive to simulate computationally. Due to the repetitive nature, some of these motion fields (e.g. turbulence in fluids) could be synthesized by procedural texturing, but procedural texturing is known for its limited generality. We apply example-based texture synthesis for motion fields. Our technique is general and can take on a variety of user inputs, including captured data, manual art, and physical/procedural simulation. This data-driven approach enables artistic effects that are difficult to achieve via previous methods, such as heart shaped swirls in fluid animation. Due to the use of texture synthesis, our method is able to populate a large output field from a small input exemplar, imposing minimum user workload. Our algorithm also allows the synthesis of output motion fields not only with the same dimension as the input (e.g. 2D to 2D) but also of higher dimension, such as 3D volumetric outputs from 2D planar inputs. This cross-dimension capability supports a convenient usage scenario, i.e. the user could simply supply 2D images and our method produces a 3D motion field with similar characteristics. The motion fields produced by our method are generic, and could be combined with a variety of large-scale low-resolution motions that are easy to specify either manually or computationally but lack the repetitive structures to be characterized as textures. We apply our technique to a variety of animation phenomena, including smoke, liquid, and group motion.
High-quality Texture-based Flow Visualization on Surfaces.

Dibin Zhou,Kangjian Wang,Yao Zheng,Yanni Wang

DOI: https://doi.org/10.1109/cadcg.2007.4407875

2008-01-01

Abstract:In this paper, we present a high-quality texture-based approach for the visualization of vector fields on surface. It adopts a scheme of streamline enhancement by applying a ID high-pass filter to flow textures along the orthogonal flow direction to enhance the intensity contrast among streamlines and improve the image quality of flow textures. The scheme of edge detection and supplement is used in our approach to modulate the characteristic of particles flow. Our algorithm is image space based and can achieve a real-time rendering performance on a personal computer by using parallel processing ability of current graphics hardware.
Text-driven Visual Prosody Generation for Embodied Conversational Agents

Jiali Chen,Yong Liu,Zhimeng Zhang,Changjie Fan,Yu Ding

DOI: https://doi.org/10.1145/3308532.3329445

2019-01-01

Abstract:In face-to-face conversations, head motions play a crucial role in encoding information, and humans are very skilled at decoding multiple messages from interlocutors' head motions. It is of great importance to endow embodied conversational agents (ECAs) with the capability of conveying communicative intention through head movements. Our work is aimed at automatically synthesizing head motions for an ECA speaking Chinese. We propose to take only transcripts as input to compute head movements, based on a statistical framework. Subjective experiments are conducted to validate the proposed statistical framework. The results show that the generated head animation is able to improve human perception in terms of naturalness and demonstrate that the head animation is synchronized with the input of synthetic speech.
Dynamic texture synthesis using a spatial temporal descriptor

Yimo Guo,Guoying Zhao,Jie Chen,Matti Pietikäinen,Zhengguang Xu

DOI: https://doi.org/10.1109/ICIP.2009.5414395

2009-01-01

ICIP

Abstract:Dynamic textures are image sequences with visual pattern repetition in time and space, such as smoke, flames, moving objects and so on. Dynamic texture synthesis is to provide a continuous and infinitely varying stream of images by doing operations on dynamic textures. Considering that the previous video texture method provides high-quality visual results, but its representation does not well explore the temporal correlation among frames, we develop a novel spatial temporal descriptor for frame description accompanied with a similarity measure on the basis of the video texture method. Compared with the previous one, our method considers both the spatial and temporal domains of video sequences in representation; moreover, combines the local and global description on each spatial-temporal plane. From experimental results, the proposed method achieves better performance in both the syntheses of natural scene and human motion. Especially, it has the characteristic to be robust to noise in remodeling videos into infinite time domain.
Wavelet-based Texture Model for Crowd Dynamic Analysis

Jing Wang,Zhijie Xu,Yanlong Cao,Yuanping Xu

DOI: https://doi.org/10.23919/iconac.2017.8082022

2017-01-01

Abstract:Crowd event detection techniques aim at solving real-world surveillance problems, such as detecting crowd anomaly and tracking specific person in a highly dynamic crowd scene. In this paper, we proposed an innovate texture-based analysis method to model crowd dynamics and us it to distinguish the crowd behaviours. To describe complicated crowd scenes, homogeneous random features have been deployed in the research for behavioural template matching. Experiment results have shown that the anomaly appearing in crowd scenes can be effectively and efficiently identified by using the devised methods.
Real-time Snowing Simulation

Changbo Wang,Zhangye Wang,Tian Xia,Qunsheng Peng

DOI: https://doi.org/10.1007/s00371-006-0012-8

2006-01-01

Abstract:A snowing scene has a unique fascination for people due to its incomparable beauty. However, little work has been presented on the real-time generation of a dynamic snowing scene, partially due to the difficulty that the simulation of a dynamic snowing process involves the complex modeling of the wind field and the interaction between wind and snow. In this paper, by fully considering the physical characteristics of wind and snow, we construct a three-dimensional wind field based on the discrete form of the Boltzmann equation. According to the interaction laws between wind and snow, we simulate the falling of snow, deposition and erosion in 3D space. Experimental results show that realistic wind-driven snow scenes under different speeds of wind with different amounts of snowfall can be rendered in real-time.
Natural Textures for Weather Data Visualization.

Ying Tang,Huamin Qu,Yingcai Wu,Hong Zhou

DOI: https://doi.org/10.1109/iv.2006.77

2006-01-01

Abstract:In this paper we present a novel method to visualize weather data with multi-layer controllable texture synthesis. Texture possesses multiple principal perceptual channels, which makes it good at encoding multiple data attributes contained in weather data. The natural textures existed in the real world especially provide plenty of choices to encode the data with visually pleasing images. A controllable texture synthesis method is developed to generate a large amount of textures which change the appearances of their individual perceptual dimensions according to the underlying distribution of data attributes. In order to encode more data attributes we further propose multi-layer texture synthesis. The background and foreground textures are separately synthesized and then combined together for display. In the end, we apply our method to some real-world weather data and demonstrate its effectiveness with a user study.
Structural Pattern Analysis for Texture Synthesis

Yizhe Li,Guoliang Chen,Qilong Zheng

DOI: https://doi.org/10.1109/IMSCCS.2006.129

2006-01-01

Abstract:Sample based texture synthesis allows the generation of arbitrarily sized textures based on a small sample image. One main drawback of the technique is that a new pixel or patch is selected only considering local information in the generated texture. Thus structural error can't assure to be avoided for structural textures. To accelerate texture synthesis process and to avoid structural mismatch in the output picture, we present in this paper a structural pattern analysis (SPA) algorithm and a texture synthesis method using the algorithm. The pattern analysis tool also quantizes the texture randomness in macro scope view. It can be used in other applications such as automatic texture classification and quality evaluation of synthesized texture. We illustrate our result with some examples
Real-time simulation of snowing scene

Changbo Wang,Zhangye Wang,Tian Xia,Qunsheng Peng

2004-01-01

Ruan Jian Xue Bao/Journal of Software

Abstract:Realistic simulation of natural scene has been one of the most challenging tasks in compute graphics. And it is even more difficult to real-time simulate snowing scene because the complex modeling of wind field and the interaction between wind and snow are involved. In this paper, based on the fully consideration of the physical characteristics of wind and snow, a three-dimensional wind field is constructed by discretizing the classic Boltzmann equation. According to the interactive laws between wind and snow, the changing rules of snow's falling, deposition and erosion are then established. Finally, by adapting a series of simplification and accelerating rendering techniques, realistic wind-driven snow scenes under different speeds of wind with different snow falling amount are rendered in real time.
Real-Time Rendering of Large-Scale Snow Scene

Yuxiang Shan,Zhangye Wang,Chunyan Yang,Qunsheng Peng

DOI: https://doi.org/10.3969/j.issn.1003-9775.2013.08.006

2013-01-01

Abstract:Real-time rendering of large-scale snow scenes with high realism has wide applications in the areas such as virtual reality, snowy disaster's prevention and rescue, military simulation and game design, etc. However, current methods cannot simultaneously simulate both the snow accumulation and falling effects for large-scale dynamic snow scenes. To solve this problem, we propose a new method of modeling and real-time rendering. To simulate the accumulation of snowflakes on different objects in the scene precisely, we present an adaptive occlusion map for the falling snowflakes, with respect to the object distribution within the current view frustum. The occlusion map will be updated in real-time allowing dynamic objects to be included in the scene. To simulate the huge amount of dynamically falling snow flakes, we adopt a view-dependent particles grouping technique to reduce the amount of the snowflakes, meantime move the view transform and evolution of snow particle system into GPU for acceleration. To enhance the realism of falling snowflake, we adopt a dynamic texture sequence of multi-rotary snowflake. Besides, a hybrid approach of geometry and texture is employed to render the distant view of large-scale snow scenes with less computation. Based on the above approaches, we successfully implemented the real-time walkthrough of large-scale snow scenes including both urban and suburb circumstances, and generate the beautiful winter view such as heavy snow accumulation on the tree branches and the covers of cars, etc.
A Hybrid Motion Data Manipulation: Wavelet Based Motion Processing and Spacetime Rectification

Feng Liu,Yueting Zhuang,Zhongxiang Luo,Yunhe Pan

DOI: https://doi.org/10.1007/3-540-36228-2_92

2002-01-01

Abstract:In this paper, we present a hybrid approach to motion data manipulation. Motion signal is decomposed into multi-resolution levels with wavelet analysis. The coarse level represents the globe pattern of a motion signal while the fine levels describe the individual styles. Special motion style can be highlighted through enhancing the corresponding level content and can be fused into other motions by texturing them with related fine levels. And multiple motions can be synthesized by multi-resolution blending to create new motions somehow like to the blended motions. Motion signals implicitly preserve constraints to keep realistic. However, the above manipulations may inviolate some constraints and result in the unrealistic artifact. Spacetime rectification is proposed to reserve the essential constraints. Our experiment shows the effectiveness of this hybrid motion data manipulation approach.
Modeling And Rendering Of Realistic Waterfall Scenes With Dynamic Texture Sprites

Yu Guan,Wei Chen,Lincan Zou,Long Zhang,Qunsheng Peng

DOI: https://doi.org/10.1002/cav.156

IF: 1.01

2006-01-01

Computer Animation and Virtual Worlds

Abstract:We propose an efficient approach for authoring dynamic and realistic waterfall scenes based on an acquired video sequence. Traditional video based techniques generate new images by synthesizing 2D samples, i.e., texture sprites chosen from a video sequence. However, they are limited to one fixed viewpoint and cannot provide arbitrary walkthrough into 3D scenes. Our approach extends this scheme by synthesizing dynamic 2D texture sprites and projecting them into 3D space, We first generate a set of basis texture sprites, which capture the representative appearance and motions of waterfall scenes contained in the video sequence. To model the shape and motion Of a new waterfall scene, we interactively construct a set of flow lines taking account of physical principles. Along each flow line, the basis texture sprites are manipulated and animated dynamically, yielding a sequence of dynamic texture sprites in 3D space. These texture sprites are displayed using the point splatting technique, which can be accelerated efficiently by graphics hardware. By choosing varied basis texture sprites, waterfall scenes with different appearance and shapes can be conveniently simulated. The experimental results demonstrate that our approach achieves realistic effects and real-time frame rates on consumer PC platforms. Copyright (c) 2006 John Wiley & Sons, Ltd.
Modeling Complex Motion: Photometric, Geometric, Dynamic, and Topological Aspects

Song-Chun Zhu,Yizhou Wang

2005-01-01

Abstract:As natural scenes contain a huge number of motion patterns generated by various stochastic processes, how to represent and model these diverse visual patterns, and how to learn and compute the visual patterns efficiently are fundamental problems in computer vision. In this thesis, we present a unified theory with a generative statistical learning and inference framework to analyze complex visual patterns in both motion and scale-space. The goal of my research is to compute the semantic content of image sequences and representing them with symbolic graphs using generic visual vocabulary learned from natural images. Based on the generic graph representation, four important modeling issues are addressed, which are photometric, geometric, dynamic and topological issues. These four aspects span sub-dimensions in image space. The unified framework aims to answer questions like "What do we perceive when we look at a video sequence?" It tries to identify the perceptually meaningful basic elements and transitions in image sequences, model the elements' attributes and interactions in time-space and scale-space, and infer the hidden dynamic graph structures. We augment the model complexity progressively to learn various complicated visual patterns. We use three challenging examples to illustrate the learning and inference on these four aspects. Example (1): Textured motion modeling. Example (2): Topological changes in complex motion modeling. Example (3): Perceptual transitions in scale-space. The integrated model is generic. It can be broadly applied to many vision applications, such as tracking, video annotation, motion segmentation, document image analysis, surveillance, and automatic cartoon animation generation. The model learning and inference is achieved by employing Markov chain Monte Carlo (MCMC) sampling method.

Analysis and Synthesis of Textured Motion

A Generative Method for Textured Motion: Analysis and Synthesis

Analysis and Synthesis of Textured Motion: Particle, Wave and Cartoon Sketch

Analysis and synthesis of textured motion: particles and waves.

The Technology of Sampled-based Texture Synthesis

Using Texture Synthesis in Fractal Pattern Design

Modeling Textured Motion : Particle, Wave and Sketch.

A Novel Method of Dynamic Textures Analysis and Synthesis.

Motion Field Texture Synthesis.

High-quality Texture-based Flow Visualization on Surfaces.

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Dynamic texture synthesis using a spatial temporal descriptor

Wavelet-based Texture Model for Crowd Dynamic Analysis

Real-time Snowing Simulation

Natural Textures for Weather Data Visualization.

Structural Pattern Analysis for Texture Synthesis

Real-time simulation of snowing scene

Real-Time Rendering of Large-Scale Snow Scene

A Hybrid Motion Data Manipulation: Wavelet Based Motion Processing and Spacetime Rectification

Modeling And Rendering Of Realistic Waterfall Scenes With Dynamic Texture Sprites

Modeling Complex Motion: Photometric, Geometric, Dynamic, and Topological Aspects