Highlight-aware two-stream network for single-image SVBRDF acquisition

Jie Guo,Shuichang Lai,Chengzhi Tao,Yuelong Cai,Lei Wang,Yanwen Guo,Ling-Qi Yan

DOI: https://doi.org/10.1145/3476576.3476693

IF: 6.2

2021-08-01

ACM Transactions on Graphics

Abstract:ACM Transactions on Graphics, <a href="https://dl.acm.org/toc/tog/2021/40/4">Volume 40, Issue 4</a>, Page 1-14, August 2021. <br/>

computer science, software engineering

What problem does this paper attempt to address?

Ultra-High Resolution SVBRDF Recovery from a Single Image

Jie Guo,Shuichang Lai,Qinghao Tu,Chengzhi Tao,Changqing Zou,Yanwen Guo

DOI: https://doi.org/10.1145/3593798

IF: 6.2

2023-01-01

ACM Transactions on Graphics

Abstract:Existing convolutional neural networks have achieved great success in recovering Spatially Varying Bidirectional Surface Reflectance Distribution Function (SVBRDF) maps from a single image. However, they mainly focus on handling low-resolution (e.g., 256 × 256) inputs. Ultra-High Resolution (UHR) material maps are notoriously difficult to acquire by existing networks because (1) finite computational resources set bounds for input receptive fields and output resolutions, and (2) convolutional layers operate locally and lack the ability to capture long-range structural dependencies in UHR images. We propose an implicit neural reflectance model and a divide-and-conquer solution to address these two challenges simultaneously. We first crop a UHR image into low-resolution patches, each of which are processed by a local feature extractor to extract important details. To fully exploit long-range spatial dependency and ensure global coherency, we incorporate a global feature extractor and several coordinate-aware feature assembly modules into our pipeline. The global feature extractor contains several lightweight material vision transformers that have a global receptive field at each scale and have the ability to infer long-term relationships in the material. After decoding globally coherent feature maps assembled by coordinate-aware feature assembly modules, the proposed end-to-end method is able to generate UHR SVBRDF maps from a single image with fine spatial details and consistent global structures.
Single-image SVBRDF capture with a rendering-aware deep network

Valentin Deschaintre,Miika Aittala,Fredo Durand,George Drettakis,Adrien Bousseau

DOI: https://doi.org/10.1145/3197517.3201378

IF: 6.2

2018-08-31

ACM Transactions on Graphics

Abstract:Texture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image based on such cues has challenged researchers in computer graphics for decades. We tackle lightweight appearance capture by training a deep neural network to automatically extract and make sense of these visual cues. Once trained, our network is capable of recovering per-pixel normal, diffuse albedo, specular albedo and specular roughness from a single picture of a flat surface lit by a hand-held flash. We achieve this goal by introducing several innovations on training data acquisition and network design. For training, we leverage a large dataset of artist-created, procedural SVBRDFs which we sample and render under multiple lighting directions. We further amplify the data by material mixing to cover a wide diversity of shading effects, which allows our network to work across many material classes. Motivated by the observation that distant regions of a material sample often offer complementary visual cues, we design a network that combines an encoder-decoder convolutional track for local feature extraction with a fully-connected track for global feature extraction and propagation. Many important material effects are view-dependent, and as such ambiguous when observed in a single image. We tackle this challenge by defining the loss as a differentiable SVBRDF similarity metric that compares the renderings of the predicted maps against renderings of the ground truth from several lighting and viewing directions. Combined together, these novel ingredients bring clear improvement over state of the art methods for single-shot capture of spatially varying BRDFs.

computer science, software engineering
SIHRNet: a fully convolutional network for single image highlight removal with a real-world dataset

Xucheng Wang,Chenning Tao,Xiao Tao,Zhenrong Zheng

DOI: https://doi.org/10.1117/1.JEI.31.3.033013

IF: 0.829

2022-01-01

Journal of Electronic Imaging

Abstract:Specular highlight in images is detrimental to accuracy in object recognition tasks. The prior model-based methods for single image highlight removal (SHIR) are limited in images with large highlight regions or achromatic regions, and recent learning-based methods do not perform well due to lack of proper datasets for training either. A network for SHIR is proposed, which is trained with losses that utilize image intrinsic features and can reconstruct a smooth and natural specular-free image from a single input highlight image. Dichromatic reflection model is used to compute the pseudo specular-free image for providing complementary information for the network. A real-world dataset with highlight images and the corresponding ground-truth specular-free images is collected for network training and quantitative evaluation. The proposed network is validated by comprehensive quantitative experiments and outperforms state-of-the-art highlight removal approaches in structural similarity and peak signal-to-noise ratio. Experimental results also show that the network could improve the recognition performance in applications of computer vision. Our source code is available at https://github.com/coach-wang/SIHRNet. (C) 2022 SPIE and IS&T
SVBRDF Recovery from a Single Image with Highlights Using a Pre-trained Generative Adversarial Network

Tao Wen,Beibei Wang,Lei Zhang,Jie Guo,Nicolas Holzschuch

DOI: https://doi.org/10.1111/cgf.14514

IF: 2.5

2022-01-01

Computer Graphics Forum

Abstract:Spatially varying bi‐directional reflectance distribution functions (SVBRDFs) are crucial for designers to incorporate new materials in virtual scenes, making them look more realistic. Reconstruction of SVBRDFs is a long‐standing problem. Existing methods either rely on an extensive acquisition system or require huge datasets, which are non‐trivial to acquire. We aim to recover SVBRDFs from a single image, without any datasets. A single image contains incomplete information about the SVBRDF, making the reconstruction task highly ill‐posed. It is also difficult to separate between the changes in colour that are caused by the material and those caused by the illumination, without the prior knowledge learned from the dataset. In this paper, we use an unsupervised generative adversarial neural network (GAN) to recover SVBRDFs maps with a single image as input. To better separate the effects due to illumination from the effects due to the material, we add the hypothesis that the material is stationary and introduce a new loss function based on Fourier coefficients to enforce this stationarity. For efficiency, we train the network in two stages: reusing a trained model to initialize the SVBRDFs and fine‐tune it based on the input image. Our method generates high‐quality SVBRDFs maps from a single input photograph, and provides more vivid rendering results compared to the previous work. The two‐stage training boosts runtime performance, making it eight times faster than the previous work.
SVBRDF Reconstruction by Transferring Lighting Knowledge

Pengfei Zhu,Shuichang Lai,Mufan Chen,Jie Guo,Yifan Liu,Yanwen Guo

DOI: https://doi.org/10.1111/cgf.14973

IF: 2.5

2023-01-01

Computer Graphics Forum

Abstract:The problem of reconstructing spatially-varying BRDFs from RGB images has been studied for decades. Researchers found themselves in a dilemma: opting for either higher quality with the inconvenience of camera and light calibration, or greater convenience at the expense of compromised quality without complex setups. We address this challenge by introducing a two-branch network to learn the lighting effects in images. The two branches, referred to as Light-known and Light-aware, diverge in their need for light information. The Light-aware branch is guided by the Light-known branch to acquire the knowledge of discerning light effects and surface reflectance properties, but without the reliance of light positions. Both branches are trained using the synthetic dataset, but during testing on real-world cases without calibration, only the Light-aware branch is activated. To facilitate a more effective utilization of various light conditions, we employ gated recurrent units (GRUs) to fuse the features extracted from different images. The two modules mutually benefit when multiple inputs are provided. We present our reconstructed results on both synthetic and real-world examples, demonstrating high quality while maintaining a lightweight characteristic in comparison to previous methods.
Correlation-aware Encoder-Decoder with Adapters for SVBRDF Acquisition

Di Luo,Hanxiao Sun,Lei Ma,Jian Yang,Beibei Wang

DOI: https://doi.org/10.1145/3680528.3687594

2024-01-01

Abstract:Capturing materials from the real world avoids laborious manual material authoring. However, recovering high-fidelity Spatially Varying Bidirectional Reflectance Distribution Function (SVBRDF) maps from a few captured images is challenging due to its ill-posed nature. Existing approaches have made extensive efforts to alleviate this ambiguity issue by leveraging generative models with latent space optimization or extracting features with variant encoder-decoders. Albeit the rendered images at input views can match input images, the problematic decomposition among maps leads to significant differences when rendered under novel views/lighting. We observe that for human eyes, besides individual images, the correlation (or the highlights variation) among input images also serves as an important hint to recognize the materials of objects. Hence, our key insight is to explicitly model this correlation in the SVBRDF acquisition network. To this end, we propose a correlation-aware encoder-decoder network to model the correlation features among the input images via a graph convolutional network by treating channel features from each image as a graph node. This way, the ambiguity among the maps has been reduced significantly. However, several SVBRDF maps still tend to be over-smooth, leading to a mismatch in the novel-view rendering. The main reason is the uneven update of different maps caused by a single decoder for map interpretation. To address this issue, we further design an adapter-equipped decoder consisting of a main decoder and four tiny per-map adapters, where adapters are employed for individual maps interpretation, together with fine-tuning, to enhance flexibility. As a result, our framework allows the optimization of the latent space with the input image feature embeddings as the initial latent vector and the fine-tuning of per-map adapters. Consequently, our method can outperform existing approaches both visually and quantitatively on synthetic and real data.
An attention-embedded GAN for SVBRDF recovery from a single image

Shi, Zeqi

DOI: https://doi.org/10.1007/s41095-022-0289-1

IF: 4.1268

2023-03-23

Computational Visual Media

Abstract:Learning-based approaches have made substantial progress in capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) from a single image with unknown lighting and geometry. However, most existing networks only consider per-pixel losses which limit their capability to recover local features such as smooth glossy regions. A few generative adversarial networks use multiple discriminators for different parameter maps, increasing network complexity. We present a novel end-to-end generative adversarial network (GAN) to recover appearance from a single picture of a nearly-flat surface lit by flash. We use a single unified adversarial framework for each parameter map. An attention module guides the network to focus on details of the maps. Furthermore, the SVBRDF map loss is combined to prevent paying excess attention to specular highlights. We demonstrate and evaluate our method on both public datasets and real data. Quantitative analysis and visual comparisons indicate that our method achieves better results than the state-of-the-art in most cases.

computer science, software engineering
Joint SVBRDF Recovery and Synthesis from a Single Image Using an Unsupervised Generative Adversarial Network

Yezi Zhao,Beibei Wang,Yanning Xu,Zheng Zeng,Lu Wang,Nicolas Holzschuch

DOI: https://doi.org/10.2312/sr.20201136

2020-01-01

Abstract:We want to recreate spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image. Producing these SVBRDFs from single images will allow designers to incorporate many new materials in their virtual scenes, increasing their realism. A single image contains incomplete information about the SVBRDF, making reconstruction difficult. Existing algorithms can produce high-quality SVBRDFs with single or few input photographs using supervised deep learning. The learning step relies on a huge dataset with both input photographs and the ground truth SVBRDF maps. This is a weakness as ground truth maps are not easy to acquire. For practical use, it is also important to produce large SVBRDF maps. Existing algorithms rely on a separate texture synthesis step to generate these large maps, which leads to the loss of consistency between generated SVBRDF maps. In this paper, we address both issues simultaneously. We present an unsupervised generative adversarial neural network that addresses both SVBRDF capture from a single image and synthesis at the same time. From a low-resolution input image, we generate a large resolution SVBRDF, much larger than the input images. We train a generative adversarial network (GAN) to get SVBRDF maps, which have both a large spatial extent and detailed texels. We employ a two-stream generator that divides the training of maps into two groups (normal and roughness as one, diffuse and specular as the other) to better optimize those four maps. In the end, our method is able to generate high-quality large scale SVBRDF maps from a single input photograph with repetitive structures and provides higher quality rendering results with more details compared to the previous works. Each input for our method requires individual training, which costs about 3 hours. CCS Concepts • Computing methodologies → Texturing;
Flexible SVBRDF Capture with a Multi‐Image Deep Network

Valentin Deschaintre,Miika Aittala,Fredo Durand,George Drettakis,Adrien Bousseau

DOI: https://doi.org/10.1111/cgf.13765

IF: 2.5

2019-07-01

Computer Graphics Forum

Abstract:<p>Empowered by deep learning, recent methods for material capture can estimate a spatially‐varying reflectance from a single photograph. Such lightweight capture is in stark contrast with the tens or hundreds of pictures required by traditional optimization‐based approaches. However, a single image is often simply not enough to observe the rich appearance of real‐world materials. We present a deep‐learning method capable of estimating material appearance from a variable number of uncalibrated and unordered pictures captured with a handheld camera and flash. Thanks to an order‐independent fusing layer, this architecture extracts the most useful information from each picture, while benefiting from strong priors learned from data. The method can handle both view and light direction variation without calibration. We show how our method improves its prediction with the number of input pictures, and reaches high quality reconstructions with as little as 1 to 10 images ‐ a sweet spot between existing single‐image and complex multi‐image approaches.</p>

computer science, software engineering
Multi-view Gradient Consistency for SVBRDF Estimation of Complex Scenes under Natural Illumination

Alen Joy,Charalambos Poullis

DOI: https://doi.org/10.48550/arXiv.2202.13017

2022-02-26

Abstract:This paper presents a process for estimating the spatially varying surface reflectance of complex scenes observed under natural illumination. In contrast to previous methods, our process is not limited to scenes viewed under controlled lighting conditions but can handle complex indoor and outdoor scenes viewed under arbitrary illumination conditions. An end-to-end process uses a model of the scene's geometry and several images capturing the scene's surfaces from arbitrary viewpoints and under various natural illumination conditions. We develop a differentiable path tracer that leverages least-square conformal mapping for handling multiple disjoint objects appearing in the scene. We follow a two-step optimization process and introduce a multi-view gradient consistency loss which results in up to 30-50% improvement in the image reconstruction loss and can further achieve better disentanglement of the diffuse and specular BRDFs compared to other state-of-the-art. We demonstrate the process in real-world indoor and outdoor scenes from images in the wild and show that we can produce realistic renders consistent with actual images using the estimated reflectance properties. Experiments show that our technique produces realistic results for arbitrary outdoor scenes with complex geometry. The source code is publicly available at: <a class="link-external link-https" href="https://gitlab.com/alen.joy/multi-view-gradient-consistency-for-svbrdf-estimation-of-complex-scenes-under-natural-illumination" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics
Single‐Image SVBRDF Estimation with Learned Gradient Descent

X. Luo,L. Scandolo,A. Bousseau,E. Eisemann

DOI: https://doi.org/10.1111/cgf.15018

IF: 2.5

2024-04-25

Computer Graphics Forum

Abstract:Recovering spatially‐varying materials from a single photograph of a surface is inherently ill‐posed, making the direct application of a gradient descent on the reflectance parameters prone to poor minima. Recent methods leverage deep learning either by directly regressing reflectance parameters using feed‐forward neural networks or by learning a latent space of SVBRDFs using encoder‐decoder or generative adversarial networks followed by a gradient‐based optimization in latent space. The former is fast but does not account for the likelihood of the prediction, i.e., how well the resulting reflectance explains the input image. The latter provides a strong prior on the space of spatially‐varying materials, but this prior can hinder the reconstruction of images that are too different from the training data. Our method combines the strengths of both approaches. We optimize reflectance parameters to best reconstruct the input image using a recurrent neural network, which iteratively predicts how to update the reflectance parameters given the gradient of the reconstruction likelihood. By combining a learned prior with a likelihood measure, our approach provides a maximum a posteriori estimate of the SVBRDF. Our evaluation shows that this learned gradient‐descent method achieves state‐of‐the‐art performance for SVBRDF estimation on synthetic and real images.

computer science, software engineering
Two-Shot Spatially-Varying BRDF and Shape Estimation

Mark Boss,Varun Jampani,Kihwan Kim,Hendrik P.A. Lensch,Jan Kautz

DOI: https://doi.org/10.1109/cvpr42600.2020.00404

2020-06-01

Abstract:Capturing the shape and spatially-varying appearance (SVBRDF) of an object from images is a challenging task that has applications in both computer vision and graphics. Traditional optimization-based approaches often need a large number of images taken from multiple views in a controlled environment. Newer deep learning-based approaches require only a few input images, but the reconstruction quality is not on par with optimization techniques. We propose a novel deep learning architecture with a stage-wise estimation of shape and SVBRDF. The earlier predictions guide each estimation, and a joint refinement network later refines both SVBRDF and shape. We follow a practical mobile image capture setting and use unaligned two-shot flash and no-flash images as input. Both our two-shot image capture and network inference can run on mobile hardware. We also create a large-scale synthetic training dataset with domain-randomized geometry and realistic materials. Extensive experiments on both synthetic and real-world datasets show that our networks trained on a synthetic dataset can generalize well to real-world images. Comparisons with recent approaches demonstrate the superior performance of the proposed approach.
Neural BRDFs: Representation and Operations

Jiahui Fan,Beibei Wang,Miloš Hašan,Jian Yang,Ling-Qi Yan

2021-01-01

Abstract:Bidirectional reflectance distribution functions (BRDFs) are pervasively used in computer graphics to produce realistic physically-based appearance. In recent years, several works explored using neural networks to represent BRDFs, taking advantage of neural networks' high compression rate and their ability to fit highly complex functions. However, once represented, the BRDFs will be fixed and therefore lack flexibility to take part in follow-up operations. In this paper, we present a form of "Neural BRDF algebra", and focus on both representation and operations of BRDFs at the same time. We propose a representation neural network to compress BRDFs into latent vectors, which is able to represent BRDFs accurately. We further propose several operations that can be applied solely in the latent space, such as layering and interpolation. Spatial variation is straightforward to achieve by using textures of latent vectors. Furthermore, our representation can be efficiently evaluated and sampled, providing a competitive solution to more expensive Monte Carlo layering approaches.
Guided Fine-Tuning for Large-Scale Material Transfer

Valentin Deschaintre,George Drettakis,Adrien Bousseau

DOI: https://doi.org/10.48550/arXiv.2007.03059

2020-07-06

Graphics

Abstract:We present a method to transfer the appearance of one or a few exemplar SVBRDFs to a target image representing similar materials. Our solution is extremely simple: we fine-tune a deep appearance-capture network on the provided exemplars, such that it learns to extract similar SVBRDF values from the target image. We introduce two novel material capture and design workflows that demonstrate the strength of this simple approach. Our first workflow allows to produce plausible SVBRDFs of large-scale objects from only a few pictures. Specifically, users only need take a single picture of a large surface and a few close-up flash pictures of some of its details. We use existing methods to extract SVBRDF parameters from the close-ups, and our method to transfer these parameters to the entire surface, enabling the lightweight capture of surfaces several meters wide such as murals, floors and furniture. In our second workflow, we provide a powerful way for users to create large SVBRDFs from internet pictures by transferring the appearance of existing, pre-designed SVBRDFs. By selecting different exemplars, users can control the materials assigned to the target image, greatly enhancing the creative possibilities offered by deep appearance capture.
Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image

Zhengqin Li,Kalyan Sunkavalli,Manmohan Chandraker

DOI: https://doi.org/10.48550/arXiv.1804.05790

2018-04-17

Abstract:We propose a material acquisition approach to recover the spatially-varying BRDF and normal map of a near-planar surface from a single image captured by a handheld mobile phone camera. Our method images the surface under arbitrary environment lighting with the flash turned on, thereby avoiding shadows while simultaneously capturing high-frequency specular highlights. We train a CNN to regress an SVBRDF and surface normals from this image. Our network is trained using a large-scale SVBRDF dataset and designed to incorporate physical insights for material estimation, including an in-network rendering layer to model appearance and a material classifier to provide additional supervision during training. We refine the results from the network using a dense CRF module whose terms are designed specifically for our task. The framework is trained end-to-end and produces high quality results for a variety of materials. We provide extensive ablation studies to evaluate our network on both synthetic and real data, while demonstrating significant improvements in comparisons with prior works.

Computer Vision and Pattern Recognition
Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images

Sai Bi,Zexiang Xu,Kalyan Sunkavalli,David Kriegman,Ravi Ramamoorthi

DOI: https://doi.org/10.48550/arXiv.2003.12642

2020-07-04

Abstract:We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object from a sparse set of only six images captured by wide-baseline cameras under collocated point lighting. We first estimate per-view depth maps using a deep multi-view stereo network; these depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture that is trained to pool features from these coarsely aligned images and predict per-view spatially-varying diffuse albedo, surface normals, specular roughness and specular albedo. We do this by jointly optimizing the latent space of our multi-view reflectance network to minimize the photometric error between images rendered with our predictions and the input images. While previous state-of-the-art methods fail on such sparse acquisition setups, we demonstrate, via extensive experiments on synthetic and real data, that our method produces high-quality reconstructions that can be used to render photorealistic images.

Computer Vision and Pattern Recognition,Graphics
Learning optimal lighting patterns for efficient SVBRDF acquisition

Kaizhang Kang,Zimin Chen,Jiaping Wang,Kun Zhou,Hongzhi Wu

DOI: https://doi.org/10.1145/3230744.3230779

2018-01-01

Abstract:Digitally acquiring high-quality material appearance from the real-world is challenging, with applications in visual effects, e-commerce and entertainment. One popular class of existing work is based on hand-derived illumination multiplexing [Ghosh et al. 2009], using hundreds of patterns in the most general case [Chen et al. 2014].
DBFGAN: Dual Branch Feature Guided Aggregation Network for remote sensing image

Shengguang Chu,Peng Li,Min Xia,Haifeng Lin,Ming Qian,Yonghong Zhang

DOI: https://doi.org/10.1016/j.jag.2022.103141

IF: 7.5

2022-12-12

International Journal of Applied Earth Observation and Geoinformation

Abstract:In Remote Sensing(RS) data analysis, Remote Sensing Change Detection(CD) is an important technology. The existing Remote Sensing Change Detection(RS-CD) methods do not fully consider the advantages and disadvantages of Convolution and Transformer in feature extraction, which will restrict the overall performance of the network to a certain extent. Therefore, this paper proposes a Dual Branch Feature Guided Aggregation Network composed of convolutional neural network(CNN) and Transformer. In the encoding stage, based on the respective characteristics of Convolution and Transformer, a dual-branch backbone network is constructed to extract the spatial information and semantic information of the image respectively; And through the Feature Guidance Aggregation Module, the two branches can guide each other for feature mining, so as to avoid the occurrence of false detection and missed detection of change areas due to insufficient fusion to the greatest extent. Finally, in the decoding stage, the different levels of features extracted by the two branches are fully used for fusion and decoding. And the experiment shows that compared with the existing methods, the mean intersection over union(MIoU) index on the four public datasets are improved by 1.25%, 1.55%, 1.38% and 1.71%.

remote sensing
Bifurcated convolutional network for specular highlight removal

Jingting Xu,Sheng Liu,Guanzhou Chen,Qianxi Liu

DOI: https://doi.org/10.1007/s11801-023-3029-6

2023-12-11

Optoelectronics Letters

Abstract:Specular highlight usually causes serious information degradation, which leads to the failure of many computer vision algorithms. We have proposed a novel bifurcated convolution neural network to tackle the problem of high reflectivity image information degradation. A two-stage process is proposed for the extraction and elimination of the specular highlight features, with the procedure starting at a coarse level and progressing towards a finer level, to ensure the generated diffuse images are less affected by visual artifacts and information distortions. A bifurcated feature selection module is designed to remove the specular highlight features, thereby enhancing the detection capability of the network. The experiments on two types of challenging datasets demonstrate that our method outperforms state-of-the-art approaches for specular highlight detection and removal. The effectiveness of the proposed bifurcated feature selection module and the overall network is also verified.

optics
A Multi-Task Network for Joint Specular Highlight Detection and Removal

Gang Fu,Qing Zhang,Lei Zhu,Ping Li,Chunxia Xiao

DOI: https://doi.org/10.1109/cvpr46437.2021.00766

2021-06-01

Abstract:Specular highlight detection and removal are fundamental and challenging tasks. Although recent methods have achieved promising results on the two tasks by training on synthetic training data in a supervised manner, they are typically solely designed for highlight detection or removal, and their performance usually deteriorates significantly on real-world images. In this paper, we present a novel network that aims to detect and remove highlights from natural images. To remove the domain gap between synthetic training samples and real test images, and support the investigation of learning-based approaches, we first introduce a dataset with about 16K real images, each of which has the corresponding ground truths of highlight detection and removal. Using the presented dataset, we develop a multi-task network for joint highlight detection and removal, based on a new specular highlight image formation model. Experiments on the benchmark datasets and our new dataset show that our approach clearly outperforms state-of-the-art methods for both highlight detection and removal.

Highlight-aware two-stream network for single-image SVBRDF acquisition

Ultra-High Resolution SVBRDF Recovery from a Single Image

Single-image SVBRDF capture with a rendering-aware deep network

SIHRNet: a fully convolutional network for single image highlight removal with a real-world dataset

SVBRDF Recovery from a Single Image with Highlights Using a Pre-trained Generative Adversarial Network

SVBRDF Reconstruction by Transferring Lighting Knowledge

Correlation-aware Encoder-Decoder with Adapters for SVBRDF Acquisition

An attention-embedded GAN for SVBRDF recovery from a single image

Joint SVBRDF Recovery and Synthesis from a Single Image Using an Unsupervised Generative Adversarial Network

Flexible SVBRDF Capture with a Multi‐Image Deep Network

Multi-view Gradient Consistency for SVBRDF Estimation of Complex Scenes under Natural Illumination

Single‐Image SVBRDF Estimation with Learned Gradient Descent

Two-Shot Spatially-Varying BRDF and Shape Estimation

Neural BRDFs: Representation and Operations

Guided Fine-Tuning for Large-Scale Material Transfer

Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image

Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images

Learning optimal lighting patterns for efficient SVBRDF acquisition

DBFGAN: Dual Branch Feature Guided Aggregation Network for remote sensing image

Bifurcated convolutional network for specular highlight removal

A Multi-Task Network for Joint Specular Highlight Detection and Removal