Materialist

Physically Based Editing Using Single-Image Inverse Rendering

Lezhong Wang1

Duc Minh Tran1

Ruiqi Cui1

Thomson TG1

Anders Bjorholm Dahl1

Siavash Arjomand Bigdeli1

Jeppe Revall Frisvad1

Manmohan Chandraker2

1Technical University of Denmark  2University of California, San Diego

Paper Code

Overview Video

Abstract

Achieving physically consistent image editing remains a significant challenge in computer vision. Existing image editing methods typically rely on neural networks, which struggle to accurately handle shadows and refractions. Conversely, physics-based inverse rendering often requires multi-view optimization, limiting its practicality in single-image scenarios. In this paper, we propose Materialist, a method combining a learning-based approach with physically based progressive differentiable rendering. Given an image, our method leverages neural networks to predict initial material properties. Progressive differentiable rendering is then used to optimize the environment map and refine the material properties with the goal of closely matching the rendered result to the input image. Our approach enables a range of applications, including material editing, object insertion, and relighting, while also introducing an effective method for editing material transparency without requiring full scene geometry. Furthermore, our envmap estimation method also achieves state-of-the-art performance, further enhancing the accuracy of image editing task. Experiments demonstrate strong performance across synthetic and real-world datasets, excelling even on challenging out-of-domain images.

Transparency Editing

Shadow Effect

Overview

Pipeline

Our inverse rendering pipeline. Given an image, we use MatNet to predict material properties, followed by envmap optimization to estimate the lighting. We then perform material properties optimization, using the envmap that yields the smallest Lre during envmap optimization as the light source. The losses Lre and Lcons guide this process, allowing Lre to be minimized while keeping the results close to the MatNet predictions.

Environment Map Estimation Results

Result 1

Material Editing Results

Result 1

Material editing example

More Results

Result 1

Transparency editing example

Result 1

Object insertion example

Result 3

Relighting and object insertion example

Result 2

Face relight example

Citation

@misc{wang2025materialistphysicallybasedediting,
      title={Materialist: Physically Based Editing Using Single-Image Inverse Rendering}, 
      author={Lezhong Wang and Duc Minh Tran and Ruiqi Cui and Thomson TG and Anders Bjorholm Dahl and Siavash Arjomand Bigdeli and Jeppe Revall Frisvad and Manmohan Chandraker},
      year={2025},
      eprint={2501.03717},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.03717}, 
}