Group Member:
- Shusen Wu
Introduction :
The idea of intrinsic images was already picked up by Barrow and Tenenbaum in 1978 and is based on the observation that humans are able to derive intrinsic characteristics from images. With intrinsic characteristics, Barrow and Tenenbaum essentially refer to a set of features such as "color, orientation, distance, size, shape, and illumination."
Decomposing an image into its intrinsic components has a wide range of applications in industry. Eliminating the shading component provides illumination-free models that could be used for relighting, retexturing, gray scale colorization, and reflectance editing.
Related Work:
Several of the authors make their source code publicly available:
- Dataset and evaluation code is available online at cs.toronto.edu/~rgrosse/intrinsic/. A script running only the Retinex algorithm can be found in my fork of the original repository by Grosse et al.: github.com/davidstutz/grosse2009-intrinsic-images.
- The approach by Bell et al. is available online at cic.uab.cat/Datasets/synthetic_intrinsic_image_dataset/.
- The dataset is available at opensurfaces.cs.cornell.edu/publications/intrinsic/.
- Gehler et al. provide MatLab code of their algorithm: people.tuebingen.mpg.de/mkiefel/projects/intrinsic/.
- Garces et al. provide their source code at giga.cps.unizar.es/~elenag/projects/EGSR2012_intrinsic/.
- Jon Barron's webpage: cs.berkeley.edu/~barron/.
Approach:
1. In this project, we are going to decompose an input RGB-D image (I) into albedo layer (A) and shading layer (S).
For every pixel p, the decomposition satisfies the equation:
Ip = Ap*Sp
Ap and Sp are performed separately in each color channel.
2. To solve this problem, according to Qifeng Chen's paper, I factorize I into 4 images: an albedo image A, a direct irradiance image D, an indirect irradiance image N, and an illumination color image C. Thus, for every pixel p, the factorization should approximately satisfies:
Ip = Ap*Dp*Np*Cp
While in log domain, the equation changes to:
ip = ap + dp + np + cp
3. We also need a regularization term for every terms so that our model has the adaptive generation ability which is defined as:
Ereg = ΣλiEi
4. Thus, Chen formulate the decomposition as an energy minimization problem:
E(x) = Edata(x) + Ereg(x)
Implementation and Analysis:
1. The traditional approach in intrinsic image decomposition is to reduce the dimensionality of the problem by representing one of the components strictly in terms of the others. From the paper, I learn that Ip = ApSp is not always valid when it is used to media, blur, chromatic distortion, and sensor noise situations. Thus, Chen introduces a soft constraint:
Edata = Σ||lum(Ip)(ip − ap − cp − 1dp − 1np)||2
2. From the paper, I learn that if two points in the scene have similar positions and similar normals, we expect them to have similar irradiance if the contribution of other objects in the scene is not taken into account. This is called direct irradiance. Besides, Chen assumes that the indirect irradiance component is smooth in three-dimensional space. These two situations should contribute to the regularization .
3. This project is based on a linear least squaresformulation and implemented by the function "lsqlin." After that, we extract albedo, direct irradiance, indirect irradiance and illumination color from the result of it.
4. Finially, we need to balance the weight between the albedo and shading images.
5. Results:
(1)
(2)
Conclusions & Discussion:
1. The depth of the RGB-D image makes the intrinsic image decompostion become more tractable and no longer need to use CNN to train the data. For those complex scenes, depth is very useful to estimate different shapes. Besides, since shading arises from the amount of reflected light by the surface, the depth can help us to get the surface orientation.
2. By using the well-known smoothness properties of direct and indirect irradiance, it can get better results in both albedo and shading estimation.
3. Incorrect and missing depth values will limit the performance of the method. To solve this problem, we can use smooth algorithm to preprocess the data or use average values from many depth maps. In Jonathan T. Barron and Jitendra Malik's paper [1], it provides us a good method to smooth the depth from the raw data. While in this project, I use a more simple function because Qifeng Chen's model is designed to be resilient to noisy input.
Reference:
1. A Simple Model for Intrinsic Image Decomposition with Depth Cues
2. Intrinsic Image Decomposition Algorithm: Algorithm and Application.
3. Intrinsic Images – Introduction and Reading List