Image deblurring via extreme channels prior code analysis (1)

The article "Image deblurring via extreme channels prior" originated from 2017 CVPR, code address: https://sites.google.com/site/renwenqi888/research/deblurring/ecp  

In order to compare with the code, the important knowledge points in the paper are sorted out:

1. The model proposed by the author:

 （10）

D(l) and B(l) represent the dark channel and the bright channel of the latent image l, respectively.

2. Bright channel:

（2）

Ω(x) represents the area centered on x.

 

Dark channel: replace the two max in the above formula with min.

 

3. Familiar scenes, respectively optimize and solve the latent image l and the fuzzy kernel k.

 

（11）

 （12）

4. Solve the L0 regularization of l with half-quadratic splitting:

（13） 

The variables p, q, and g are introduced, which are similar to D(l), 1-B(l), and ▽l, respectively. among them

 

Then alternately solve for l, p, q, g, and fix other variables as fixed values. For example, when solving for l, pq and g are considered to be a fixed value:

（14） 

Compared to the previous formula, μ||g||0, λ||p||0 and η||q||0 are fixed values ​​and have nothing to do with l, so they are removed when solving l. To save trouble, replace 1-B(l) with D(1-l). Because D(·) is non-linear, turn it into a linear operation M, which represents the mapping from pixel to his dark channel value (the pixel at this position is mapped to the lowest pixel value in a certain range centered on this position)

（15） 

 

In the deblurring process, we use the intermediate latent image to calculate M. When the latent image is close to the clear image, Ml is very close to D(l). Use the mapping to represent D(l) and D(1-l), so, (8) is transformed into

(16)

The solution of (16) is:

 

(17)

among them

 

▽h and ▽v represent horizontal and vertical gradients.

Solve for g, p, and q separately:

(18-20) 

The solution of (18) is

(21)

(19-20) the same reason

5. Estimation of fuzzy kernel k: In order to solve it quickly, replace ||l*kb|| with ||▽l*k-▽b||

(22) 

The final solution is:

(23)

6. Preset value: μ=λ=η=0.004, γ=2 The size of the dark channel block is 35 (the current pixel is the center, a block of 35*35 size, that is, 17 rows/columns from top, bottom, left, and right)

7. The overall process

 

The main entrance of the code is demo_deblurring.m. Two test images are attached to the downloaded compressed package. The parameters of real_blur_img3.png and summerhouse.jpg are set to 115-117 and 22-24 respectively. Just cancel the original comments. , The running results of the two images are as follows:

Corresponding to input image (fuzzy image), intermediate latent image, fuzzy core, output image (clear image)

1. Demo_deblurring main function

1.1 First, add the addresses of three folders, which store the methods and input images of others that will be called later:

addpath(genpath('image'));
addpath(genpath('whyte_code'));
addpath(genpath('cho_code'));

After 1.2, four variables of the structure opts are defined:

opts.prescale = 1; %%downsampling 没用到
opts.xk_iter = 5; %% the iterations  4.2中用到，每次计算k和潜在图像的迭代次数
opts.gamma_correct = 1.0;%用于2.1
opts.k_thresh = 20;%用于2.9

They represent (?), the number of iterations is five, the gray scale stretch value is 1 (no gray scale stretch), and the threshold value is 20.

1.3 Next, define the input image name (filename), blur kernel size (kernel_size), oversaturation (saturation), dark channel weight (lambda_dark), gradient weight (lambda_grad), color correction value (gamma_correct), tv Regularization parameter (lambda_tv), l0 regularization parameter (lambda_l0), whether to ring (weight_ring).

filename = 'image\real_blur_img3.png'; opts.kernel_size = 35;  saturation = 0;
lambda_dark = 4e-3; lambda_grad = 4e-3;opts.gamma_correct = 1.0;
lambda_tv = 0.001; lambda_l0 = 5e-4; weight_ring = 1;

1.4 Next, read in the image and mark it as y.

y = imread(filename);

1.5 Convert the image to double format (previously uint8 format) and scale the value to 0~1 . If it is a color image, convert it to grayscale and save it as yg.

if size(y,3)==3
    yg = im2double(rgb2gray(y));
else
    yg = im2double(y);
end

1.6 Call the custom blind_deconv ( 2 ) function, pass in the yg, lambda_dark, lambda_grad, opts parameters, and return the estimated fuzzy kernel kernel and the intermediate latent image interim_latent.

[kernel, interim_latent] = blind_deconv(yg, lambda_dark, lambda_grad, opts);

1.7 Convert y to 0-1 double type:

y = im2double(y);

1.8 If the input is not saturated, call the custom ringing_artifacts_removal ( 12 ) function, pass in y, the previously obtained fuzzy kernel kernel, lambda_tv, lambda_l0, weight_ring and other parameters, and return to get the final clear image Latent; otherwise, call the custom The function whyte_deconv ( due to the length of the space, we will organize this one later ) (under the whyte_code folder), pass in y and the fuzzy kernel kernel, and get a clear image Latent.

if ~saturation
    %% 1. TV-L2 denoising method
    Latent = ringing_artifacts_removal(y, kernel, lambda_tv, lambda_l0, weight_ring);
else
    %% 2. Whyte's deconvolution method (For saturated images)
    Latent = whyte_deconv(y, kernel);
end

1.9 Normalize the fuzzy kernel k, and write the generated fuzzy kernel k, clear image Latent and intermediate latent image interim_latent (1.6 return result) to the specified location respectively.

k = kernel - min(kernel(:));
k = k./max(k(:));
imwrite(k,['results\' filename(7:end-4) '_kernel.png']);
imwrite(Latent,['results\' filename(7:end-4) '_result.png']);
imwrite(interim_latent,['results\' filename(7:end-4) '_interim_result.png']);

2. The custom function blind_deconv called in step 1.6

Function format

function [kernel, interim_latent] = blind_deconv(y, lambda_dark, lambda_grad, opts)

y is a gray-scale fuzzy image, double type, with a value of 0-1, corresponding to the yg variable of (1). This function is used for multi-scale blind convolution.

2.1 If the input opts.gamma_correct is not 1, perform gray scale stretching:

if opts.gamma_correct~=1
    y = y.^opts.gamma_correct;%灰度拉伸
end

If gamma_correct is less than 1, it will stretch to high contrast (brighter); if gamma_correct is greater than 1, it will stretch to low contrast (darker).

2.2 Calculate the number of pyramid layers num_scales

ret = sqrt(0.5);
%%
maxitr=max(floor(log(5/min(opts.kernel_size))/log(ret)),0); %迭代次数 6次
num_scales = maxitr + 1;

2.3 Calculate the fuzzy kernel k1*k2 of each layer and store them in k1list and k2list:

retv represents the zoom factor, the first value is 1, the second value is 1/sqrt(2) times...

retv=ret.^[0:maxitr];
k1list=ceil(opts.kernel_size*retv);%每次迭代估计的模糊核大小
k1list=k1list+(mod(k1list,2)==0);%保证大小为奇数
k2list=ceil(opts.kernel_size*retv);
k2list=k2list+(mod(k2list,2)==0);%k1*k2大小

For example, when the fuzzy kernel size is 35, both k1 and k2 are 35 25 19 13 9 7

2.4 Iteration, from coarse to fine (fuzzy kernel from big to small, pyramid from top to bottom), the fuzzy kernel of each step is initially re-interpolated with the value of the previous fuzzy kernel, if it is the first iteration, it is initialized to one Square matrix ks, the middle two values ​​of the (minisize-1)/2th row are 0.5, and the others are 0, such as a 5*5:

(Why not the middle line?)

If it is not the first time, adjust the current size according to the result of the previous step ks.

for s = num_scales:-1:1
  if (s == num_scales)
      %%
      % at coarsest level, initialize kernel
      ks = init_kernel(k1list(s));%第一层 初始化
      k1 = k1list(s);
      k2 = k1; % always square kernel assumed
  else
    % upsample kernel from previous level to next finer level
    k1 = k1list(s);
    k2 = k1; % always square kernel assumed
    
    % resize kernel from previous level
    ks = resizeKer(ks,1/ret,k1list(s),k2list(s));
    
  end
%for循环还没有结束 直至2.9

Initialize fuzzy kernel function

function [k] = init_kernel(minsize)
  k = zeros(minsize, minsize);
  k((minsize - 1)/2, (minsize - 1)/2:(minsize - 1)/2+1) = 1/2;%最中间两个为0.5 其他为0
end

Resize function, part of the explanation is added in the comment

The purpose of calling fixsize is that if you resize directly, the size will be rounded up according to the multiple, which may be different from the expected size (k1 k2), so adjust it to k1*k2 through fixsize. (If I resize directly to [k1 k2])

function k=resizeKer(k,ret,k1,k2)
%%
% levin's code
k=imresize(k,ret);%放大根2倍，大小向上取整
k=max(k,0);%去掉小于0的值
k=fixsize(k,k1,k2);
if max(k(:))>0
    k=k/sum(k(:));%归一化
end
end
%% 
function nf=fixsize(f,nk1,nk2)
[k1,k2]=size(f);

while((k1~=nk1)|(k2~=nk2))%循环增改模糊核的大小直至k1=nk1 k2=nk2
    
    if (k1>nk1)%如果resize的k的行数比预期多
        s=sum(f,2);%每行的sum求和
        if (s(1)<s(end))
            f=f(2:end,:);%如果第一行的和比最后一行小，那么模糊核删除第一行
        else
            f=f(1:end-1,:);%否则删除最后一行
        end
    end
    
    if (k1<nk1)%如果小于的话则需要增补一行
        s=sum(f,2);
        if (s(1)<s(end))
            tf=zeros(k1+1,size(f,2));
            tf(1:k1,:)=f;%在最后一行后补一行0
            f=tf;
        else
            tf=zeros(k1+1,size(f,2));
            tf(2:k1+1,:)=f;%在第一行前补一行0
            f=tf;
        end
    end
    %列和行同理
    if (k2>nk2)
        s=sum(f,1);
        if (s(1)<s(end))
            f=f(:,2:end);
        else
            f=f(:,1:end-1);
        end
    end
    
    if (k2<nk2)
        s=sum(f,1);
        if (s(1)<s(end))
            tf=zeros(size(f,1),k2+1);
            tf(:,1:k2)=f;
            f=tf;
        else
            tf=zeros(size(f,1),k2+1);
            tf(:,2:k2+1)=f;
            f=tf;
        end
    end
    
    
    
    [k1,k2]=size(f);
    
end

nf=f;
end

2.5 Downsampling the image to get ys:

  cret=retv(s);
  ys=downSmpImC(y,cret);

cret represents the current downsampling multiple.

Downsampling function (the principle is not very clear, so there is no too much explanation):

function sI=downSmpImC(I,ret)
%% refer to Levin's code
if (ret==1) %如果ret为1，结果为本身
    sI=I;
    return
end
%%%%%%%%%%%%%%%%%%%

sig=1/pi*ret;

g0=[-50:50]*2*pi;
sf=exp(-0.5*g0.^2*sig^2);
sf=sf/sum(sf);
csf=cumsum(sf);%csf是sf的累积
csf=min(csf,csf(end:-1:1));%csf和csf反转的最小
ii=find(csf>0.05);

sf=sf(ii);
sum(sf);

I=conv2(sf,sf',I,'valid');%首先求I的各列与向量 sf 的卷积，然后求每行结果与向量 sf' 的卷积。
%valid：舍去需要补零的部分：裁剪为（ma-mb+1）*（na-nb+1） mb*nb为卷积核大小

[gx,gy]=meshgrid([1:1/ret:size(I,2)],[1:1/ret:size(I,1)]);

sI=interp2(I,gx,gy,'bilinear');%重插值
end

2.6 If it is the first iteration, call the custom function threshold_pxpy_v1 ( 3 ), enter the current layer image ys and the size of the blur kernel, and enter the threshold threshold.

  if (s == num_scales)
    [~, ~, threshold]= threshold_pxpy_v1(ys,max(size(ks)));
  end

2.7 Call the custom function blind_deconv_mainBDF ( 4 ), input the current layer image ys, blur kernel ks, lambda_dark and lambda_grad, threshold threshold and opts to get the estimated blur kernel ks, new weights lambda_dark and lambda_grad and potential image interim_latent.

  [ks, lambda_dark, lambda_grad, interim_latent] = blind_deconv_mainBDF(ys, ks, lambda_dark,...
      lambda_grad, threshold, opts);

2.8 Centralize the fuzzy core, call the custom function adjust_psf_center ( 11 ), input the fuzzy core ks, output the adjusted fuzzy core ks, set the value less than 0 to 0, and normalize it.

   ks = adjust_psf_center(ks);
   ks(ks(:)<0) = 0;
   sumk = sum(ks(:));
   ks = ks./sumk;

2.9 If it is the last layer, then ks is the final fuzzy kernel kernel. If opts.k_thresh (the initial default of 1.2) is greater than 0, divide the value of the fuzzy kernel kernel whose value is less than the maximum value by the threshold opts.k_thresh and set the value to 0. For example, the maximum value of kernel is 0.8 and opts.k_thresh is 20. Then set the value of kernel less than 0.04 to 0; otherwise, set the value of less than 0 to 0. Finally, normalize.

  if (s == 1)
    kernel = ks;
    if opts.k_thresh>0
        kernel(kernel(:) < max(kernel(:))/opts.k_thresh) = 0;
    else
        kernel(kernel(:) < 0) = 0;
    end
    kernel = kernel / sum(kernel(:));
  end
end%for迭代结束

At this point, the first for loop in the 2.4 iteration ends

2.10 End of function blind_deconv

end