Image composition and fusion is really an amazing technology. So I am going to use several articles to do some detailed introduction on this. The picture below is an outline. This article will introduce three techniques: cut-and-paste, alpha blending, multiand blending
1. Direct cut-and-paste technique (cut-and-paste)
Of all related techniques, the most intuitive and simple is cut-and-paste, which is often used by photographers in post-production, and if used properly, it can also create pleasing results.
For example, how was the photo below made?
In fact, it is made by superimposing different parts of the two photos in a certain order. You can take a look at the source image below:
As long as we put the background of the left picture first, then superimpose the portrait of Bill Gates, and finally superimpose the foreground of the left picture, the final image can be synthesized.
The technology of determining the foreground and background from the image is called matting, which corresponds to Image Matting in English, and the process of seamlessly pasting the extracted part into the target image is called image synthesis, which corresponds to Image Compositing in English. Accuracy, I quote the original text of "Computer Vision: Algorithms and Applications" as follows:
If the Cut-and-paste technique is applied properly, it can produce quite artistic images, as shown in the picture below, which should be formed by pasting a lot of texture images on the original characters and dogs.
But many times, this simple technique will only produce frustrating results. For example, if you want to paste two penguins in the image below, it is really a 50 cents special effect.
Since cutting and pasting is too rough and the generated photos are not natural enough, is there a better way? Next, let's talk about a better method: Alpha Blending
2. Alpha Fusion
Alpha fusion is an upgraded version of cut-and-paste, which looks like this if expressed as a formula:
Since the Mask used above is binary, the fused image is very unnatural, as you can see from the fused image
If you do a little feathering on the Mask map, the effect will be much better , see the picture below. There are many ways to feather the Mask image. The most direct method is to perform a certain scale of Gaussian filtering on it. I have already talked about it in the basic image filtering.
Therefore, the effect of Alpha fusion depends on how we correctly set Alpha Mask . If we want to cut out a part of a picture and merge it into another new picture, it mainly involves two steps:
- Accurate matting to obtain Alpha Mask
- Appropriate feathering for Alpha Mask to make the fusion more natural
Accurate calculation of Alpha Mask is often a complicated matter. Let's look at a slightly more complicated example below. Sometimes we want to overlap two photos together to generate a photo, such as the following two photos:
How to integrate them? There are many people who have come up with some clever ways. Now introduce one:
Step 1: Generate a distance transform map of the two images
Each pixel value in this distance transform map represents the distance between the corresponding pixel in the original image and its nearest black pixels (zero pixels). You can do it with the distanceTransform function if you use OpenCV , or the bwdist function if you use Matlab .
Step 2: Calculate Mask
Next, the Alpha fusion method is used to obtain the final photo as follows. You can see that the two photos have been blended together well, and of course you can see some subtle unnaturalness.
3. Multi-band fusion
To make the Alpha fusion result appear natural, the key point is to choose an appropriate fusion window size. I use the following example to illustrate this problem.
Let's blend the following two photos together.
If the central axis of the image is selected as the dividing line between the two images after fusion, the fusion process can be expressed as:
This is actually a kind of Alpha fusion. In fact, the above process is:
in:
Therefore, the performance of the final image depends on the overlapping area of effective pixels of the two images.
So the question is, how big is this overlapping area?
I have the following figure to illustrate the effect of different window sizes:
A rule of thumb is to:
- The size of the fusion window should be equivalent to the size of the largest salient feature in the output image, and the number of frequencies contained in the generated image should be a power of 2 from the spectrum
- The size of the fusion window should be less than 2 times the size of the smallest significant feature, and the maximum frequency <= 2*minimum frequency from the spectrum of the output image
However, we all know that natural images usually contain many different features, and the spectrum span is also very large. At this time, it will be very difficult to select the fusion window. Is there a better way?
This introduces a new fusion method: multi-band fusion . The following diagram illustrates this idea:
So to sum up the process is:
1. For the two photos to be fused, construct a Laplacian pyramid
2. For each layer of the pyramid, apply the following formula for fusion
3. Use the fused pyramid to reconstruct the output image
Let's look at a famous example of how an apple and an orange are fused together through the midline
Using the above process, let's take a look at the integration of each layer of the pyramid:
OK, so if you reconstruct the original image, you can see that the multi-band fusion we are talking about here, that is, the Laplacian pyramid fusion, is actually semi-automatically calculating the best fusion window and weight size, which is essentially a kind of Linear Fusion:
There are also many interesting examples in the book " Computer Vision: Algorithms and Applications ", such as:
In the actual implementation process, in order to save the amount of calculation, it can be considered to use only two layers of pyramids for fusion:
This is very useful when multiple images need to be merged. For example, we need to merge multiple photos into a panoramic image below:
Here I simply put multiple cropped photos together without fusion.
If two layers of pyramid fusion are performed, we can fuse the low-frequency signal and the high-frequency signal separately, and the obtained image is as follows:
From these two layers of images we can reconstruct the final image:
Since two-layer pyramid fusion, or even multi-layer pyramid fusion, takes into account information of different frequencies in the image at the same time, the effect of fusion is usually better than that of single-layer Alpha fusion (linear fusion). Just compare the part of the image with the above fused image, and you can see that the content of the pyramid fusion is clearer:
4. Summary
We saw a lot of examples of image fusion in the beginning, I hope they have managed to catch your attention. Today, I will introduce several basic methods of image synthesis and fusion, including cut-and-paste, alpha fusion, and multi-band fusion . They can basically be summarized as linear fusion between images, among which multi-band fusion is a multi-layer linear fusion. My next lecture will talk about a very effective fusion method: Poisson fusion .
