GAMES101 Notes_Lec05~06_Rasterization

1 Viewport transformation - from clipping space to screen space Canonical Cube to Screen

1.1 What is a screen What is a screen?

• In graphics, the screen is considered abstractly as a two-dimensional array, and each element in the array is a pixel An array of pixels
• Size of the array: resolution
• A typical kind of raster display

Raster == screen in German
Rasterize == drawing onto the screen

1.2 Pixels Pixel (short for “picture element”)

• For now: A pixel is a little square with uniform colo
• The color of a pixel can be represented by three RGB values. Color is a mixture of (red, green, blue)

1.3 Defining the screen space

• The coordinates/positions of the blue pixels in the figure are (2, 1), and the pixels are expressed in the form of (x, y), where x and y are integers. Pixels' indices are in the form of (x, y), where both x and y are integers
• Pixel coordinates/positions in the graph range from (0, 0) to (3, 2), and pixels range from (0, 0) to (width-1, length-1) Pixels' indices are from (0, 0) to (width - 1, height -1)
• The center of the blue pixel in the figure is (2.5, 1.5), and the center of the pixel is at (x+0.5, y+0.5) Pixel (x, y) is centered at (x + 0.5, y + 0.5)
• The coverage range of the screen in the figure is (0, 0) to (4, 3), and the screen coverage range is (0, 0) to (width, height)
  

1.4 Viewport transform Viewport transform

• Ignore the z-axis coordinate transformation
• The original $[-1，1{]}^2$ plane transformations to$[0,width]\times [0，高]$ Transform in xy plane:$[-1，1{]}^2$to$[0，width]\times [0，height]$
• 视口变换矩阵 Viewport transform matrix:
  $$M_{viewport}=\left(\begin{array}{cccc}\frac{width}{2}& 0& 0& \frac{width}{2}\\ 0& \frac{height}{2}& 0& \frac{height}{0}\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right)$$

2 Rasterization - from graphics in screen space to pixels Rasterizing Triangles into Pixels

2.1 Different Raster Displays Different Raster Displays

2.1.1 Cathode Ray Tube

• OscilloscopeOscilloscope
  • Oscilloscope Art https://www.youtube.com/watch?v=rtR63-ecUNo
• Early CRT display Raster Display CRT
  • Interlaced (Interlaced) reduces bandwidth, and now this idea is also used in some video compression algorithms

Frame buffer: the storage area of ​​the picture displayed on the screen Frame Buffer: Memory for a Raster Display

2.1.2 Flat Panel Displays

• LCD/Liquid Crystal DisplayLiquid Crystal Display

  • Blocking or transmitting light by twisting polarization through liquid crystals
    

• LED/Light emitting diode arrayLight emitting diode array

• Electrophoretic (Electronic Ink) Display

  • The picture is formed by controlling the up and down changes of black and white particles, but the disadvantage is that the refresh rate is low
    

2.2 Triangles - Fundamental Shape Primitives

2.2.1 Why triangles Why triangles?

• The most basic polygon Most basic polygon
  • All polygons can be split into triangles Break up other polygons
• Has unique properties Unique properties
  • Unless folded into two triangles, it will always be one surface Guaranteed to be planar
  • The interior and exterior of the triangle are very clear Well-defined interior
  • Well-defined method for interpolating values ​​at vertices over triangle (barycentric interpolation)

2.2.2 Sampling

• The process of finding the corresponding values ​​at different positions on a function is sampling Evaluating a function at a point is sampling.
• Sampling is a process of discretizing a function We can discretize a function by sampling.

for (int x = 0; x < xmax; ++x)
	output[x] = f(x);

• Sampling is a core concept in graphics Sampling is a core idea in graphics.

2.2.3 inside函数 Define Binary Function: inside(tri, x, y)

• Rasterization is the process of sampling a two-dimensional function Rasterization = Sampling A 2D Indicator Function
• Whether a point is inside a triangle can be judged by cross product? Recall: Three Cross Products
• If a point happens to be on the boundary of a triangle, it is either not handled or treated specially. There are generally strict regulations in graphics APIs
  $$inside(t,x,y)=\{\begin{array}{l}1Point(x,y)intrianglet\\ 0Otherwise\end{array}$$

for(int x = 0; x <= xmax; x++)
{
    for(int y = 0; y <= ymax; y++)
    {
        image[x][y] = inside(tri, x + 0.5, y + 0.5);
    }
}

2.2.4 Bounding box optimization Use a Bounding Box

• AABB bounding box Axis-aligned bounding box
  • This bounding box is not optimized for diagonally slender objects
    

The Incremental Triangle Traversal method is more suitable for slender and rotated triangles

2.3 Rasterization on Real Displays

2.3.1 Real LCD Screen Pixels (Closeup)

• Different LCD screens are arranged in different ways. The bayer pattern has more green pixels because the human eye is more sensitive to green.
  

2.3.2 Other Display Methods Aside: What About Other Display Methods?

• The printed color is a subtractive color system. In order to save ink, printing is usually printed in halftone
  

2.4 Sampling Sampling

2.4.1 Sampling is Ubiquitous in Computer Graphics

• Rasterization = Sample 2D Positions Rasterization = Sample 2D Positions
• Photography = sampling information on the photosensitive element (discrete optical information into pixels on the image) Photograph = Sample Image Sensor Plane
• Video = Sample Sample Time in Time Time

2.4.2 Sampling Artifacts (Errors / Mistakes / Inaccuracies) in Computer Graphics

• 锯齿 Jaggies(Staircase Pattern)-sampling in space

  • Since the sampling rate is not high enough for the signal, the signal is aliased and aliased
    

• Moiré Patterns in Imaging-undersampling images

  • The left image skips odd rows and columns to get the right image
    

• Wagon Wheel Illusion (False Motion)-sampling in time

  • Sampling objects that rotate at high speeds may appear visually inverted
    

• Cause of distortion

  • Signal changing too fast and sampling too slow

2.5 Principle analysis of anti-aliasing/aliasing Antialiased

2.5.1 Time domain and frequency domain (supplement) Spatial Domain&Time Domain&Frequency Domain

• The time domain is the real world, the only domain that actually exists. Because our experiences are developed and validated in the time domain, we are used to events happening in chronological order. For example, when we go to a concert to listen to a piece of music, this piece of music will be transmitted to our ears in the form of sound waves. As time goes by, we can fully appreciate this piece of music. This happens in the time domain.

• In the frequency domain , this piece of music is just a score composed of specific notes, which is eternal and has nothing to do with time.
  

• Fourier transform can transform the signal from the time domain to the frequency domain, because from the perspective of the time domain or space domain, the signal is the superposition of a variety of sine waves with different frequencies, and from the perspective of the frequency domain, the signal's The frequency spectrum is so simple, which is convenient for our observation and calculation.

• The method of converting time-domain signals (signals can be periodic and non-periodic signals) into frequency domain and analyzing them is called spectrum analysis . Its purpose is to decompose the complex time-domain waveform into several single harmonic components through some transformation to study, so as to obtain the frequency structure of the signal and the information of each harmonic and phase. This kind of transformation can be Fourier series or Fourier transform. The purpose of both is the same, which is to convert the time domain signal into the frequency domain for signal analysis.

2.5.2 Fourier Transform

• Represent a function as a weighted sum of sines and cosines Represent a function as a weighted sum of sines and cosines
  

$$f(x)=\frac{A}{2}+\frac{2A\_\cos (t\omega }{Pi}-\frac{2A\_\cos (3t\omega )}{3p.m}+\frac{2A\_\cos (5t\omega )}{5p.m}-\frac{2A\_\cos (7t\omega )}{7p.m}+\cdots$$

• Space domain/time domain and frequency domain can be converted to each other by Fourier transform and inverse Fourier transform Fourier Transform Decomposes A Signal Into Frequencies
  

2.5.3 Aliases

• If the sampling frequency is too small compared to the signal frequency, the restored function will be inaccurate. High-frequency signal is insufficiently sampled: reconstruction incorrectly appears to be from a low frequency signal
  

• Two frequencies that are indistinguishable at a given sampling rate are called "aliases"

2.5.4 Filtering (Getting rid of certain frequency contents) Filtering (Getting rid of certain frequency contents)

• The spectrogram of the image after Fourier transform (after centering) Visualizing Image Frequency Content

  • Reference: https://blog.csdn.net/dazhuan0429/article/details/85774692
  • Most of the information is concentrated in the middle (low frequency), and a small amount is distributed around (high frequency)
    

• High-pass filter (to get the border) High-pass filt (Edges)

  • Only high-frequency information can pass through, and then inverse Fourier transform the spectrogram to get the image
    

• Low-pass filter (Blur)

  • Only low frequency information can pass through
    

• Other processing of spectrograms, removing high frequency and low frequency information Filter Out Low and High Frequencies
  

2.5.5 Convolution Convolution

Understanding convolution and image convolution operations in mathematics: https://www.bilibili.com/video/BV1VV411478E?spm_id_from=333.999.0.0

• Simplified understanding of convolution in graphics

  • Perform certain operations on the image through a certain filter
    

• Convolution Theorem

  • The convolution in the time domain is equal to the product in the frequency domain, and the convolution in the frequency domain is equal to the product in the time domain (the two have a dual relationship) Convolution in the spatial domain is equal to multiplication in the frequency domain, and vice versa

• Realize image processing by convolution theorem

  • method one:
    • Filter by convolution in the spatial domain
  • Method Two:
    • Transform the time domain into the frequency domain by Fourier transform Transform to frequency domain (Fourier transform)
    • Multiply by Fourier transform of convolution kernel
    • Convert the frequency domain back to the time domain by inverse Fourier transform Transform back to spatial domain (inverse Fourier)
      

• The small cubes in the image correspond to low-pass filters, and the larger cubes correspond to filters that only allow lower frequencies to pass
  

2.5.6 Reasons for aliasing caused by sparse sampling

• Sampling is to repeat the spectral content of the original signal, that is, sampling in the time domain is equivalent to performing cycle extension in the frequency domain Sampling = Repeating Frequency Contents
  

• From the perspective of frequency, if the sampling is relatively sparse, aliasing will occur when the frequency spectrum of the signal is repeated, and aliasing will occur.
  

2.6 Antialiasing

2.6.1 How to reduce the anti-aliasing phenomenon from the principle analysis How Can We Reduce Aliasing Error?

• Method 1: Increase the sampling rate ncrease sampling rate
  • Essentially increasing the distance between replicas in the Fourier domain when sampling
  • Higher resolution displays, sensors, framebuffers…
  • But: costly & may need very high resolution
• Method 2: Antialiasing
  • Making Fourier contents “narrower” before repeating, ie Filtering out high frequencies before sampling
    

2.6.2 Specific methods of anti-aliasing

• Pre-Filtering Before Sampling

  • The simplest way is to use a pixel-sized low-pass filter to find the average Antialiasing By Averaging Values ​​in Pixel Area
  • When rasterizing a triangle, the average value within a pixel area is equal to the coverage of the triangle in that area
    
    

• An approximate method of measuring coverage by oversampling - MSAA (Multi-Sample Anti-Aliasing)

  • Approximate the effect of the 1-pixel box filter by sampling multiple locations within a pixel and averaging their values ​​by sampling multiple locations within a pixel and averaging their values
  • Step 1: Take NxN samples in each pixel. Average the NxN samples “inside” each pixel.
  • Step 2: Average the NxN samples “inside” each pixel
  • Step 3: Sample the corresponding signal and get the result. This is the corresponding signal emitted by the display
    

2.6.3 Others

• No free lunch！
  • MSAA will increase performance overhead, but in actual calculation, the distribution of points will be more effective, and some points will be multiplexed by surrounding pixels, so it does not double the performance overhead
• Two other important anti-aliasing schemes Milestones (personal idea)
  • FXAA (Fast Approximate AA) - through image matching for jagged images, replace the jagged edges, fast
  • TAA (Temporal AA) - related to time changes, through detection, some results of the previous frame will be multiplexed in the next frame
• Super resolution / super sampling Super resolution / super sampling
  • From low resolution to high resolution
  • Essentially still “not enough samples” problem Essentially still “not enough samples” problem
  • DLSS(Deep Learning Super Sampling)

3 Visibility / occlusion Visibility / occlusion

3.1 Painter's Algorithm

• Draw objects from far to near, first draw the distant objects and then draw the nearby objects to cover the previous objects nspired by how painters paint, paint from back to front, overwrite in the framebuffer
• It is feasible to a certain extent, the time complexity of sorting n triangles is $O(n\log n)$ Requires sorting in depth $O(n\log n)$ for n triangles)
• When there is a mutual occlusion relationship, the depth relationship between objects cannot be defined, and the painter's algorithm cannot solve Can have unresolvable depth order
  

3.2 Depth buffer Z-Buffer

3.2.1 The principle of depth buffer

• Store current min. z-value for each sample (pixel) at the closest distance to the camera
• The final output color information of each pixel is stored as a frame buffer, and the depth information is stored as a depth buffer. Frame buffer stores color values, depth buffer (z-buffer) stores depth ![[Pasted image 20220128172703.png]
  ]

3.2.2 Depth buffer algorithm Z-Buffer Algorithm

Note: For the sake of simplification, the depth value z is always positive, the smaller the closer, the larger the farther
IMPORTANT: For simplicity we suppose z is always positive (smaller z -> closer, larger z -> further)

• Initialize the depth value of each pixel (sampling point) in the depth buffer to infinity. When each triangle is rasterized, compare the depth value of each pixel with the depth buffer. If the depth value of the pixel is less than Depth buffer, then update the depth buffer, otherwise discard
  

for (each triangle T)  
	for (each sample (x,y,z) in T)  
		if (z < zbuffer[x,y])           // closest sample so far  
			framebuffer[x,y] = rgb;     // update color  
			zbuffer[x,y] = z;           // update depth  
		else  
			;                           // do nothing, this sample is occluded

• The time complexity of sorting n triangles is $O\left(n\right)$ $O(n)$ for n triangles (assuming constant coverage)
• The order in which triangles enter the depth buffer is irrelevant to the result
• The depth buffer algorithm is one of the most important algorithms, and it is applied in almost all GPU hardware. Most important visibility algorithm, implemented in hardware for all GPUs
• Z-Buffer cannot handle transparent objects, which require special handling