As of now we’ve been passing the normal vectors directly from the vertex shader to the fragment shader. However, the calculations we’ve been doing in the fragment shader are all done in world space coordinates, so shouldn’t we transform the normal vectors to world space coordinates as well? Basically yes, but it’s not as simple as simply multiplying it with a model matrix.
First of all, normal vectors are only direction vectors and do not represent a specific position in space. Also, normal vectors do not have a homogeneous coordinate (the w component of a vertex position). This means that translations do and should not have any effect on the normal vectors. So if we want to multiply the normal vectors with a model matrix we want to remove the translation part of the matrix by taking the upper-left 3x3 matrix of the model matrix (note that we could also set the w component of a normal vector to 0 and multiply with the 4x4 matrix; this removes translation as well). The only transformations we want to apply to normal vectors are scale and rotation transformations.
Second, if the model matrix would perform a non-uniform scale, the vertices would be changed in such a way that the normal vector is not perpendicular to the surface anymore, so we can’t transform the normal vectors with such a model matrix. The following image shows the effect such a model matrix (with non-uniform scaling) has on a normal vector:
Whenever we apply a non-uniform scale (note: uniform scales won’t hurt the normals since their directions do not change, just their magnitude which is easily fixed by normalizing them) the normal vectors are not perpendicular to the corresponding surface anymore which distorts the lighting.
The trick of fixing this behavior is to use a different model matrix specifically tailored for normal vectors. This matrix is called the normal matrix and uses a few linear algebraic operations to remove the effect of wrongly scaling the normal vectors. If you want to know how this matrix is actually calculated I suggest the following article.
The normal matrix is defined as ’the transpose of the inverse of the upper-left corner of the model matrix’.
article :