Normal Matrix
Normals are funny. They're vec3's, since you don't want perspective on normals. And they don't actually scale quite right--a 45 degree surface with a 45 degree normal, scaled by glScalef(1,0.1,1), drops the surface down to near 0 degrees, but actually tilts the normal *up*, in the opposite direction from the surface, to near 90 degrees.
Mathematically, if between two points a and b on the surface, dot(n,b-a)==0, then after applying a matrix M to the points, you want the normal to still be perpendicular. The question is, what matrix N do you have to apply to the normal to make this happen? In other words, find N such that
dot( N * n , M * a - M * b) == 0
We can solve this by noting that dot product can be expresed as matrix multiplication--dot(x,y) = transpose(x) * y, where we treat an ordinary column-vector as a little matrix, and flip it horizontally. So
transpose(N * n) * (M*a - M*b) == 0 (as above, but write using transpose and matrix multiplication)
transpose(N * n) * M * (a-b) == 0 (collect both copies of M)
transpose(n) * transpose(N) * M * (a-b) == 0 (transpose-of-product is product-of-transposes in opposite order)
OK. This is really similar to our assumption that the original normal was perpendicular to the surface--that dot(n,b-a) == transpose(n) * (a-b) == 0. In fact, the only difference is the new matrices wedged in the middle. If we pick N to make the term in the middle the identity, then our new normal will be perpendicular to the surface too:
transpose(N) * M == I (the identity matrix)
This is the definition for matrix inverses, so the "normal matrix" N = transpose(inverse(M)).
If you look up the GLSL definition for "gl_NormalMatrix", it's defined as "the transpose of the inverse of the gl_ModelViewMatrix". Now you know why!
http://www.cs.uaf.edu/2007/spring/cs481/lecture/01_23_matrices.html