LookAt Matrix

Imagine putting a camera in a virtual world. What information must we provide in order to unambiguously situate it in the world? These two pieces of information probably come to mind first:

1. The location of the camera.
2. The location the camera is looking at.

For example, we might position a player at the top of a hill looking down at the charred ruins of a village. Or at the bottom of a mountain looking up to its clouded peak. Or at a table watching the eyes of a shifty card player from a neighboring planet. But position and direction alone are not quite enough to uniquely specify the view. We don't know if the player's feet are on the ground or if the player is suspended upside down in the talons of an eagle. These two situations lead to very different views of the world. To nail down which view we want, we must provide one additional piece of information:

3. Which way is up.

With these three pieces of information, all of which are in world space coordinates, we can build a matrix that transforms the world into the eye's line of sight. Many graphics libraries call this matrix the look-at matrix and provide a function with this interface to build it:

content_copyfunction lookAt(from, to, worldUp)
  // build a matrix

function lookAt(from, to, worldUp)
  // build a matrix

The parameters correspond to the three pieces of information needed to situate the camera. The from parameter is the eye's position, to is the position being looked at, and worldUp is a normalized vector pointing in the direction that the viewer considers to be up.

The matrix that transforms world space into eye space is composed of two operations: a translation that puts the camera at the origin, and a rotation that swings the focal direction to the negative z-axis. These two matrices combine in pseudocode to form the eyeFromWorld matrix:

content_copyfunction lookAt(from, to, worldUp)
  translater = ...
  rotater = ...
  eyeFromWorld = rotater * translater
  return eyeFromWorld

function lookAt(from, to, worldUp)
  translater = ...
  rotater = ...
  eyeFromWorld = rotater * translater
  return eyeFromWorld

The translation matrix must turn the camera's world space position into \(\begin{bmatrix}0&0&0\end{bmatrix}\), which it does by subtracting away the camera's position:

content_copyfunction lookAt(from, to, worldUp)
  translater = translate(-from.x, -from.y, -from.z);
  // ...

function lookAt(from, to, worldUp)
  translater = translate(-from.x, -from.y, -from.z);
  // ...

The rotation matrix is more involved. Before we can construct the rotation matrix, we must be aware of these helpful and non-obvious properties of all rotation matrices:

• The first row of a rotation matrix corresponds to the axis of the incoming space that will become the x-axis of the outgoing space. For example, if you want the world vector \(\begin{bmatrix}a&b&c\end{bmatrix}\) to become eye vector \(\begin{bmatrix}1&0&0\end{bmatrix}\), then you'd form this rotation matrix:

• The second row of a rotation matrix corresponds to the axis of the incoming space that will become the y-axis of the outgoing space.
• The third row of a rotation matrix corresponds to the axis of the incoming space that will become the z-axis of the outgoing space.
• All rows are pendendicular to each other. Mathematicians call a set of independent perpendicular vectors a basis. Basis vectors form the axes of a coordinate system. This fact is important because if we have two rows of the matrix figured out, we can automatically compute the third.

We derive the rows of the rotation matrix using these facts and the three parameters. Let's start with third row. The world vector that maps to eye vector \(\begin{bmatrix}0&0&1\end{bmatrix}\) is the normalized vector that leads from the object of focus to the camera's position:

content_copynormalize(from - to)

normalize(from - to)

Later on, however, we will need a vector that points from the eye position to the focal position. We might as well compute this inverse now. This vector is the forward vector:

content_copyforward = normalize(to - from)

forward = normalize(to - from)

The forward vector is the camera's focal direction. It is often used to aim projectiles in games, and it becomes the negative z-axis in eye space. Its inverse becomes the positive z-axis and therefore forms the third row of the rotation matrix:

We also need the right vector, which is the world vector that becomes eye vector \(\begin{bmatrix}1&0&0\end{bmatrix}\) and drops into first row of the matrix. This vector aligns with the viewer's outstretched right arm. At first blush, the parameters given to lookAt don't seem to offer much information about this right direction. However, we do know the forward and up directions. The right vector is perpendicular to both of these. If we cross them and normalize the result, we'll have our right vector:

content_copyright = normalize(cross(forward, worldUp))

right = normalize(cross(forward, worldUp))

Our rotation matrix is a little more complete:

The only row missing is the world vector that becomes eye vector \(\begin{bmatrix}0&1&0\end{bmatrix}\). One of the parameters to our lookAt function is the world's up vector, so it must be the one to form the middle row, right? No, not always.

Remember the player standing at the top of the hill? The world's up vector is probably \(\begin{bmatrix}0&1&0\end{bmatrix}\). But if the player is looking down at the village, then the forward vector and up vector are not perpendicular to each other. In rotation matrices, all vectors must be perpendicular. We generally can't use worldUp directly in our matrix.

The up vector that goes in our matrix must be perpendicular to the right and forward vectors we've already computed. Again, we have an operation that can produce perpendicular vectors. We cross the right and forward vectors to get the mathematically correct up vector:

content_copyup = normalize(cross(right, forward))

up = normalize(cross(right, forward))

The camera's up vector forms the second row of our matrix:

This rotation matrix effectively swings the world so that any objects along the focal direction fall onto the negative z-axis of eye space. When combined with the translation matrix described earlier, we have our eyeFromWorld matrix.