Picking by Color

Inverse transformations, raycasting, and collision detection involve some heavy mathematics. Sometimes all we want to do is figure out what object was clicked on, an operation called picking. We don't need an exact location of the mouse in 3D space. In such cases, we may be able to take this very different approach to the problem of picking: render each object in a unique color and, on a mouse event, use the color under the mouse to identify the clicked object.

Click on the spheres in this renderer to change which is selected:

This renderer has two different render methods and two different shaders. The default shader shades the spheres with diffuse lighting. The other shader shades the spheres with a unique shade of red. Toggle the checkbox to see this second method. No lighting is applied; the shade of red must be constant across a sphere to serve as an identifier.

On a mouse event, the scene is first rendered in red. WebGL's readPixels method is called in order to get the color of the pixel under the mouse. The pixel's red component is the index of the sphere that was clicked on. The picking logic might look something like this up event listener:

window.addEventListener('pointerup', event => {
  // Use redness under mouse to determine clicked sphere.
  renderRed();
  const pixel = new Uint8Array(4);
  gl.readPixels(event.clientX, canvas.height - event.clientY, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixel);
  pickedIndex = pixel[0];

  render();
});

window.addEventListener('pointerup', event => {
  // Use redness under mouse to determine clicked sphere.
  renderRed();
  const pixel = new Uint8Array(4);
  gl.readPixels(event.clientX, canvas.height - event.clientY, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixel);
  pickedIndex = pixel[0];

  render();
});

The Uint8Array is a type provided by JavaScript for holding raw bytes. This code assumes that there are no more than 256 spheres in the scene. That's the most that can be uniquely identified using only the red byte of the color. If we have more than 256 objects, we'll need to use more than just the red channel.