= Mouse Picking Demystified =
April 5, 2005
== Introduction ==
There comes a time in every 3D game where the user needs to click on something in the scene. Maybe he needs to select a unit in an RTS, or open a door in an RPG, or delete some geometry in a level editing tool. This conceptually simple task is easy to screw up since there are so many little steps that can go wrong.
The problem is this: ''given the mouse's position in window coordinates, how can I determine what object in the scene the user has selected with a mouse click?''
One method is to generate a ray using the mouse's location and then intersect it with the world geometry, finding the object nearest to the viewer. Alternatively we can determine the actual 3-D location that the user has clicked on by sampling the depth buffer (giving us (''x,y,z'') in viewport space) and performing an inverse transformation. Technically there is a third approach, using a selection or object ID buffer, but this has numerous limitations that makes it impractical for widespread use.
This article describes using the ''inverse transformation'' to derive world space coordinates from the mouse's position on screen.
Before we worry about the inverse transformation, we need to establish how the standard forward transformation works in a typical graphics pipeline.
== The View Transformation ==
The standard view transformation pipeline takes a point in model space and transforms it all the way to viewport space (sometimes known as window coordinates) or, for systems without a window system, screen coordinates. It does this by transforming the original point through a series of coordinate systems:
{{{
#!latex-math-hook
\begin{eqnarray*}
&Model \\ &\downarrow \\
&World \\ &\downarrow \\
&View \\ & \downarrow \\
&Clip \\ & \downarrow \\
&Normalized Device\\ &\downarrow \\
&Viewport
\end{eqnarray*}
}}}
Not each step is discrete. OpenGL has the {{{GL_MODELVIEW}}} matrix, '''{{{M}}}''', that transforms a point from model space to view space, using a right-handed coordinate system with +Y up, +X to the right, and -Z into the screen.
{{{
#!latex-math-hook
\begin{eqnarray*}
{\boldsymbol M}{\boldsymbol p}&=&{\boldsymbol v} \\
\text{where} \\
{\boldsymbol M}&=&\text{modelview transformation matrix} \\
\boldsymbol{p} &=& \text{point in model space} \\
\boldsymbol{v} &=& \text{point in view space} \\
\end{eqnarray*}
}}}
Another matrix, {{{GL_PROJECTION}}}, then transforms the point from view space to homogeneous clipping space. Clip space is a right-handed coordinate system (+Z into the screen) contained within a canonical clipping volume extending from {{{(-1,-1,-1)}}} to {{{(+1,+1,+1)}}}:
{{{
#!latex-math-hook
\begin{eqnarray*}
{\boldsymbol P}{\boldsymbol v}&=&{\boldsymbol c} \\
\text{where} \\
{\boldsymbol P}&=&\text{projection matrix} \\
\boldsymbol{v} &=& \text{point in view space} \\
\boldsymbol{c} &=& \text{point in clip space} \\
\end{eqnarray*}
}}}
After clipping is performed the perspective divide transforms the homogeneous coordinate to a Cartesian point in normalized device space. Normalized device coordinates are left-handed, with ''w'' = 1, and are contained within the canonical view frustum from {{{(-1,-1,-1)}}} to {{{(+1,+1,+1)}}}:
{{{
#!latex-math-hook
\begin{eqnarray*}
{\boldsymbol n}&=&\frac{{\boldsymbol c}}{{\boldsymbol c}_w} \\
\text{where} \\
{\boldsymbol n}&=&\text{point in normalized device coordinate} \\
{\boldsymbol c}&=&\text{clipped point in homogeneous clipping space} \\
\end{eqnarray*}
}}}
Finally there is the viewport scale and translation, which transforms the normalized device coordinate into viewport (window) coordinates. Another axis inversion occurs here; this time +Y goes down instead of up (some window systems may place the origin at another location, such as the bottom left of the window, so this isn't always true). Viewport depth values are calculated by rescaling normalized device coordinate Z values from the range {{{(-1,1)}}} to {{{(0,1)}}}, with 0 at the near clip plane and 1 at the far clip plane. Note: any user specified depth bias may impact our calculations later.
{{{
#!latex-math-hook
\begin{eqnarray*}
{\boldsymbol V}{\boldsymbol n}&=&{\boldsymbol w} \\
\text{where} \\
{\boldsymbol V}&=&\text{viewport transformation matrix} \\
{\boldsymbol n}&=&\text{point in normalized device coordinates} \\
{\boldsymbol w}&=&\text{point in viewport/window coordinates} \\
\end{eqnarray*}
}}}
This pipeline allows us to take a model space point, apply a series of transformation, and get a window space point at the end.
Our ultimate goal is to transform the mouse position (in window space) all the way ''back'' to world space. Since we're not rendering a model, model space and and world space are the same thing.
== The Inverse View Transformation ==