I have implemented masking in OpenGL according to the following concept:
This should work:
glEnable(GL_BLEND); // Use a simple blendfunc for drawing the background glBlendFunc(GL_ONE, GL_ZERO); // Draw entire background without masking drawQuad(backgroundTexture); // Next, we want a blendfunc that doesn't change the color of any pixels, // but rather replaces the framebuffer alpha values with values based // on the whiteness of the mask. In other words, if a pixel is white in the mask, // then the corresponding framebuffer pixel's alpha will be set to 1. glBlendFuncSeparate(GL_ZERO, GL_ONE, GL_SRC_COLOR, GL_ZERO); // Now "draw" the mask (again, this doesn't produce a visible result, it just // changes the alpha values in the framebuffer) drawQuad(maskTexture); // Finally, we want a blendfunc that makes the foreground visible only in // areas with high alpha. glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA); drawQuad(foregroundTexture);
This is fairly tricky, so tell me if anything is unclear.
Don't forget to request an alpha buffer when creating the GL context. Otherwise it's possible to get a context without an alpha buffer.
Edit: Here, I made an illustration.
Edit: Since writing this answer, I've learned that there are better ways to do this:
The way described in this answer works and is not particularly worse in performance than these 2 better options, but is less elegant and less flexible.