Consider a cutout texture that contains a solid shape surrounded by transparency. Let's say this is a tree, although it could equally well be a cat or an overweight Italian plumber. Our tree is opaque and colored green:
tree = (0, 255, 0, 255)
The surrounding pixels are transparent:
border = (0, 0, 0, 0)
We are drawing over the top of a blue sky:
background = (0, 0, 255)
If we draw without filtering, everything works as expected. Opaque pixels will replace the background, leaving solid green, while the transparent pixels have no effect, leaving solid blue.
What if the texture needs to be filtered? For instance our tree could be positioned in such a way that some destination pixels are covered half by opaque green and half by the transparent border. First the filtering hardware interpolates between these two parts of the texture:
filtered = (tree + border) / 2 = (0, 128, 0, 128)
Now the alpha blending hardware combines this filtered color with our blue background:
result = lerp(background, filtered.rgb, filtered.a) = (0, 64, 128)
Huh? Halfway between a green tree and blue background should be (0, 128, 128), not (0, 64, 128). The output is darker than we wanted.
It seems logical that if a pixel has zero alpha, its RGB value should be irrelevant, right? Not so when filtering is enabled...
Filtering applies equally to the RGB and alpha channels. When used on the alpha channel of a cutout texture it will produce new fractional alpha values around the edge of the shape, which makes things look nice and antialiased. But filtering also produces new RGB colors, part way in between the RGB of the solid and transparent parts of the texture. Thus the RGB values of supposedly transparent pixels can bleed into our final image.
This most often results in dark borders around alpha cutouts, since the RGB of transparent pixels is often black. Depending on the texture, the bleeding could alternatively be white, pink, etc.
What we really wanted was:
filter( blend(tree, background), blend(border, background) )
But instead we got:
blend( filter(tree, border), background )
Because texture filtering and alpha blending are not associative, these do not produce the same result.
What if we change our tree texture to have green in the RGB channels of its transparent areas, replacing (0, 0, 0, 0) with (0, 255, 0, 0)? We will still get color bleeding around the edges, but because the transparent RGB now matches the color of the main image, this will not look so ugly.
This workaround can be useful, but is far from perfect:
If you get bored of having to manually fix-up alpha cutout textures, you could automate it using a custom content processor. This could check your textures as part of the build process, automatically replacing the RGB of transparent pixels with a copy of the RGB from the closest opaque pixel.
The best way to fix this is by using premultiplied alpha. Stay tuned for my next post...