SpriteBatch billboards in a 3D world

A while ago I wrote about how to use SpriteBatch with a custom vertex shader, but didn't go into detail about how to set up matrices for drawing sprites in 3D.

The fundamentals are simple:

SpriteBatch generates vertex data containing Vector3 positions (plus texture coordinates and tint colors)
Each sprite is parallel to the XY plane
XY coordinates are computed from the position, rotation, origin and scale SpriteBatch.Draw parameters
The layerDepth SpriteBatch.Draw parameter is stored directly into the Z coordinate
If you use a Draw overload that does not specify a layerDepth, it defaults to zero

You can move sprites to any 3D position by applying the appropriate transforms to this vertex data. If you have a detailed understanding of shader coordinate systems, you now know how to position text in a 3D world or use SpriteBatch for 3D particle systems. If not, keep reading...

The identity transform

Let's make this concrete by working through a real example. Start by downloading the Billboards sample (which I chose because it has a fully movable camera, so we can easily view our 3D sprites from different directions). Add a Sprite Font asset to the content project, and call it "font". Add these fields to the BillboardGame class:

    SpriteBatch spriteBatch;
    SpriteFont spriteFont;
    BasicEffect basicEffect;

Initialize them in the LoadContent method:

    spriteBatch = new SpriteBatch(GraphicsDevice);
    spriteFont = Content.Load<SpriteFont>("font");

    basicEffect = new BasicEffect(GraphicsDevice)
    {
        TextureEnabled = true,
        VertexColorEnabled = true,
    };

At the end of the Draw method (right before the call to base.Draw) we will render some text using SpriteBatch plus BasicEffect:

    spriteBatch.Begin(0, null, null, null, null, basicEffect);
    spriteBatch.DrawString(spriteFont, "hello, world!", Vector2.Zero, Color.White);
    spriteBatch.End();

But when we run the program, no text appears! What gives?

SpriteBatch is generating vertices for a text string with (0,0) as its top left corner, sized so one unit equals one texel of the font texture
In the SpriteBatch coordinate system, X increases to the right and Y increases downward
Since we didn't set the BasicEffect transform matrices, it defaults to identity and passes these positions through unchanged
The output is interpreted in homogenous projection space, which ranges from -1 to 1
So our SpriteBatch vertex positions are way off the edge of the screen!
In homogenous projection space, X increases to the right while Y increases upward
This is the opposite way up to the SpriteBatch coordinate system, so our sprites are inside out and thus get backface culled

We can make the text visible by changing our drawing code to turn off backface culling and scale the text down to a fraction its original size:

    spriteBatch.Begin(0, null, null, null, RasterizerState.CullNone, basicEffect);
    spriteBatch.DrawString(spriteFont, "hello, world!", Vector2.Zero, Color.White, 0, Vector2.Zero, 0.005f, 0, 0);
    spriteBatch.End();

Now the message is visible, but ugly, stretched and upside down. We could fix this by carefully adjusting our scale factor to match the viewport size and aspect ratio, but that is a pain to get right and not my idea of fun. A better option is to change the BasicEffect transform matrices so the BasicEffect vertex shader will automatically apply the necessary coordinate transform.

The default SpriteBatch transform

To make BasicEffect emulate the default SpriteBatch behavior, we need an orthographic projection matrix. This should match the viewport size, and must invert the Y axis to convert SpriteBatch coordinates (where Y increases downward) to homogenous projection space (where Y increases upward):

    Viewport viewport = GraphicsDevice.Viewport;

    basicEffect.Projection = Matrix.CreateOrthographicOffCenter(0, viewport.Width, viewport.Height, 0, 0, 1);

    spriteBatch.Begin(0, null, null, null, null, basicEffect);
    spriteBatch.DrawString(spriteFont, "hello, world!", Vector2.Zero, Color.White);
    spriteBatch.End();

Now we can draw sprites the right way up and with no unwanted stretching, but everything appears blurry. This is because we have not correctly accounted for the texel centering offset, so our texture is unexpectedly filtered. We can fix this by adding a half pixel offset before the orthographic projection:

    Viewport viewport = GraphicsDevice.Viewport;

    basicEffect.Projection = Matrix.CreateTranslation(-0.5f, -0.5f, 0) * 
                             Matrix.CreateOrthographicOffCenter(0, viewport.Width, viewport.Height, 0, 0, 1);

    spriteBatch.Begin(0, null, null, null, null, basicEffect);
    spriteBatch.DrawString(spriteFont, "hello, world!", Vector2.Zero, Color.White);
    spriteBatch.End();

Tada! Even though we are using BasicEffect, everything now looks exactly the same as with the default SpriteBatch vertex shader. A lot of effort to end up exactly back where we started :-)

But wait, there's more...

Drawing sprites in 3D

We can set the BasicEffect projection matrices to anything we like. Try this version:

    Vector3 textPosition = new Vector3(0, 45, 0);

    basicEffect.World = Matrix.CreateScale(1, -1, 1) * Matrix.CreateTranslation(textPosition);
    basicEffect.View = view;
    basicEffect.Projection = projection;

    const string message = "hello, world!";
    Vector2 textOrigin = spriteFont.MeasureString(message) / 2;
    const float textSize = 0.25f;

    spriteBatch.Begin(0, null, null, DepthStencilState.DepthRead, RasterizerState.CullNone, basicEffect);
    spriteBatch.DrawString(spriteFont, message, Vector2.Zero, Color.White, 0, textOrigin, textSize, 0, 0);
    spriteBatch.End();

Notes:

We use the world matrix to invert the Y axis (so text will appear the right way up) and to translate the text to our desired position
We reuse the same view and projection matrices the sample has already computed, so the text appears in the same 3D scene as the rest of the terrain
As we move the camera around, we can view the text from different angles
Because text is alpha blended, we use DepthStencilState.DepthRead and draw it after all opaque geometry
We specify RasterizerState.CullNone so the text will be visible from both sides

Drawing billboard sprites

Sweet, we have text in 3D! But it is fixed in place, with a static location and orientation. If the camera moves to view the text side on, it can no longer be read. If we were displaying something like a floating label over the head of a character, we'd probably want it to rotate and always face the camera. This is easily achieved using Matrix.CreateConstrainedBillboard:

    Vector3 textPosition = new Vector3(0, 45, 0);

    basicEffect.World = Matrix.CreateConstrainedBillboard(textPosition, textPosition - cameraFront, Vector3.Down, null, null);
    basicEffect.View = view;
    basicEffect.Projection = projection;

    const string message = "hello, world!";
    Vector2 textOrigin = spriteFont.MeasureString(message) / 2;
    const float textSize = 0.25f;

    spriteBatch.Begin(0, null, null, DepthStencilState.DepthRead, RasterizerState.CullNone, basicEffect);
    spriteBatch.DrawString(spriteFont, message, Vector2.Zero, Color.White, 0, textOrigin, textSize, 0, 0);
    spriteBatch.End();

Making billboards efficient

Fear not, the end is in sight...

What if we are drawing not just one piece of text, but a particle system containing hundreds or thousands of sprites? We could use the same code shown above, drawing each particle as a separate billboard. But because we are creating a separate billboard matrix for each sprite, which is set onto the BasicEffect, which is then passed to SpriteBatch.Begin, we would have to use a separate SpriteBatch Begin/End block for every single particle! It'll work, but this will not be efficient.

We want to draw all the particles as a single batch, which means we cannot afford to change BasicEffect properties from one sprite to the next. We can still use BasicEffect for the projection matrix, but apply the view matrix transform on the CPU, then pass the resulting view space position to SpriteBatch.Draw, including its Z value as the SpriteBatch layerDepth:

    Matrix invertY = Matrix.CreateScale(1, -1, 1);

    basicEffect.World = invertY;
    basicEffect.View = Matrix.Identity;
    basicEffect.Projection = projection;

    spriteBatch.Begin(0, null, null, DepthStencilState.DepthRead, RasterizerState.CullNone, basicEffect);

    for each billboard sprite
    {
        Vector3 textPosition = new Vector3(0, 45, 0);

        Vector3 viewSpaceTextPosition = Vector3.Transform(textPosition, view * invertY);

        const string message = "hello, world!";
        Vector2 textOrigin = spriteFont.MeasureString(message) / 2;
        const float textSize = 0.25f;

        spriteBatch.DrawString(spriteFont, message, new Vector2(viewSpaceTextPosition.X, viewSpaceTextPosition.Y), Color.White, 0, textOrigin, textSize, 0, viewSpaceTextPosition.Z);
    }

    spriteBatch.End();

This produces the same result as the previous Matrix.CreateConstrainedBillboard example, but will be more efficient if we have many sprites to display.

Hopefully you now understand how any 2D particle system can be extended to draw in 3D, and how SpriteBatch can be a good way to draw 3D particles on Windows Phone.