Back when multisampling first showed up, GPU hardware was slow, so triangle counts were low. Complex curved shapes were approximated by a small number of faceted triangles, each of which was big enough to cover many screen pixels.
This created an odd situation where early articles about multisampling focused on how it could smooth jaggies along the straight edges of these enormous triangles. We’d become so used to the faceted appearance of computer graphics that we could geek out over how beautifully antialiased our triangle edges were, without ever stopping to notice “whoah, that dude’s entire head is made of just 5 triangles!”
But hardware improved. Triangle counts went up. A high end modern game typically uses somewhere between 4000 and 6000 triangles for a main character.
Consider a typical gameplay situation:
With this much detail in our models, faceted geometry is a thing of the past. Instead we have a new problem: we just got Nyquisted!
Remember: to avoid aliasing we must take at least twice as many output samples as our highest frequency input signal. But here we have FEWER output pixels than input triangles. It is intuitively obvious that this must cause aliasing. With fewer pixels than triangles, some triangles will inevitably not appear in the output image at all. As the object moves, which triangles are lucky enough to get themselves a pixel will randomly vary, so the image shimmers with aliasing as individual triangles pop in and out of view.
In fact there is a narrow sweet spot for how many triangles a model should contain. For best quality, you want each triangle to cover exactly two screen pixels. If triangles are smaller than this, you cross the Nyquist threshold and get aliasing. If larger, your silhouette will appear polygonal and faceted.
It is not entirely intuitive that the sweet spot is two pixels per triangle as opposed to just one, yet this is true. Consider a circle being approximated as a series of straight line segments. As you increase the number of segments, the circle becomes more perfectly round. By the time each line segment reaches two pixels in length, the circle is perfect. Adding more line segments beyond this point will not give any improvement to the curvature of the shape.
It is obviously impractical to keep all our triangles exactly the same size on screen, given that models must be drawn at different sizes as they or the player move around the world. So how can we draw high detail geometry without aliasing?
Multisampling (or supersampling if you can afford it) helps by giving more headroom before we hit the Nyquist threshold, but this alone is not enough to avoid all geometry aliasing.
Normalmaps can be a powerful technique. We have great features for avoiding aliasing when resampling textures, so any time we can take fine detail out of our geometry and replace it with a normalmap, that will help control aliasing. We often think of normalmaps as purely a performance optimization (replacing expensive geometry with a cheap texture map) but they can also boost visual quality (replacing a data representation that cannot be easily filtered with one that supports anisotropic filtering and mipmaps).
Finally, it is important to consider level-of-detail model variants. When the object is far away, replacing that 6000 triangle model with a simpler 1000 or 500 triangle alternative will not only boost perf, but also reduce geometry aliasing artifacts.
Moral: when it comes to triangle counts, more is not always better! Beware of trying to draw more triangles than you have screen pixels to represent. That way lieth the land of aliasing.