Steve Hazen has some great articles about the kind of math used in 3D game programming:
Be sure to follow the links in that second article – looks like he has a lot more content planned too!