How serializers work

Originally posted to Shawn Hargreaves Blog on MSDN, Monday, August 11, 2008

To understand the differences between the XNA Framework IntermediateSerializer and the standard .NET XmlSerializer, it can be useful to look at how they are implemented.

Fundamentally, all serializers work in a similar way:

• Use the .NET Reflection API to scan over an object
• Find all the fields and properties that need to be serialized
• Get the value of each member
• Write each value to the output XML file

At a basic level, this is easy stuff. You can write a simple serializer in just one page of C# code.

Trouble is, reflection is SLOOOW. If we actually went through all those steps for every object we encountered, it would take hours to serialize a large and complex object!

To speed things up, the Windows implementation of XmlSerializer uses a radically different approach:

• When you create an XmlSerializer, you must tell it what type(s) you intend to serialize
• The XmlSerializer constructor uses reflection to scan over the specified type and find what data it contains
• It generates a C# function that will efficiently serialize that particular type
• It dynamically compiles the C# code into IL instructions
• This means there is no need for the Serialize or Deserialize methods to use reflection at all: they can just call into a highly optimized C# function

Once you understand this implementation detail, you will release some of the behaviors of XmlSerializer are inevitable results of its design:

• XmlSerializer can only serialize public fields and properties, because C# does not allow the generated serializer function to access private fields of some other class
• XmlSerializer cannot handle arbitrary polymorphic types, requiring attributes to predeclare all the possible subtypes, because if it did not know these types ahead of time, it would not be possible to generate a correct C# function to handle them all

When I implemented IntermediateSerializer for the XNA Framework Content Pipeline, I chose a different strategy:

• Lazily initialize serializer state the first time each type is encountered
• Use reflection to scan over the type and find what data it contains
• Use Reflection.Emit to generate dynamic methods for getting and setting member data, and for constructing new instances of types
• Store these generated methods in a table
• Serializing a type is now just a matter of walking through this table and calling the lazily generated helper methods

These implementation choices behave a little differently to XmlSerializer:

• IntermediateSerializer has no problem with arbitrary polymorphic types, and does not require all the possible subclasses to be predeclared, because it can lazily create new helper methods any time it encounters a new type
• Because it generates IL rather than C#, IntermediateSerializer can access internal and private members as well as publics
• As a direct result of the previous point, IntermediateSerializer cannot work in a partial trust environment

So which approach is better? Both work. The XmlSerializer implementation is perhaps slightly faster, but my way is more flexible. I suspect one of the main reasons XmlSerializer works the way it does is that it predates the DynamicMethod class, which was added in the 2.0 CLR. I wonder whether the core framework guys would design XmlSerializer the same way today, if they had access to the more powerful and dynamic reflection features of more recent CLR versions.

What about Xbox? It doesn't have Reflection.Emit or DynamicMethod, which is why IntermediateSerializer can only be used during the Content Pipeline build process, and is not available from your runtime Xbox code. But Xbox doesn't have CSharpCodeProvider, either, so how come XmlSerializer works at runtime?

The answer is that the .NET Compact Framework provides an entirely different implementation of XmlSerializer. This has the same API as the Windows version, and generates the same XML, but it does so using regular reflection calls, which are slow and cause a lot of boxing. This is fine for small tasks like loading a config file or saving a hiscore table, but wouldn't work so well for an entire level. Yet another reason why the Content Pipeline precompiles data into XNB format, and avoids trying to deserialize XML at runtime.