Specifying DataTemplate.DataType with a custom type extension

I really got into your question, and here is what I found.

Q: Why only those three (String, TypeExtension, and StaticExtension) are allowed?

A: By design. If you could write any custom markup extension to be used as a key in a dictionary, what side effects would this introduce? Consider you have Binding as a value of DataType... I'm pretty sure you can add dozen issues related to dictionary keys dynamic nature.

Q: What is special about the processing of the XAML at that point?

A. At that point you have BAML creation. The problem comes from internal class BamlRecordWriter, but the message doesn't describe actual problem. When you specify custom markup extension as DataType, it takes a DataTemplate's child, and checks it if it's assignable from string, TypeExtension or StaticExtension (see BamlRecordWriter.WriteElementStart() function). Indeed. Not your extension (which is assignable to TypeExtension), but first child (which is not assignable). Now you have this strange "cannot be of type" thing. Although it looks like a BamlRecordWriter's bug, I think they left it intentionally. Until it doesn't let you use custom markup extension as a DataType value, who cares about error message?

Q: Is there another way to accomplish this (data template selection based on types that may be nullable) without resorting to a DataTemplateSelector?

A: Yes, kind of. First of all you can have standard TypeExtension do all the dirty job for you:

<DataTemplate DataType="{x:Type TypeName=System:Nullable`1[[System.Int32]]}">
</DataTemplate>

But in most of the cases (if not all the time) you will not see the templating results. Why? Now it comes to boxing rules for nullable types. Boxing a non-null nullable value type boxes the value type itself, not the System.Nullable that wraps the value type. Thus default template selector will look for DataTemplate with DataType of T not of Nullable<T>.

I may not understand exact problem you are trying to solve with nullable extension, but you may want to wrap nullables into your own ref type, write one DataTemplate for the wrapper and use DataTemplate.Triggers, to choose content appearance. Well, this looks like reinvented data template selector :)...

NB: I'm not a MS guy, and my findings are based on Reflector and my own experience (which is not as big as I would like it to be ). In any case, hope I could help :).

Cheers

The syntax

DataType="{x:Type TypeName=System:Nullable`1[[System.Int32]]}">

doesn't seem to work for user defined types :(

Actually one other way is to create a base non-generic type. Set first data template to that type and bind ContentPresenter.Content to the property which holds object of T. Then create other data templates for whatever T.