Best exception for an invalid generic type argument

I would avoid NotSupportedException. This exception is used in the framework where a method is not implemented and there is a property indicating that this type of operation is not supported. It doesn't fit here

• Shameless self Reference: http://blogs.msdn.com/jaredpar/archive/2008/12/12/notimplementedexception-vs-notsupportedexception.aspx

I think InvalidOperationException is the most appropriate exception you could throw here.

NotSupportedException sounds like it plainly fits, but the documentation clearly states that it should be used for a different purpose. From the MSDN class remarks:

There are methods that are not supported in the base class, with the expectation that these methods will be implemented in the derived classes instead. The derived class might implement only a subset of the methods from the base class, and throw NotSupportedException for the unsupported methods.

Of course, there's a way in which NotSupportedException is obviously good enough, especially given its common-sense meaning. Having said that, I'm not sure if it's just right.

Given the purpose of Unconstrained Melody ...

There are various useful things that can be done with generic methods/classes where there's a type constraint of "T : enum" or "T : delegate" - but unfortunately, those are prohibited in C#.

This utility library works around the prohibitions using ildasm/ilasm ...

... it seems like a new Exception might be in order despite the high burden of proof we justly have to meet before creating custom Exceptions. Something like InvalidTypeParameterException might be useful throughout the library (or maybe not - this is surely an edge case, right?).

Will clients need to be able to distinguish this from BCL Exceptions? When might a client accidentally call this using a vanilla enum? How would you answer the questions posed by the accepted answer to What factors should be taken into consideration when writing a custom exception class?

Generic programming should not throw at runtime for invalid type parameters. It should not compile, you should have a compile time enforcement. I don't know what IsFlag<T>() contains, but perhaps you can turn this into a compile time enforcement, like trying to create a type that is only possible to create with 'flags'. Perhaps a traits class can help.

Update

If you must throw, I'd vote for InvalidOperationException. The reasoning is that generic types have parameters and errors related to (method) parameters are centered around the ArgumentException hierarchy. However, the recommendation on ArgumentException states that

if the failure does not involve the arguments themselves, then InvalidOperationException should be used.

There is at least one leap of faith in there, that method parameters recommendations are also to be applied to generic parameters, but there isn't anything better in the SystemException hierachy imho.