Why are flexible types not allowed in record type definitions?

You pretty much answered your own question here... The second option works, because you introduce another type argument, i.e. you abstract over 'actions. The point being, that you need cannot really define generic values in a record definition. Consider, that the second option is not generic with respect to types, as 'state and 'actions are defined.