Operator '|' cannot be applied to operands of type 'System.Enum' and 'System.Enum'

Why is this?

In cases where the compiler knows the enumeration's underlying type, the compiler can perform bitwise operations without any issues. In cases where the compiler doesn't know the underlying type, it cannot know whether you want an 8-bit, a 16-bit, a 32-bit, or perhaps even a 64-bit operation, and just gives up entirely. Note also that the compiler has no way of knowing that neither of your two enumeration values is null, and that the compiler has no way of knowing that the two enumeration values have the same type or even width.

Is there any way around this?

You can know that you will never be dealing with enumerations larger than 64 bits, and that a 64-bit operation will produce the correct results even for 8-bit enumeration types here. Therefore, you can help the compiler by writing your operations as 64-bit operations explicitly.

static Enum Or(Enum a, Enum b)
{
    // consider adding argument validation here

    if (Enum.GetUnderlyingType(a.GetType()) != typeof(ulong))
        return (Enum)Enum.ToObject(a.GetType(), Convert.ToInt64(a) | Convert.ToInt64(b));
    else
        return (Enum)Enum.ToObject(a.GetType(), Convert.ToUInt64(a) | Convert.ToUInt64(b));
}

similarly for And.

Using the accepted answer, I crafted this converter to bind multiple checkboxes to a [Flags]Enum. Note: this converter uses a class member, so do not reuse the same converter instance for multiple sets of bindings.

XAML:

<StackPanel>
    <StackPanel.Resources>
        <local:EnumFlagConverter x:Key="myConverter" />
    </StackPanel.Resources>
    <CheckBox Content="Option1" IsChecked="{Binding TheEnum, Converter={StaticResource myConverter}, ConverterParameter={x:Static local:MyEnum.Option1}}" />
    <CheckBox Content="Option2" IsChecked="{Binding TheEnum, Converter={StaticResource myConverter}, ConverterParameter={x:Static local:MyEnum.Option2}}" />
    <CheckBox Content="Option3" IsChecked="{Binding TheEnum, Converter={StaticResource myConverter}, ConverterParameter={x:Static local:MyEnum.Option3}}" />
</StackPanel>

C#:

public class EnumFlagConverter : IValueConverter
{
    public Enum CurrentValue { get; set; }
    public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
    {
        var theEnum = value as Enum;
        CurrentValue = theEnum;
        return theEnum.HasFlag(parameter as Enum);
    }

    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        var theEnum = parameter as Enum;
        if ((bool)value)
            CurrentValue = CurrentValue.Or(theEnum);
        else
            CurrentValue = CurrentValue.And(theEnum.Not());
        return CurrentValue;
    }
}


public static class Extensions
{
    public static Enum Or(this Enum a, Enum b)
    {
        // consider adding argument validation here
        if (Enum.GetUnderlyingType(a.GetType()) != typeof(ulong))
            return (Enum)Enum.ToObject(a.GetType(), Convert.ToInt64(a) | Convert.ToInt64(b));
        else
            return (Enum)Enum.ToObject(a.GetType(), Convert.ToUInt64(a) | Convert.ToUInt64(b));
    }

    public static Enum And(this Enum a, Enum b)
    {
        // consider adding argument validation here
        if (Enum.GetUnderlyingType(a.GetType()) != typeof(ulong))
            return (Enum)Enum.ToObject(a.GetType(), Convert.ToInt64(a) & Convert.ToInt64(b));
        else
            return (Enum)Enum.ToObject(a.GetType(), Convert.ToUInt64(a) & Convert.ToUInt64(b));
    }
    public static Enum Not(this Enum a)
    {
        // consider adding argument validation here
        return (Enum)Enum.ToObject(a.GetType(), ~Convert.ToInt64(a));
    }
}