How to make readonly structs XML serializable?

To satisfy your requirements all you need is:

[Serializable]
[DataContract]
public readonly struct MyStruct {
    [DataMember]
    private readonly double number;

    public MyStruct(double number)
        => this.number = number;
}

Test code:

var target = new MyStruct(2);
// with Data Contract serializer
using (var ms = new MemoryStream()) {
    var s = new DataContractSerializer(typeof(MyStruct));
    s.WriteObject(ms, target);
    ms.Position = 0;
    var back = (MyStruct) s.ReadObject(ms);
    Debug.Assert(target.Equals(back));
}

// with Json.NET
var json = JsonConvert.SerializeObject(target);
var jsonBack = JsonConvert.DeserializeObject<MyStruct>(json);
Debug.Assert(target.Equals(jsonBack));

// with binary formatter
using (var ms = new MemoryStream()) {
    var formatter = new BinaryFormatter();
    formatter.Serialize(ms, target);
    ms.Position = 0;
    var back = (MyStruct) formatter.Deserialize(ms);
    Debug.Assert(target.Equals(back));
}

Update. Since you also need to support XmlSerializer, you can use some unsafe code to achieve your requirements:

[Serializable]    
public readonly struct MyStruct : ISerializable, IXmlSerializable
{        
    private readonly double number;
    public MyStruct(double number)
        => this.number = number;

    private MyStruct(SerializationInfo info, StreamingContext context)
        => this.number = info.GetDouble(nameof(this.number));

    XmlSchema IXmlSerializable.GetSchema() {
        return null;
    }

    unsafe void IXmlSerializable.ReadXml(XmlReader reader) {
        if (reader.Read()) {
            var value = double.Parse(reader.Value, CultureInfo.InvariantCulture);
            fixed (MyStruct* t = &this) {
                *t = new MyStruct(value);
            }
        }
    }

    void IXmlSerializable.WriteXml(XmlWriter writer) {
        writer.WriteString(this.number.ToString(CultureInfo.InvariantCulture));
    }

    public void GetObjectData(SerializationInfo info, StreamingContext context) {
        info.AddValue(nameof(number), this.number);
    }
}

While you can successfully use unsafe, Unsafe.AsRef, or FieldInfo.SetValue to mutate the value in some scenarios, this is technically invalid code and may result in undefined behavior.

From ECMA-335:

[Note: The use of ldflda or ldsflda on an initonly field makes code unverifiable. In unverifiable code, the VES need not check whether initonly fields are mutated outside the constructors. The VES need not report any errors if a method changes the value of a constant. However, such code is not valid. end note]

Likewise from the official API Docs for FieldInfo.SetValue:

This method cannot be used to set values of static, init-only (readonly in C#) fields reliably. In .NET Core 3.0 and later versions, an exception is thrown if you attempt to set a value on a static, init-only field.

The runtime is technically free to make optimizations around initonly fields and currently does in the case of certain static, initonly fields.

You might be interested in the new init only setters feature coming in C# 9 (https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-9#init-only-setters). This provides a valid way to set properties as part of the property initializer syntax and will get the appropriate support/changes to ensure they work successfully and result in valid code.

As a last resort, the readonliness can be "cast away" via Unsafe.AsRef from https://www.nuget.org/packages/System.Runtime.CompilerServices.Unsafe

Assuming you you are ok with limited use of unsafe code, casting away readonliness is a bit nicer than fixed and can work with managed types.

The "almost immutable" struct is a known problem. It is a relatively rare case for which there is no nice and safe solution right now.

Adding a language feature that would allow selectively making only some members of a struct readonly is one of the proposed long-term solutions.