Deserialize json with known and unknown fields

I thought I'd throw my hat in the ring since I had a similar problem recently. Here's an example of the JSON I wanted to deserialize:

{
    "agencyId": "agency1",
    "overrides": {
        "assumption.discount.rates": "value: 0.07",
        ".plan": {
            "plan1": {
                "assumption.payroll.growth": "value: 0.03",
                "provision.eeContrib.rate": "value: 0.35"
            },
            "plan2": {
                ".classAndTier": {
                    "misc:tier1": {
                        "provision.eeContrib.rate": "value: 0.4"
                    },
                    "misc:tier2": {
                        "provision.eeContrib.rate": "value: 0.375"
                    }
                }
            }
        }
    }
}

This is for a system where overrides apply at different levels and are inherited down the tree. In any case, the data model I wanted was something that would allow me to have a property bag with these special inheritance rules also supplied.

What I ended up with was the following:

public class TestDataModel
{
    public string AgencyId;
    public int Years;
    public PropertyBagModel Overrides;
}

public class ParticipantFilterModel
{
    public string[] ClassAndTier;
    public string[] BargainingUnit;
    public string[] Department;
}

public class PropertyBagModel
{
    [JsonExtensionData]
    private readonly Dictionary<string, JToken> _extensionData = new Dictionary<string, JToken>();

    [JsonIgnore]
    public readonly Dictionary<string, string> Values = new Dictionary<string, string>();

    [JsonProperty(".plan", NullValueHandling = NullValueHandling.Ignore)]
    public Dictionary<string, PropertyBagModel> ByPlan;

    [JsonProperty(".classAndTier", NullValueHandling = NullValueHandling.Ignore)]
    public Dictionary<string, PropertyBagModel> ByClassAndTier;

    [JsonProperty(".bargainingUnit", NullValueHandling = NullValueHandling.Ignore)]
    public Dictionary<string, PropertyBagModel> ByBarginingUnit;

    [OnSerializing]
    private void OnSerializing(StreamingContext context)
    {
        foreach (var kvp in Values)
            _extensionData.Add(kvp.Key, kvp.Value);
    }

    [OnSerialized]
    private void OnSerialized(StreamingContext context)
    {
        _extensionData.Clear();
    }

    [OnDeserialized]
    private void OnDeserialized(StreamingContext context)
    {
        Values.Clear();
        foreach (var kvp in _extensionData.Where(x => x.Value.Type == JTokenType.String))
            Values.Add(kvp.Key, kvp.Value.Value<string>());
        _extensionData.Clear();
    }
}

The basic idea is this:

1. The PropertyBagModel on deserialization by JSON.NET has the ByPlan, ByClassAndTier, etc. fields populated and also has the private _extensionData field populated.
2. Then JSON.NET calls the private OnDeserialized() method and that will move the data from _extensionData to Values as appropriate (or drop it on the floor otherwise - presumably you could log this if it was something you wanted to know). We then remove the extra gunk from _extensionData so it doesn't consume memory.
3. On serialization, the OnSerializing method gets calls where we move stuff into _extensionData so it gets saved.
4. When serialization has finished, OnSerialized gets called and we remove the extra stuff from _extensionData.

We could further delete and recreate the _extensionData Dictionary when needed but I didn't see a real value in this as I'm not using tons of these objects. To do this we'd just create on OnSerializing and delete on OnSerialized. On OnDeserializing, instead of clearing, we could free it.

This is a way you could solve it, although I don't like it that much. I solved it using Newton/JSON.Net. I suppose you could use the JsonConverter for deserialization aswell.

private const string Json = "{\"id\":7908,\"name\":\"product name\",\"_unknown_field_name_1\":\"some value\",\"_unknown_field_name_2\":\"some value\"}";

    [TestMethod]
    public void TestDeserializeUnknownMembers()
    {
        var @object = JObject.Parse(Json);

        var serializer = new Newtonsoft.Json.JsonSerializer();
        serializer.MissingMemberHandling = Newtonsoft.Json.MissingMemberHandling.Error;
        serializer.Error += (sender, eventArgs) =>
            {
                var contract = eventArgs.CurrentObject as Contract ?? new Contract();
                contract.UnknownValues.Add(eventArgs.ErrorContext.Member.ToString(), @object[eventArgs.ErrorContext.Member.ToString()].Value<string>());
                eventArgs.ErrorContext.Handled = true;
            };

        using (MemoryStream memoryStream = new MemoryStream(Encoding.UTF8.GetBytes(Json)))
        using (StreamReader streamReader = new StreamReader(memoryStream))
        using (JsonReader jsonReader = new JsonTextReader(streamReader))
        {
            var result = serializer.Deserialize<Contract>(jsonReader);
            Assert.IsTrue(result.UnknownValues.ContainsKey("_unknown_field_name_1"));
            Assert.IsTrue(result.UnknownValues.ContainsKey("_unknown_field_name_2"));
        }
    }

    [TestMethod]
    public void TestSerializeUnknownMembers()
    {
        var deserializedObject = new Contract
        {
            id = 7908,
            name = "product name",
            UnknownValues = new Dictionary<string, string>
        {
            {"_unknown_field_name_1", "some value"},
            {"_unknown_field_name_2", "some value"}
        }
        };

        var json = JsonConvert.SerializeObject(deserializedObject, new DictionaryConverter());
        Console.WriteLine(Json);
        Console.WriteLine(json);
        Assert.AreEqual(Json, json);
    }
}

class DictionaryConverter : JsonConverter
{
    public DictionaryConverter()
    {

    }

    public override bool CanConvert(Type objectType)
    {
        return objectType == typeof(Contract);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var contract = value as Contract;
        var json = JsonConvert.SerializeObject(value);
        var dictArray = String.Join(",", contract.UnknownValues.Select(pair => "\"" + pair.Key + "\":\"" + pair.Value + "\""));

        json = json.Substring(0, json.Length - 1) + "," + dictArray + "}";
        writer.WriteRaw(json);
    }
}

class Contract
{
    public Contract()
    {
        UnknownValues = new Dictionary<string, string>();
    }

    public int id { get; set; }
    public string name { get; set; }

    [JsonIgnore]
    public Dictionary<string, string> UnknownValues { get; set; }
}
}

An even easier option to tackling this problem would be to use the JsonExtensionDataAttribute from JSON .NET

public class MyClass
{
   // known field
   public decimal TaxRate { get; set; }

   // extra fields
   [JsonExtensionData]
   private IDictionary<string, JToken> _extraStuff;
}

There's a sample of this on the project blog here

UPDATE Please note this requires JSON .NET v5 release 5 and above

See https://gist.github.com/LodewijkSioen/5101814

What you were looking for was a custom JsonConverter