JSON Serializer – Attributes – Reflection

Jason

You can find a use case here if you do not wish to learn anything but rather just use it.

Here are the tools mentioned in the article:

This is an alpha version of an upcoming free asset. I won’t be explaining in details as it would take too long. Instead, I would go through the main idea and main lines of code. The missing part is most likely a repetition with little alteration or similar logic.
The purpose of this article is not to explain the library from A to but only some aspects and topics being demonstrated in the asset. IF you feel you are being robbed in the process and should be explained how’s and why’s, then I will review…

Why and what is JSON

In an another article, we were dealing with xml serialization, this time we will look at JSON. The principle is quite similar but Unity makes it a slight harder on us this time.

This is going to be long and may be overwhelming. It might be a good idea to stop every now and then and try the code for yourself with plenty of Debug to fully understand what is going on. Also, I shortened it a bit, cropping the redundant parts.

JSON stands for JavaScript Object Notation. Just like xml, it is a basic text file with specific syntax and extension. The file needs to comply with this page and uses the .json extension.
It works with a dictionary principle of key-value pair (KVP). So, you need to know the key, then you can ask if it exists, and you get the value corresponding.

The whole file is one object, in xml it would have a root, json does not. Then comes the first layer of key. Each key is itself an object reference and is always a string. Each key has a value which can be:

  • a simple data (literal, bool or string)
  • an object
  • an array of objects

Here is a simple Json file with only value as data:

{
    "id":1,
    "name": "Adam Sandler",
    "funny": false,
    "goodmovies" : null
}

You can validate your json file with some online validator like jsonlint.

  • file starts and closes with { }
  • a key is always a string
  • key and value are separated with colon
  • coma to separate two data
  • last one has no coma indicating the last item (some parser will not detect that)

Most of all, a key is unique, if you’d have two “name” keys, the system would return one of them arbitrary and your program would not be predictable.

Now here is an example with an object

{
    "id": 1,
    "name": "James Cameron",
    "director": true,
    "movie": {
        "title": "Terminator",
        "year": "1984",
        "confirmedplagiat": true
    }
}

movie is a representation of a movie object. The principle remains the same, “movie” is the key and the value is an object containing KVP. Here is a C# version of the movie object:

public class Movie
{
    public int id;
    public int year; // would require a conversion
    public bool confirmedplagiat;
}

The final case, if you need many movies you need an array.

{
    "id": 1,
    "name": "James Cameron",
    "director": true,
    "movie": [
        {
            "title": "Terminator",
            "status": "plagiat",
            "originalwriter":"Harlan Ellison"
        },
        {
            "title": "Avatar",
            "status": "plagiat",
            "originalstory": "Pocahontas"
        },
        {
            "title": "Terminator 2",
            "status": "sequel"
        },
        {
            "title": "Piranha 2",
            "status": "sequel"
        },
        {
            "title": "Alien return",
            "status": "sequel"
        },
        {
            "title": "True lies",
            "status": "remake"
        },
        {
            "title": "Titanic",
            "status": "basedontruestory"
        },
        {
            "title": "Abyss",
            "status": "original"
        }
    ]
}

The array is contained within [] marking the start and end. Each item is an object of type movie. So our array is of type Movie[]. Notice also that some objects contains extra information. This means the movie object will have to include those or they will be discarded. For instance, removing an item may be done on purpose to make it null or use the default value.

Json is used by most of the famous API, think Twitter, Facebook,…
Here is a Facebook result from the Graph API:

{
  "id": "12345678", 
  "birthday": "1/1/1950", 
  "first_name": "Chris", 
  "gender": "male", 
  "last_name": "Colm", 
  "link": "http://www.facebook.com/12345678", 
  "location": {
    "id": "110843418940484", 
    "name": "Seattle, Washington"
  }, 
  "locale": "en_US", 
  "name": "Chris Colm", 
  "timezone": -8, 
  "updated_time": "2010-01-01T16:40:43+0000", 
  "verified": true
}

Using JSON in Unity

JSON in .NET ships with versions 4x, but Unity charges for a free version of .NET included in Mono which is limited to 3.5 for syntax and 2.5 for .NET (not entirely sure about versions but not the problem here). Just to say, there is no JSON parser in Unity by default. But there are plenty of them out there, MiniJson, SimpleJson and others. They all bring something, some are lightweight, some more advanced, all in all, it is all about what you need. I tend to use the Boomlagoon Json (free on AssetStore), not sure it is the best one but it is just the one I am used to. More OOP approach in the syntax in my opinion.

A parser is a class containing methods that take a text file and convert it into a JSON object. Considering the text complies with json requirements, the parser is able to know when a key starts and ends and where the value is from there. If the json file misses a coma or a colon, then the parser cannot work and returns an error.

The parser is also able to convert a given object to a valid json file using the appropriate methods again.

For instance if we want to get the name of the location in the Facebook file:

private void Start()
{
    string json = File.ReadAllText(path);
    JSONObject jsonObject = JSONObject.Parse(string);
    if(jsonObject == null){return;}// File is most likely not a valid json file
    JSONObject location = jsonObject.GetObject("location");
    if(location == null){return;} // location object is not there or we spelt wrong
    string locationName = location.GetString("name");
    // There we have a string containing the location 
}
// Other common syntax
private void Start()
{
    string json = File.ReadAllText(path);
    JSONObject jsonObject = JSONObject.Parse(string);
    if(jsonObject == null){return;}// File is most likely not a valid json file
    JSONObject location = jsonObject["location"].AsObj;
    if(location == null){return;} // location object is not there or we spelt wrong
    string locationName = location["name"];
    // There we have a string containing the location 
}

This approach is fine but when the file gets bigger or dealing with array, the code becomes quickly difficult to read and debug. Also, you have to make a big deserialization upfront or keep a reference to your JSONObject to be used when needed (and maybe passed around). A serializer would be welcome so that a couple of lines would return a valid C# object to be used. Just like we did in the xml article.

You can use this to convert your json to C#.

[System.Serializable]
public class FBUser
{
  [JSONItem("id", typeof(string))]
  public string id; 
  [JSONItem("id", typeof(string))]
  public string birthday
  // ...
  [JSONItem("location", typeof(Location))]
  public Location location;
  // ...
  [JSONItem("timezone", typeof(int))]
  public int timezone;
  [JSONItem("updated_time", typeof(string))]
  public string updated_time;
  [JSONItem("verified", typeof(bool))]
  public bool verified;
}
[System.Serializable]
public class Location
{
    [JSONItem("id", typeof(string))]
    public string id;
    [JSONItem("name", typeof(string))]
    public string name;
}

void Start()
{
    string json = File.ReadAllText(path);
    FBUser fbUser = JSONSerializer.Deserialize(json, typeof(FBUser));
    if(fbUser == null){return;} // Failed to deserialize
    Debug.Log(fbUser.location.name);
}

Wouldnt this be convenient? No need for a JSON parser to be used anymore by the consuer and one line to rule them all.

But…if Unity does not not ship with a default JSON parser, then it won’t ship with a JSON serializer either. And this is where we need to come up with our own.

.NET contains a JSON serializer that is not available in Unity by default. It uses a data contract attribute system.

Attributes

We shall first define what are the requirements to build a serializer. Xml serializer and our example make use of attributes. But why is that?
First we will consider deconstructing a json file into a C# object, so called deserialization.

Type instance = (Player)JSONSerialize.Deserialize(typeof(Type), jsonFile);

We pass the file and a type and it returns a reference that is cast into the appropriate type. Using generic we could avoid the explicit cast (Same same but different).

We want that with this single line, the code would traverse the object and create matches between C# members and json keys.

This is where attributes will help us greatly. An attribute is a just a basic object. The only difference and requirement to make it an attribute is the base class. Inheriting from System.Attribute allows to use a different syntax.

public class JSONAttribute{} // Basic class no attribute
public class JSONAttribute : System.Attribute{} // Made it an attribute with inheritance
[System.Serializable]
[AttributeUsage(AttributeTargets.All)]
public class JSONAttribute : System.Attribute{}// Addded attribute to it

// Final version 
[System.Serializable]
[AttributeUsage(AttributeTargets.All)]
public class JSONAttribute : System.Attribute
{
    public readonly string data;
    public readonly Type type;
    public JSONAttribute(string data, Type type)
    {
        this.type = type;
        this.data = data;
    }
}

Our class contains two members that we set in the ctor, just like a basic class. The class is even set as abstract because we plan on using inheritance and no instance of JSONAttribute should be used.

Our attributes are meant to represent each of the three types of json data.

public class JSONObjectAttribute : JSONAttribute
{
    public JSONObjectAttribute(string data, Type type) : base(data, type) { }
}

public class JSONItemAttribute : JSONAttribute
{
    public JSONItemAttribute(string data, Type type) : base(data, type) { }
}

public class JSONArrayAttribute : JSONAttribute
{
    public JSONArrayAttribute(string data, Type type) : base(data, type) {  }
}

The attributes given to the parent class are passed down so we do not need to add them.
But then what is the point of an attribute. Let’s use an example to better demonstrate.

public class FBUser
{
    [JSONItem("id", typeof(int))]
    public int id;
    [JSONItem("first_name", typeof(string))]
    public string first_name;
}

Notice that we named JSONItemAttribute but only use JSONItem. The Attribute extension is implied and not necessary. What you see is the object ctor being called and passed information. Those will fill the data in the JSONItem instance that is being created.

An attribute adds meta information to an object, the attribute does not know about the object it decorates. Here we saying that the name item of FBUser is a JSONItem. So it is a integer but also a JSONItem. We gave it an extra type. As well, we made it Serializable (it was already though).

Reading attributes with reflection

Reflection methods will look into the meta data in the manifest of the application, think of it like a text file containing info on items, some kind of glossary of the app. Reflection tends to be slow so it is better to consider this kind of actions at a non-critical moment like the Start.

Let’s have a first look at how to start with our deserializer:

public class JSONSerialize
{
    public static object Deserialize(Type rootType, string json)
    {
        object rootObject = Activator.CreateInstance(rootType);
        JSONObject jsonObject = JSONObject.Parse(json); // parse JSON file into JSONObject
        if (json == null) { throw new Exception("Missing json file"); }
        // Iterate all members of root object
        foreach (MemberInfo member in rootType.GetMembers())
        {
            // Iterate through all fields with JSONAttribute from root object
            JSONAttribute[] attrFields = (JSONAttribute[])member.GetCustomAttributes(typeof(JSONAttribute), true);
            foreach (JSONAttribute attrField in attrFields)
            {
                Type attrFieldType = attrField.type;    // Get the type of the field
                string data = attrField.data;           // Get the data name of the field
                FieldInfo info = rootType.GetField(data); // Create a connection with the field
                if (info == null) { continue; }
             
                // Type is either object, item or array. Could use a int to get faster
                if (attrField is JSONObjectAttribute){ }
                else if (attrField is JSONItemAttribute) { }
                else if (attrField is JSONArrayAttribute){ }
           }
        }
    return rootObject;
    }
}

We shall decompose all that because there is quite a bit of new and interesting stuff.

object rootObject = Activator.CreateInstance(rootType);
JSONObject jsonObject = JSONObject.Parse(json);

First line create an instance of the passed type. In our example, we would pass FBUser type and this would create an empty/default instance object of type FBUser. So rootObject is an object reference pointing to an object of type FBUser.

The second line parses the json file.

foreach (MemberInfo member in rootType.GetMembers()){}

If you went through the FSM articles, you saw GetMethod, this is the same principle for members. In this case, it returns all the members of the root type and now we can see if they contain our attributes.

JSONAttribute[] attrFields = (JSONAttribute[])member.GetCustomAttributes(typeof(JSONAttribute), true);
foreach (JSONAttribute attrField in attrFields){}

Since all attributes inherit from JSONAttribute, we can look for that type, and it will return an array. Then we can iterate through our array. At that point, we have connections with all the field of that object that bear an attribute, indirectly, we can access the objects/data themselves.

string data = attrField.data;           // Get the data name of the field
FieldInfo info = rootType.GetField(data); // Create a connection with the field
if (info == null) { continue; }

First we get the info from the attributes, those are the ones we passed in the constructors. Then more reflection, we look for a field that is named like the attribute content and finally if the object does not have the member, we continue and get to the next one.

Serializer/Deserializer

We now have the back bone of our deserializer, the serializer will work in a similar fashion but backwards. Now we need to get and assign data to the C# object.

First, we consider the three different types of attributes, again using reflection:

 if (attrField is JSONItemAttribute)
{
}
else if (attrField is JSONObjectAttribute)
{
}
else if (attrField is JSONArrayAttribute)
{              
}

This allows us to perform different actions if we set the member to be an item, an object or an array. One way to slightly speed this up could be to store a integer in the attribute. Object gets 0, item gets 1 and array gets 2. We can then use that value instead of the is keyword which requires a cast since internally the compiler attempts a cast which returns true/false based on success/failure.
It would also allows to use a switch since references don’t work in switches. In this case, the last case would not require the check of all previous to get there, switch jumps there directly. I leave it up to you.

Objects

Let’s get rid of the second case since it happens to be the most simple one.
Basically, if we are dealing with an object type member, it means we want to set that member target with an object that should be deserialized. Mmmm sounds like a Inception concept where one object is actually inside an another. Well, yes, and this is called recursion. We will recursively called the deserialization process until the bottom object only contains item.

JSONObject childJsonObject = jsonObject.GetObject(data);
string jsonFile = childJsonObject.ToString();
object obj = Deserialize(attrFieldType, jsonFile);
info.SetValue(rootObject, obj);

This is exactly what we started with. The last method will set the target value via reflection.

Items

Items is slight bit longer but not that much trickier. Same process without recursion. Main problem is that we need to check for the type so that we can get the object appropriately.

if (attrFieldType == typeof(int))
{
    JSONValue value = jsonObject.GetValue(data);
    if (value != null) { info.SetValue(rootObject, (int)value.Number); }
    string valueStr = jsonObject.GetString(data);
    if (valueStr != null)
    {
        int intResult = 0;
        if (int.TryParse(valueStr, out intResult) == true)
        {
            info.SetValue(rootObject, intResult);
        }
    }
}

That does for integers and same works for floats with different cast. Strings and booleans get the same treatment with less lines. Since only the process is important, I won’t paste the other cases.

We get the value from json, and if not null then we pass it to the member target. If it failed, it may be that your integer was stored as a string. So, same again with string this time. Based on that, you should understand what goes for the other types.

Arrays

Last type is array, the process is somehow similar to object, only do we need to create new objects in a for loop to get all items in the array.

JSONArray jsonArray = jsonObject.GetArray(data);
if (jsonArray == null) 
{
    continue;
}
object o = Array.CreateInstance(attrFieldType, jsonArray.Length);
SetJSONArray(attrFieldType, jsonArray, o);
info.SetValue(rootObject, o);

Always the same, looking in json, then creating the object, and set the member target. Let’s look at the methods doing it all.

private static void SetJSONArray(Type attrFieldType, JSONArray jsonArray, object o)
{
    int length = jsonArray.Length;
    object[] objs = (object[])o;
    for (int i = 0; i < length; i++)
    {
        JSONObject obj = jsonArray[i].Obj;
        objs[i] = Deserialize(attrFieldType, obj.ToString());
    }
}

We get passed the type of array, looking back at our movie example, we would have an array of Movie object (Movie []), we get the JSON file for the content and an object reference to our array we created. The o reference is a pointer to a chunk of memory but it is not a Movie [] as it is. With the cast, we get to used a reference to it as an array.
Keep in mind, the framework does not know that we are dealing with Movie so we need to keep it neutral. Then we get all item from the json array, one by one, and deserialize again. Finally, we return so it can be added to the target of the parent object.

Quick summary

So the principle is simple, you just need to know what tools to use. The same process is repeated, get from json, check what we are dealing with, if item, assign the value and next. If object, create it and start the process again to fill all items. If array, create the array and start the process again to fill all items of the array.

Serialization

For the final part, I will fly over the topic as it is quite the same as Deserializing but the other way around.

public static void Serialize(object obj, JSONObject jsonObject)
{
    Type rootType = obj.GetType();
    foreach (var member in rootType.GetMembers())
    {
        // Iterate through all fields with JSONAttribute from root object
        JSONAttribute[] attrFields = (JSONAttribute[])member.GetCustomAttributes(typeof(JSONAttribute), true);
        foreach (JSONAttribute attrField in attrFields)
        {
            string data = attrField.data;           // Get the data name of the field
            FieldInfo info = rootType.GetField(data); // Create a connection with the field
            if (info == null)                   // If no field next (probably wrong design of the class)
            {
                continue;
            }
            Type attrFieldType = attrField.type;    // Get the type of the field  
            Type attrType = attrField.GetType();    // Get the type of the attribute
            // Type is either object, item or array.
            if (attrType == typeof(JSONObjectAttribute))
            {
                object val = info.GetValue(obj);
                JSONObject jsonChild = new JSONObject();
                jsonObject.Add(data, jsonChild);
                Serialize( info.GetValue(obj), jsonChild);
            }
            else if (attrType == typeof(JSONItemAttribute)){}
            else if (attrType == typeof(JSONArrayAttribute)){}
        }
    }
}

I put only part of the code here but you should see that the patterns are here again. We pass an object and a JSONObject. The process is inverted, we want to add information to the JSONObject instead of extracting. We first get the type we are dealing with and look for all members that contain a JSONAttribute. The rest is similar or understandable.

object val = info.GetValue(obj);
JSONObject jsonChild = new JSONObject();
jsonObject.Add(data, jsonChild);
Serialize( info.GetValue(obj), jsonChild);

This time, we get the value from the object, that is the actual value we want to store. For instance, we mistakenly added a funny movie and modified the Adam Sandler object, we now want to save that new value.
The new value is in val reference (info.GetValue(obj) gets the actual value stored in the variable), we then create a new JSONObject, use the Add method from the JSON framework and finally serialize that again in order to get all item saved as well.

Once we are done for all types and objects we still need to write that down somewhere.

The method shown above could be called from inside another method.

public static void Serialize(string path, object obj)
{
    using (TextWriter text = new StreamWriter(path))
    {
        JSONObject jsonObject = new JSONObject();
        Serialize( obj, jsonObject);
        text.Write(jsonObject.ToString());
    }
}

This is an overload of the Serialize method. The user calls it and passes a path to save the file and the object. The framework goes on to create a StreamWriter used to physically write the content and the JSONObject. Those are passed to the Serialize method (the one seen before), and once everything is done, the json file returns a string that is saved on disk by the StreamWriter.

You can find a use case here.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s