How to parse JSON with variable keys in Scala Play?
Update 2016-07-28
The solution described below breaks Referential Transparency because of the use of the return keyword and is not something I would recommend today. Nevertheless, I am not leaving it as it for historical reasons.
Intro
The issue here is that you need to find someplace to save the key for each Size object in the Image object. There are two ways to do this, one is to save it in the Size object itself. This makes sense because the name is intimately related to the Size object, and it is convenient to store it there. So lets explore that solution first.
A quick note on Symmetry
Before we dive into any solutions, let me first introduce the concept of symmetry. This the idea that when you read any Json value, you can use your Scala model representation to go back to exactly the same Json value.
Symmetry when dealing with marshalled data is not strictly required, indeed sometimes it is either not possible, or enforcing it would be too costly without any real gain. But usually it is fairly easy to achieve and it makes working with the serialization implementation much nicer. In many cases it is required as well.
Save name in Size
import play.api.libs.json.Format
import play.api.libs.json.JsPath
import play.api.libs.json.Reads
import play.api.libs.json.JsValue
import play.api.libs.json.JsResult
import play.api.libs.json.JsSuccess
import play.api.libs.json.JsError
import play.api.libs.json.JsObject
import play.api.libs.json.Json
final case class Foo(images: Option[Image])
object Foo {
implicit val fooFormat: Format[Foo] = Json.format[Foo]
}
final case class Image(sizes: Seq[Size])
object Image {
implicit val imagesFormat: Format[Image] =
new Format[Image] {
/** @inheritdoc */
override def reads(json: JsValue): JsResult[Image] = json match {
case j: JsObject => {
JsSuccess(Image(j.fields.map{
case (name, size: JsObject) =>
if(size.keys.size == 3){
val valueMap = size.value
valueMap.get("path").flatMap(_.asOpt[String]).flatMap(
p=> valueMap.get("height").flatMap(_.asOpt[Int]).flatMap(
h => valueMap.get("width").flatMap(_.asOpt[Int]).flatMap(
w => Some(Size(name, p, h, w))
))) match {
case Some(value) => value
case None => return JsError("Invalid input")
}
} else {
return JsError("Invalid keys on object")
}
case _ =>
return JsError("Invalid JSON Type")
}))
}
case _ => JsError("Invalid Image")
}
/** @inheritdoc */
override def writes(o: Image): JsValue = {
JsObject(o.sizes.map((s: Size) =>
(s.name ->
Json.obj(
("path" -> s.path),
("height" -> s.height),
("width" -> s.width)))))
}
}
}
final case class Size(name: String, path: String, height: Int, width: Int)
In this solution Size does not have any Json serialization or deserialization directly, rather it comes as a product of the Image object. This is because, in order to have symmetric serialization of your Image object you need to keep not only the parameters of the Size object, path, height, and width, but also the name of the Size as specified as the keys on the Image object. If you don't store this you can't go back and forth freely.
So this works as we can see below,
scala> import play.api.libs.json.Json
import play.api.libs.json.Json
scala> Json.parse("""
| {
| "large":{
| "path":"http://url.jpg",
| "width":300,
| "height":200
| },
| "medium":{
| "path":"http://url.jpg",
| "width":200,
| "height":133
| }
| }""")
res0: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","width":300,"height":200},"medium":{"path":"http://url.jpg","width":200,"height":133}}
scala> res0.validate[Image]
res1: play.api.libs.json.JsResult[Image] = JsSuccess(Image(ListBuffer(Size(large,http://url.jpg,200,300), Size(medium,http://url.jpg,133,200))),)
scala>
And very importantly it is both safe and symmetric
scala> Json.toJson(res0.validate[Image].get)
res4: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","height":200,"width":300},"medium":{"path":"http://url.jpg","height":133,"width":200}}
scala>
A quick note on Safety
In production code, you never, never, never want to use the .as[T] method on a JsValue. This is because if the data is not what you expected, it blows up without any meaningful error handling. If you must, use .asOpt[T], but a much better choice in general is .validate[T], as this will produce some form of error on failure that you can log and then report back to the user.
Probably a Better Solution
Now, probably a better way to do accomplish this would be to change the Image case class declaration to the following
final case class Image(s: Seq[(String, Size)])
and then keep Size as you originally had it,
final case class Size(path: String, height: Int, width: Int)
Then you merely need to do the following to be safe and symmetric.
If we do this then the implementation becomes much nicer, while still being safe and symmetric.
import play.api.libs.json.Format
import play.api.libs.json.JsPath
import play.api.libs.json.Reads
import play.api.libs.json.JsValue
import play.api.libs.json.JsResult
import play.api.libs.json.JsSuccess
import play.api.libs.json.JsError
import play.api.libs.json.JsObject
import play.api.libs.json.Json
final case class Foo(images: Option[Image])
object Foo {
implicit val fooFormat: Format[Foo] = Json.format[Foo]
}
final case class Image(sizes: Seq[(String, Size)])
object Image {
implicit val imagesFormat: Format[Image] =
new Format[Image] {
/** @inheritdoc */
override def reads(json: JsValue): JsResult[Image] = json match {
case j: JsObject =>
JsSuccess(Image(j.fields.map{
case (name, size) =>
size.validate[Size] match {
case JsSuccess(validSize, _) => (name, validSize)
case e: JsError => return e
}
}))
case _ =>
JsError("Invalid JSON type")
}
/** @inheritdoc */
override def writes(o: Image): JsValue = Json.toJson(o.sizes.toMap)
}
}
final case class Size(path: String, height: Int, width: Int)
object Size {
implicit val sizeFormat: Format[Size] = Json.format[Size]
}
Still works as before
scala> Json.parse("""
| {
| "large":{
| "path":"http://url.jpg",
| "width":300,
| "height":200
| },
| "medium":{
| "path":"http://url.jpg",
| "width":200,
| "height":133}}""")
res1: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","width":300,"height":200},"medium":{"path":"http://url.jpg","width":200,"height":133}}
scala> res1.validate[Image]
res2: play.api.libs.json.JsResult[Image] = JsSuccess(Image(ListBuffer((large,Size(http://url.jpg,200,300)), (medium,Size(http://url.jpg,133,200)))),)
scala> Json.toJson(res1.validate[Image].get)
res3: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","height":200,"width":300},"medium":{"path":"http://url.jpg","height":133,"width":200}}
But with the benefit that Size is now reflective of real Json, that is you can serialize and deserialize just Size values. Which makes it both easier to work with and think about.
TL;DR Commentary on reads in the First Example
Although I would argue that the first solution is somewhat inferior to the second solution, we did use some interesting idioms in the first implementation of reads that are very useful, in a more general sense, but often not well understood. So I wanted to take the time to go through them in more detail for those who are interested. If you already understand the idioms in use, or you just don't care, feel free to skip this discussion.
flatMap chaining
When we attempt to get the values we need out of valueMap, at any an all steps things can go wrong. We would like to handle these cases reasonably without catastrophic exceptions being thrown.
To accomplish this we use the Option value and common flatMap function to chain our computation. There are really two steps we do for each desired value, get the value out of valueMap and we force it to the proper type using the asOpt[T] function. Now the nice thing is that both valueMap.get(s: String) and jsValue.asOpt[T] both return Option values. This means that we can use flatMap to build our final result. flatMap has the nice property that if any of the steps in the flatMap chain fail, i.e. return None, then all other steps are not run and the final result is returned as None.
This idiom is part of general Monadic programming that is common to functional languages, especially Haskell and Scala. In Scala it is not often referred to as Monadic because when the concept was introduced in Haskell it was often explained poorly leading to many people disliking it, despite it in fact being very useful. Due to this, people are often afraid to use the "M word" with respect to Scala.
Functional Short Circuiting
The other idiom that is used in reads, in both versions, is short circuiting a function call by using the return keyword in scala.
As you probably know, use of the return keyword is often discouraged in Scala, as the final value of any function is automatically made into the return value for the function. There is however one very useful time to use the return keyword, that is when you are calling a function that represents a repeated call over something, such as the map function. If you hit some terminal condition on one of the inputs you can use the return keyword to stop the execution of the map call on the remaining elements. It is somewhat analogous to using break in a for loop in languages like Java.
In our case, we wanted to ensure certain things about the elements in the Json, like that it had the correct keys and types, and if at any point any of our assumptions were incorrect, we wanted to return the proper error information. Now we could just map over the fields in the Json, and then inspect the result after the map operation has completed, but consider if someone had sent us very large Json with thousands of keys that did not have the structure we wanted. We would have to apply our function to all of the values even if we knew we had an error after only the first application. Using return we can end the map application as soon we know about an error, without having to spend time apply the map application across the rest of the elements when the result is already known.
Anyway, I hope that little bit of pedantic explanation is helpful!
Suppose you want deserialize into the following case classes:
case class Size(name: String, path: String, width: Int, height: Int)
case class Image(sizes: List[Size])
case class Foo(..., image: Option[Image])
There are many ways to make this work via custom Reads implementations. I'm going to use the reads macro for Size:
implicit val sizeReads = Json.reads[Size]
Then, since the sizes aren't an actual array within the image object, I'll just make them into one to take advantage of the Reads[Size] I already have. I can transform a given JsValue that's being validated as an Image into a JsObject. I can then grab the fields from the JsObject which will be a Seq[(String, JsValue)]. In this case, the String is the size descriptor, and the JsValue is the object containing all the values for that size. I'll just merge them together into a single object, and make a JsArray out of the Seq.
From there, all I need to do is validate that JsArray as a List[Size], and map it to an Image.
implicit val imageReads = new Reads[Image] {
def reads(js: JsValue): JsResult[Image] = {
val fields: Seq[JsValue] = js.as[JsObject].fields.map { case (name, values) =>
Json.obj("name" -> name) ++ values.as[JsObject]
}
JsArray(fields).validate[List[Size]].map(Image(_))
}
}
Then Foo can also use the reads macro.
implicit val fooReads = Json.reads[Foo]
Example:
case class Foo(something: String, image: Option[Image])
val json = Json.parse("""{
"something":"test",
"image":{
"large":{
"path":"http://url.jpg",
"width":300,
"height":200
},
"medium":{
"path":"http://url.jpg",
"width":200,
"height":133
}
}
}""")
scala> json.validate[Foo]
res19: play.api.libs.json.JsResult[Foo] = JsSuccess(Foo(test,Some(Image(List(Size(large,http://url.jpg,300,200), Size(medium,http://url.jpg,200,133))))),)
Implementing a Writes[Image] is a little easier if you take advantage of Json.obj to mimic the structure of the output JSON you want. Since the output JSON doesn't actually use an array, we also need to merge the list of sizes back into a single object, which we can do using foldLeft.
implicit val writes = new Writes[Image] {
def writes(img: Image): JsValue = {
img.sizes.foldLeft(new JsObject(Nil)) { case (obj, size) =>
obj ++ Json.obj(
size.name -> Json.obj(
"path" -> size.path,
"width" -> size.width,
"height" -> size.height
)
)
}
}
}