What is the most efficient way to deserialize huge JSON responses into POJOs using Jackson?

Daniel Kesner :

I have a Vert.x/Java web service that makes a REST call to another service and receives a paginated response that looks similar to the following:

{
  "pagination": {
    "count": 500,
    "totalPages": 279,
    "totalResources": 139255,
    "next": "https://some-service.com/some-endpoint?next=(some-hash)"
  },
  "objects": [
   {
     // About 200 fields per "object" mixed in various levels of nesting,
     // only about 10 of which I want to deserialize
     "model": {
       "A field": "A value"
     },
     "topLevelField": "value",
     "someMoreNestedData": {
       // ...
     }
   }
  ]
}

The maximum response size of 500 "objects" is around 4-5MB, or about 100k lines of text. After receiving the response, I want to use Jackson to deserialize the array of objects into a flattened data model that consists of the 10 fields per object that my system cares about, discarding the rest. I also need to record the pagination information in the 'pagination' node.

I've implemented the deserialization using two classes that are very similar to these (Lombok annotations omitted to save space):

@JsonIgnoreProperties(ignoreUnknown = true)
public class RawResponse {
    @JsonIgnore
    private Integer count;
    @JsonIgnore
    private Integer totalPages;
    @JsonIgnore
    private Integer totalResources;
    @JsonIgnore
    private String next;
    @JsonProperty("objects")
    private List<MyCustomObject> products;

    @JsonSetter("pagination")
    public void deserializePaginationNode(Map<String,Object> paginationNode) {
        if (MapUtils.isEmpty(paginationNode)) {
           log.error("Failed to deserialize pagination node: {}", Arrays.toString(paginationNode.entrySet().toArray()));
           return;
        }

        this.count = (Integer) paginationNode.get("count");
        this.totalPages = (Integer) paginationNode.get("totalPages");
        this.totalResources = (Integer) paginationNode.get("totalResources");
        this.next = (String) paginationNode.get("next");
    }

Within MyCustomObject, I use @JsonIgnore annotations in conjunction with @JsonSetter("node name") to force Jackson to use my methods to deserialize:

@JsonIgnoreProperties(ignoreUnknown = true)
public class MyCustomObject {
    @JsonIgnore
    private List<String> someField;
    @JsonIgnore
    private String anotherField;

    // ...

    @JsonSetter("model")
    public void deserializeModelNode(Map<String,Object> modelNode) {
        if (MapUtils.isEmpty(modelOfferingNode)) {
            log.error("Failed to deserialize model node: {}", Arrays.toString(modelOfferingNode.entrySet().toArray()));
            return;
        }

        this.field = (List<String>) modelNode.get("field");
        this.anotherField = (String) modelOfferingNode.get("anotherField");
    }

This approach works, but I'm curious if there's a more efficient implementation that achieves the same result of flattening a complex data structure of ~200 fields at various nested levels into a flat structure using Jackson. For example, I know you can use @JsonDeserialize and write a lower-level deserializer that works with more primitive data types like JsonNode, etc. Does anyone know of a good alternative?

Dulaj Kulathunga :

Parsing Large JSON Files using Jackson Streaming API Example

Jackson's Streaming API. Jackson is one of the most popular JSON processing framework and provides three main model to parse and process JSON data including Streaming API, data binding and tree model. Out of these three, Streaming works at lowest level and can be used to parse huge JSON response upto even giga bytes of size