StreamingJsonReader

Introduction

This article describes the StreamingJsonReader class, which is used to read JSON data. It allows callbacks to be attached to logical paths of data in a JSON payload, which means that data can be extracted while the JSON is being parsed.

This is particularly useful when extracting a small set of data from a large JSON structure. For example, when pulling BrandSet data out of a large JSON payload that is primarily intended to be consumed by the UI code. This avoids the need to parse the entire payload into memory and then crawl the graph to extract the data.

This page discusses how to use this class.

Details

This section explains how the StreamingJsonReader class.

Code

There are four public methods, shown below. Two give you the ability to define callback methods that implment the StreamingJsonReaderCallback interface, while two are used to read in JSON data, either from a file or an input stream.

API: StreamingJsonReader class

public class StreamingJsonReader {

    /**
     * Adds a new callback to the specified JSON path. This is called every time the
     * parser finds a token that matches the path. The JSON parser will be pointing
     * at the matching token.
     */
    public void addCallback(String path, StreamingJsonReaderCallback callback) {
    }

    /**
     * Adds a new callback to the specified JSON path. This is called every time the
     * parser finds a token that matches the path. The JSON parser will be pointing
     * to the *value* token *after* the matching token. Do not use this for paths that
     * don't reference a value, as it can cause tokens to be skipped.
     */
    public void addValueCallback(String path, StreamingJsonReaderCallback callback) {
    }

    /** Reads the given JSON file. */
    public void read(File file)  {
    }

    /** Reads the givenJSON input stream. */
    public void read(InputStream stream) {
    }
}

Paths

This section describes a JSON "path". A path is simply a text string that informs the parser which node it is interested in.

Let’s examine a typical JSON file and identify its paths.

Examining paths in a JSON file; this is a small section of a typical "recipes" file

                                    // Path to given node:
[                                   // "["
  {                                 // "[{" (1)
    "recipeName": "Lemonade",       // "[{recipeName"
    "maxFlowRate": 80,              // "[{maxFlowRate"
    "recipeId": 21020,              // "[{recipeId"
    "ingredientsMix": [             // "[{ingredientsMix["
      {                             // "[{ingredientsMix[{"
        "partId": 1048708,          // "[{ingredientsMix[{partId"
        "volVolPercent": 0.1        // "[{ingredientsMix[{volVolPercent"
      },                            // "[{ingredientsMix[}"
      {                             // "[{ingredientsMix[{" (next 4 lines are duplicate paths)
        "partId": 1049088,          // "[{ingredientsMix[{partId"
        "volVolPercent": 0.662      // "[{ingredientsMix[{volVolPercent"
      }                             // "[{ingredientsMix[}"
    ],                              // "[{ingredientsMix[]"
    "preferredVersion": true,       // "[{preferredVersion"
    "caloricValuePerOz": 13.18379,  // "[{caloricValuePerOz"
    "recipeVersion": 5              // "[{recipeVersion"
  },                                // "[}" (2)
  {                                 // "[{"
    "recipeName": "Limeade",        // etc.
    "maxFlowRate": 82,
    "recipeId": 1903,
    "ingredientsMix": [
      {
        "partId": 1048600,
        "volVolPercent": 99.007
      },
      {
        "partId": 1048900,
        "volVolPercent": 0.662
      },
      {
        "partId": 1048901,
        "volVolPercent": 0.331
      }
    ],
    "preferredVersion": false,
    "caloricValuePerOz": 0.39822,
    "recipeVersion": 3
  }
]

1	The start of object in top-level array.
2	The end of object in top-level array.

Note that there are two drinks in this recipe data: Lemonade and Limeaid. How would we read various pieces of this?

To get a callback when a recipe starts, the logical path is [{.
To get a callback with every recipe ID, the path is [{recipeId.
To get a callback with every ingredient part ID, the path is [{ingredientsMix[{partId.
To get a callback when a recipe ends, the logical path is [}.

As you can see, the path just describes the set of array [], object {}, and data names that allow you to arrive at a given piece of data.

Value vs. Regular callbacks

Using the JSON data from above, let’s examine the difference between the two callback methods.

Comparing addCallback() and addValueCallback() methods

public class MyClass {

    private String fieldName;
    private String fieldData;

    public MyClass() {
        // When this method is called, fieldName = "recipeId":
        addCallback("[{recipeId", parser -> fieldName = parser.getValueAsString());

        // When this method is called, fieldValue = "21020":
        addValueCallback("[{recipeId", parser -> fieldData = parser.getValueAsString());
    }
}

So you can see how addCallback() gives you the name of node, while addValueCallback() gives you the value associated with that node.

Typically, but not always, addCallback() is used to mark the beginning and ending of blocks, while addValueCallback() is used to retrieve actual data.

The following three examples should make this clear, and will also show how to use the read() methods.

Example #1 (Simple)

In this first example, we’ll read a variety of simple data values from a JSON file.

Real-life JSON files could be massive

Keep in mind that real JSON data files could be thousands of lines long and contain all manner of data types, objects, arrays, maps, etc. This is one benefit of using our streaming JSON parser: you don’t care about anything except your desired data.

JSON Data #1

There are six types of basic JSON data that we’ll process:

integer
string
double
boolean
long
nullable (all are nullable, this is merely a demonstration of such)

File "streaming-json-reader-example1.json"

{
  "integer": 11,
  "string": "Jack",
  "double": 22.33,
  "boolean": true,
  "long": 1234567890987654321,
  "nullable": null
}

Java Data Bean #1

This simple data bean holds our data.

Data bean for our results

@Data
public class Results {
    private int intValue;
    private String stringValue;
    private double doubleValue;
    private boolean boolValue;
    private long longValue;
    private String nullableValue = "erasable";
}

Streaming JSON Reader #1

This is the streaming reader that extracts the data from the JSON file. Notice that we extend the StreamingJsonReader class.

Streaming reader that parses the data fields

public class MyStreamingReader1 extends StreamingJsonReader {

    @Getter
    private Results results;

    MyStreamingReader1() {
        // (1)
        // Generates a callback when the top-level object is opened,
        // therefore create a Results object:
        addCallback("{", parser -> results = new Results());

        // (2)
        // Generates a callback when the top-level object is closed,
        // therefore store this Results object:
        addCallback("}", parser -> done());

        // (3)
        //@formatter:off
        addValueCallback("{integer",  parser -> results.setIntValue     (parser.getValueAsInt()));
        addValueCallback("{string",   parser -> results.setStringValue  (parser.getValueAsString()));
        addValueCallback("{double",   parser -> results.setDoubleValue  (parser.getValueAsDouble()));
        addValueCallback("{boolean",  parser -> results.setBoolValue    (parser.getValueAsBoolean()));
        addValueCallback("{long",     parser -> results.setLongValue    (parser.getValueAsLong()));
        addValueCallback("{nullable", parser -> results.setNullableValue(parser.getValueAsString()));
        //@formatter:on
    }

    public void done() {
    }
}

Let’s look at the code:

(1) The parser detects the top-level object, and therefore instantiates a Results object
(2) The parser detects the end of the top-level object, and we can do whatever
(3) We extract all of the fields

Usage #1

The following code snippet shows how to use your streaming reader.

How to use our streaming reader

public class MyClass {

    private static final String FILE = "streaming-json-reader-example1.json";

    @SneakyThrows
    void soSomething() {

        // Instantiate the reader:
        MyStreamingReader1 myStreamingReader1 = new MyStreamingReader1();

        // Read the JSON data:
        myStreamingReader1.read(getClass().getClassLoader().getResourceAsStream(FILE));

        // Retrieve the results:
        Results results = myStreamingReader1.getResults();
    }
}

Example #2 (Intermediate)

In this second example, we’ll read an array of beverages from a JSON file. Also included is a sub-object "name" that contains two fields, "full" and "abbr".

JSON Data #2

This is the JSON data we’ll process. Notice that there are two beverages, "Coke" and "Diet Coke".

File "streaming-json-reader-example2.json"

[
  {
    "id": 123,
    "name": {
      "full": "Coke",
      "abbr": "C"
    }
  },
  {
    "id": 456,
    "name": {
      "full": "Diet Coke",
      "abbr": "DC"
    }
  }
]

Java Data Bean #2

This simple data bean holds information about each beverage. We’ll store each of these into a list.

Data bean that holds beverage information

@Data
public class Beverage {
    private int id;
    private String name;
    private String abbr;
}

Streaming JSON Reader #2

This is the streaming reader that extracts the data from the JSON file.

Streaming reader that parses array of items

public class MyStreamingReader2 extends StreamingJsonReader {

    @Getter
    private final List<Beverage> beverages = new ArrayList<>();  // results stored here
    private Beverage beverage;

    public MyStreamingReader2() {
        // (1)
        // "[{" generates a callback every time an object in the top level list is opened,
        // therefore create a Beverage object:
        addCallback("[{", parser -> beverage = new Beverage());

        // (2)
        // "[}" generates a callback every time an object in the top level list is closed,
        // therefore store this Beverage into our list of beverages:
        addCallback("[}", parser -> beverages.add(beverage));

        // (3)
        // Retrieve "id":
        addValueCallback("[{id", parser -> beverage.setId(parser.getValueAsInt()));

        // (4)
        // Retrieve "name.full":
        addValueCallback("[{name{full", parser -> beverage.setName(parser.getValueAsString()));

        // (5)
        // Retrieve "name.abbr":
        addValueCallback("[{name{abbr", parser -> beverage.setAbbr(parser.getValueAsString()));
    }
}

Let’s look at the code:

(1) The parser detects a new beverage object, therefore we instantiate a Beverage
(2) The parset detects the end of this beverage’s data, therefore we add it to our list of beverages
(3) We extract the id field
(4) We extract the name.full field
(5) We extract the name.abbr field

Usage #2

The following code snippet, similar to the first, demonstrates how to use the streaming reader.

Using the streaming reader

public class MyClass {

    private static final String FILE = "streaming-json-reader-example2.json";

    @SneakyThrows
    void doSomething() {

        // Instantiate the reader:
        MyStreamingReader2 myStreamingReader2 = new MyStreamingReader2();

        // Read the JSON data:
        myStreamingReader2.read(getClass().getClassLoader().getResourceAsStream(FILE));

        // Retrieve the results:
        List<Beverage> beverages = myStreamingReader.getBeverages();
    }
}

Example #3 (Advanced)

For this third example, we’re going to read in some more interesting data types:

list of simple data types ("list-of-numbers")
list of simple objects ("list-of-letters")
list of compound objects ("list-of-people")
map of simple key/value pairs ("map-of-key-value-pairs")
map with complex value types ("map-of-state-capitals")

JSON Data #3

Here’s the JSON data for the five advanced data types.

File "streaming-json-reader-example3.json"

[
  {
    "list-of-numbers": [
      101,
      102,
      103
    ],
    "list-of-letters": [
      {
        "letter": "A"
      },
      {
        "letter": "B"
      }
    ],
    "list-of-people": [
      {
        "first-name": "Fred",
        "last-name": "Flintstone",
        "age": 44
      },
      {
        "first-name": "Betty",
        "last-name": "Rubble",
        "age": 39
      }
    ],
    "map-of-key-value-pairs": {
      "key1": "value1",
      "key2": "value2",
      "key3": "value3"
    },
    "map-of-state-capitals": {
      "MO": {
        "name": "Missouri",
        "capital": "Jefferson City"
      },
      "IL": {
        "name": "Illinois",
        "capital": "Springfield"
      }
    }
  }
]

Java Data Beans #3

These Java beans hold the results.

Java bean that holds all results

@Data
public class MyData {

    private List<Integer> listOfNumbers = new ArrayList<>();
    private List<String> listOfLetters = new ArrayList<>();
    private List<Person> listOfPeople = new ArrayList<>();

    private Map<String, String> mapOfKeyValuePairs = new HashMap<>();
    private Map<String, State> mapOfStateCapitals = new HashMap<>();
}

Java bean that holds "person" data

@Data
@NoArgsConstructor
@AllArgsConstructor
public class Person {
    private String firstName;
    private String lastName;
    private Integer age;
}

Java bean that holds "state" data

@Data
@NoArgsConstructor
@AllArgsConstructor
public class State {
    private String name;
    private String capital;
}

Streaming JSON Reader #3

The following streaming reader configures the parsing statements that handles these advanced data types.

Advanced JSON parser

public class MyStreamingReader3 extends StreamingJsonReader {

    @Getter
    private MyData myData;
    private Person person;
    private String key;
    private String value;
    private String stateKey;
    private String stateName;
    private String stateCapital;

    MyStreamingReader3() {
        // (1)
        // Beginning and ending of the outer array of objects:
        addCallback("[{", parser -> myData = new MyData());
        addCallback("[}", parser -> done());

        // (2)
        // Add callback for simple list of values:
        addCallback("[{list-of-numbers[*", parser -> myData.getListOfNumbers().add(parser.getValueAsInt()));

        // (3)
        // Add callback for list of objects, which we only read one value from:
        addValueCallback("[{list-of-letters[{letter", parser -> myData.getListOfLetters().add(parser.getValueAsString()));

        // (4)
        // Add callbacks for list of objects, which we read multiple values from:
        //@formatter:off
        addCallback     ("[{list-of-people[{",           parser -> person = new Person());
        addValueCallback("[{list-of-people[{first-name", parser -> person.setFirstName(parser.getValueAsString()));
        addValueCallback("[{list-of-people[{last-name",  parser -> person.setLastName (parser.getValueAsString()));
        addValueCallback("[{list-of-people[{age",        parser -> person.setAge      (parser.getValueAsInt()));
        addCallback     ("[{list-of-people[}",           parser -> myData.getListOfPeople().add(person));
        //@formatter:on

        // (5)
        // Add callbacks for map of key/value pairs:
        addCallback("[{map-of-key-value-pairs{*", parser -> {
            key = parser.getValueAsString();
            parser.nextToken();
            value = parser.getValueAsString();
            myData.getMapOfKeyValuePairs().put(key, value);
        });

        // (6)
        // Add callbacks for map of key/value objects:
        //@formatter:off
        addCallback     ("[{map-of-state-capitals{*",         parser -> stateKey     = parser.getValueAsString());
        addValueCallback("[{map-of-state-capitals{*{name",    parser -> stateName    = parser.getValueAsString());
        addValueCallback("[{map-of-state-capitals{*{capital", parser -> stateCapital = parser.getValueAsString());
        addCallback     ("[{map-of-state-capitals{*}",        parser -> myData.getMapOfStateCapitals().put(stateKey, new State(stateName, stateCapital)));
        //@formatter:on
    }

    private void done() {
    }
}

Let’s look at each individual section:

(1) The parser detects the beginning and end of the top-level array of object
(2) Retrieves a simple array of numbers; note the use of addCallback() instead of addValueCallback()
(3) Retrieves a list of single-value objects
(4) Retrieves a list of multi-value objects
(5) Retrieves a map of simple key-value pairs; note the parser.nextToken() call
(6) Retrieves a map of complex objects

Usage #3

Again, the usage is identical to the other examples.