JSON Lexer

JSON Lexer Programming Guide

Overview

The JSON Lexer provides a powerful interface for parsing and accessing JSON data in NetBurner applications. This guide covers version 3.3.9 and later, which includes significant improvements over earlier versions.

Core Components

The JSON Lexer interface is defined in json_lexer.h and provides two primary object types:

ParsedJsonDataSet

Represents a complete JSON object that has been parsed and stored internally. This object contains the entire tokenized JSON structure in an optimized format using NetBurner poolBuffers.

JsonRef

A reference object that acts as a pointer to a location within a ParsedJsonDataSet. It simplifies navigation through complex JSON hierarchies by providing a clean, intuitive interface for accessing nested data.

┌─────────────────────────────────────────┐
│         ParsedJsonDataSet               │
│  ┌───────────────────────────────────┐  │
│  │  Internal tokenized JSON storage  │  │
│  └───────────────────────────────────┘  │
│                    ^                    │
│                    |                    │
│              ┌─────┴─────┐              │
│              │  JsonRef  │              │
│              │ (pointer) │              │
│              └───────────┘              │
└─────────────────────────────────────────┘

Version Notes

The interface introduced in version 3.3.9 supersedes the previous API. While legacy code continues to work, all new development should use the JsonRef-based interface described in this guide for improved clarity and ease of use.

Loading JSON Data

There are three primary methods to populate a ParsedJsonDataSet object:

Method 1: From Text Blob

Load JSON directly from a text string or character array.

// Example from: \nburn\examples\JsonLexer\ParseTest
const char* jdata = "{\"key\": \"value\"}";
size_t jlen = strlen(jdata);
ParsedJsonDataSet pjd((const char *)jdata, jlen, true);

Method 2: From Web Server

Retrieve JSON from an external HTTP endpoint.

// Examples in:
// \nburn\examples\JSON\GetJsonFromServer
// \nburn\examples\JSON\PostAndGetJsonExternalServer
// \nburn\examples\WebClient\EarthQuake
 
ParsedJsonDataSet pjds;
// Use web client functions to fetch data into pjds

Method 3: From File Descriptor

Read JSON from any file descriptor source (web socket, serial port, etc.).

ParsedJsonDataSet pjds;        // Declare the object
pjds.ReadFrom(int fd);          // Read from file descriptor

Data Flow Diagram

 JSON Source              ParsedJsonDataSet           JsonRef Access
┌────────────┐           ┌─────────────────┐        ┌──────────────┐
│ Text Blob  │           │                 │        │              │
│    or      │────>──────│  Tokenization   │───<────│  Navigate &  │
│ Network    │           │  & Storage      │        │  Extract     │
│    or      │           │                 │        │              │
│ File Desc. │           └─────────────────┘        └──────────────┘
└────────────┘

Accessing JSON Data

Example JSON Structure

Consider the following JSON data:

{
  "Anumber": 1234,
  "AString": "ImaString",
  "AnObject": {
    "Item1": 1,
    "Item2": "TheItem2"
  },
  "AnArray": [1, 2, 3, 4]
}

Basic Access Patterns

Simple Values

// Extract integer
int i = pjds("ANumber");
 
// Extract strings (two methods)
NBString s = pjds("AString");           // NBString object
const char* pStr = pjds("AString");     // Raw pointer (valid until dataset destroyed)

Nested Objects

Use JsonRef to simplify access to nested structures:

// Create a reference to the nested object
JsonRef jr = pjds("AnObject");
 
// Access nested values through the reference
int i1 = jr("Item1");
NBString s2 = jr("Item2");

Array Access

Use array indexing to access elements:

// Iterate through array elements
for(int i = 0; i < 4; i++) {
    printf("Item[%d] = %d\n", i, pjds("AnArray")[i]);
}

Access Operator Summary

The ParsedJsonDataSet and JsonRef objects use the following notation:

• ("name") - Find element by name
• [index] - Access array element by index

Navigation Example

pjds
 │
 ├─> ("ANumber") ──> int value
 │
 ├─> ("AString") ──> string value
 │
 ├─> ("AnObject") ──> JsonRef
 │    │
 │    ├─> ("Item1") ──> int value
 │    └─> ("Item2") ──> string value
 │
 └─> ("AnArray")
      │
      ├─> [0] ──> value
      ├─> [1] ──> value
      ├─> [2] ──> value
      └─> [3] ──> value

Type Conversion Operators

The ParsedJsonDataSet supports automatic type conversion through operator overloading:

operator bool()      const { return PermissiveCurrentBool(); }
operator float()     const { return (float)CurrentNumber(); }
operator double()    const { return CurrentNumber(); }
operator uint8_t()   const { return (uint8_t)CurrentNumber(); }
operator int()       const { return (int)CurrentNumber(); }
operator uint16_t()  const { return (uint16_t)CurrentNumber(); }
operator uint32_t()  const { return (uint32_t)CurrentNumber(); }
operator int8_t()    const { return (int8_t)CurrentNumber(); }
operator int16_t()   const { return (int16_t)CurrentNumber(); }
operator int32_t()   const { return (uint32_t)CurrentNumber(); }
operator time_t()    const { return (time_t)CurrentNumber(); }
operator const char*() const { return CurrentString(); }
operator NBString()  const { return (NBString)CurrentString(); }

Validation Functions

IsValid() Method

Verify that a JsonRef points to valid data:

pjds("AnArray")[3].IsValid();    // returns true (valid element)
pjds("AnArray")[4].IsValid();    // returns false (out of bounds)

Type Checking Functions

Determine the type of JSON data:

IsNumber()    // Check if element is a number
IsObject()    // Check if element is an object
IsString()    // Check if element is a string
IsBool()      // Check if element is boolean
IsNull()      // Check if element is null
IsArray()     // Check if element is an array

Validation Flow

┌──────────────┐
│  Access JSON │
│   Element    │
└──────┬───────┘
       │
       v
┌──────────────┐     Yes    ┌──────────────┐
│  IsValid()?  │─────>──────│  Check Type  │
└──────┬───────┘            └──────┬───────┘
       │                           │
       │ No                        v
       v                    ┌──────────────┐
┌──────────────┐            │ Extract Data │
│ Handle Error │            └──────────────┘
└──────────────┘

Example References

Complete working examples demonstrating these concepts:

• Basic Parsing: \nburn\examples\JsonLexer\ParseTest
• Complex Parsing: \nburn\examples\webclient\Earthquake

The Earthquake example demonstrates parsing a large, complex JSON structure from a live web service and extracting specific fields efficiently.

Earthquake Example Application

Overview

This example retrieves earthquake data from the USGS API and parses the complex JSON response to display recent seismic activity.

Source Code

#include <init.h>
#include <stdio.h>
#include <ctype.h>
#include <buffers.h>
#include <json_lexer.h>
#include <webclient/http_funcs.h>
#include <nbtime.h>
 
const char* url = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_day.geojson";
const char* AppName = "EarthQuake";
 
void UserMain(void* pd)
{
    init();                                       // Initialize network stack
    EnableSystemDiagnostics();                    // Enable diagnostics
    WaitForActiveNetwork(TICKS_PER_SECOND * 5);   // Wait for DHCP address
 
    printf("Application started\n");
 
    bool bTimeSet = SetTimeNTPFromPool();
    
    if(bTimeSet) {
        printf("Time Set\r\n");
    }
 
    ParsedJsonDataSet JsonResult;
    bool result = DoGet(url, JsonResult);
 
    if (result)
    {
        // Get earthquake count from metadata
        int32_t nQuakes = JsonResult("metadata")("count");
        printf("In the last Day we have had %ld Earthquakes > 4.5\r\n", nQuakes);
 
        // Iterate through earthquake events
        for (int i = 0; i < nQuakes; i++)
        {
            // Create a reference to properties for this earthquake
            JsonRef OneQuakeProperties = JsonResult("features")[i]("properties");
            
            if (OneQuakeProperties.Valid())
            {
                double magnitude = OneQuakeProperties("mag");
                NBString place_name = OneQuakeProperties("place");
                printf("Magnitude :%2.1f at %s", magnitude, place_name.c_str());
 
                if (bTimeSet)
                {
                    time_t when = OneQuakeProperties("time");
                    when /= 1000;
                    time_t now = time(NULL);
                    time_t delta = (now - when);
                    int sec = delta % 60;
                    int mins = (delta / 60) % 60;
                    int hour = (delta / 3600);
                    printf(" %02d:%02d:%02d ago", hour, mins, sec);
                }
                printf("\r\n");
            }
            else
            {
                printf("Parse error failed to find array element %d\r\n", i);
            }
        }
    }
    else
    {
        printf("Failed to contact server\r\n");
    }
 
    while (1)
    {
        OSTimeDly(TICKS_PER_SECOND * 1);
    }
}

JSON Structure Navigation

The earthquake data follows this hierarchy:

root
 │
 ├─> ("metadata")
 │    └─> ("count") ──> number of earthquakes
 │
 └─> ("features") ──> array
      │
      └─> [i] ──> array element
           │
           └─> ("properties")
                ├─> ("mag") ──> magnitude
                ├─> ("place") ──> location string
                └─> ("time") ──> timestamp

Data Extraction Flow

DoGet(url)
    │
    v
ParsedJsonDataSet
    │
    ├─> metadata
    │    └─> count ────────> Loop Control
    │
    └─> features[i]
         │
         └─> properties
              ├─> mag ──────> Display
              ├─> place ────> Display
              └─> time ─────> Calculate & Display

Example Output

Application started
Time Set
In the last Day we have had 17 Earthquakes > 4.5
Magnitude :4.5 at Fiji region 01:44:12 ago
Magnitude :4.6 at 128 km W of San Juan, Peru 10:06:35 ago
Magnitude :4.5 at 7 km NW of Papayal, Peru 10:32:50 ago
Magnitude :4.9 at 56 km SSE of Sand Point, Alaska 10:47:22 ago
Magnitude :4.9 at 22 km SSE of Padong, Philippines 11:05:28 ago
Magnitude :5.1 at 70 km WNW of San Antonio de los Cobres, Argentina 12:25:58 ago
Magnitude :5.3 at 116 km SSE of Kushiro, Japan 13:12:09 ago
Magnitude :4.6 at 112 km SSE of Kushiro, Japan 13:13:00 ago
Magnitude :4.8 at 129 km WNW of Pangai, Tonga 13:49:43 ago
Magnitude :4.9 at 256 km ESE of Ust'-Kamchatsk Staryy, Russia 20:15:25 ago
Magnitude :4.6 at 120 km NNE of Tobelo, Indonesia 20:23:53 ago
Magnitude :4.9 at 57 km ESE of Koseda, Japan 21:28:56 ago
Magnitude :4.8 at 24 km NE of La Paz, Philippines 22:38:44 ago
Magnitude :4.5 at northern Qinghai, China 23:10:02 ago
Magnitude :4.6 at 8 km S of Padong, Philippines 23:13:24 ago
Magnitude :4.5 at Fiji region 23:19:04 ago
Magnitude :5.1 at 108 km SSE of Hihifo, Tonga 23:39:10 ago

USGS JSON Response Structure

Response Schema

The USGS earthquake API returns data in GeoJSON format with the following structure:

{
  "type": "FeatureCollection",
  "metadata": { ... },
  "features": [ ... ],
  "bbox": [ ... ]
}

Metadata Section

"metadata": {
  "generated": <timestamp>,
  "url": <string>,
  "title": <string>,
  "status": <number>,
  "api": <version>,
  "count": <number>
}

Features Array

Each earthquake event is represented as a feature:

"features": [
  {
    "type": "Feature",
    "properties": {
      "mag": <number>,
      "place": <string>,
      "time": <timestamp>,
      "updated": <timestamp>,
      "url": <string>,
      "status": <string>,
      "tsunami": <0|1>,
      ...
    },
    "geometry": {
      "type": "Point",
      "coordinates": [<lon>, <lat>, <depth>]
    },
    "id": <string>
  },
  ...
]

Parsing Strategy

Step 1: Validate Response
         │
         v
Step 2: Extract Metadata
         │
         ├─> Get earthquake count
         └─> Verify API status
         │
         v
Step 3: Iterate Features Array
         │
         v
Step 4: For Each Feature
         │
         ├─> Get properties reference
         ├─> Extract magnitude
         ├─> Extract location
         ├─> Extract timestamp
         └─> Format output

Complete JSON Sample

The following shows a condensed version of the actual JSON returned by the USGS API. To view the complete response structure when running the example, uncomment this line in the source code:

// JsonResult.PrintObject(true);

Sample Structure

{
    "type": "FeatureCollection",
    "metadata": {
        "generated": 1666815488000,
        "url": "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_day.geojson",
        "title": "USGS Magnitude 4.5+ Earthquakes, Past Day",
        "status": 200,
        "api": "1.10.3",
        "count": 17
    },
    "features": [
        {
            "type": "Feature",
            "properties": {
                "mag": 4.5,
                "place": "Fiji region",
                "time": 1666809240464,
                "updated": 1666814388040,
                "url": "https://earthquake.usgs.gov/earthquakes/eventpage/us7000ikie",
                "status": "reviewed",
                "tsunami": 0,
                "type": "earthquake"
            },
            "geometry": {
                "type": "Point",
                "coordinates": [-178.3577, -20.3848, 551.981]
            },
            "id": "us7000ikie"
        }
    ],
    "bbox": [-178.3577, -24.0942, 10, 166.3788, 55.5213, 590.211]
}

Best Practices

Memory Management

• JsonRef objects are lightweight references, not copies
• const char* strings are valid only while the ParsedJsonDataSet exists
• Use NBString for persistent storage

Error Handling

Always validate before accessing data:

JsonRef element = pjds("SomeKey");
if (element.IsValid()) {
    // Safe to access
    int value = element;
} else {
    // Handle missing or invalid data
}

Performance Considerations

┌─────────────────────────────────────────────────────────────────┐
│  Best Practice: Cache JsonRef Objects                           │
│                                                                 │
│  Good:                                                          │
│    JsonRef props = pjds("features")[i]("properties");           │
│    double mag = props("mag");                                   │
│    NBString place = props("place");                             │
│                                                                 │
│  Avoid:                                                         │
│    double mag = pjds("features")[i]("properties")("mag");       │
│    NBString place = pjds("features")[i]("properties")("place"); │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Type Safety

Use type checking before conversion:

if (element.IsNumber()) {
    double value = element;
}
if (element.IsString()) {
    NBString text = element;
}

Advanced Usage

Nested Array Navigation

// Access deeply nested structures
JsonRef deepElement = pjds("level1")("level2")[index]("level3");

Dynamic Key Access

// Build key names at runtime
NBString keyName = "dynamic_" + NBString(i);
int value = pjds(keyName.c_str());

Array Iteration Patterns

// Safe iteration with validation
JsonRef array = pjds("myArray");
if (array.IsArray()) {
    for (int i = 0; array[i].IsValid(); i++) {
        // Process array[i]
    }
}

Summary

The JSON Lexer provides a powerful, intuitive interface for working with JSON data in NetBurner applications:

• ParsedJsonDataSet handles parsing and storage
• JsonRef simplifies navigation of complex structures
• Automatic type conversion reduces boilerplate code
• Validation functions ensure safe data access
• Performance optimizations through internal poolBuffer management

For complete working examples, refer to the NetBurner example applications in the \nburn\examples directory.