Mapping Google Datatable JSON format to Sencha GXT ListStores for charting

By now, if you’ve been keeping up with my blog, you’ll notice that we make extensive use of charting in our application. We decided to go with Sencha GXT charting because we were under time pressure to finish our product for a release. However, we weren’t so crunched for time that I decided to model the data in an unintelligent way. Since charts essentially graphically display tabular data, I wanted to ensure that if we were to move to a different graphing package (which we are, btw) in the future, our data was set up for easy consumption.

After tooling around a bit, I decided on google datatable JSON formatting for our response data from queries made to our server. Google datatable formatting essentially just mimics tabular data in JSON formatting. It’s also easily consumable by google or angular charts. Having used angular charts in the past, I knew that the data format would map nicely to charts. However, the immediate problem still loomed, which is how to convert from google datatable JSON format to consumable ListStores for consumption by GXT charts?

Solution: In the end, I ended up writing an adapter with specific implementations for consuming our backend charting data. I needed to write a Sencha/GXT specific one. Here’s an example of the data we receive from our REST query.

//Note: some data omitted for readability
{
    "offset": 0,
    "totalCount": 1,
    "items": [
        {
            "title": "Registration Summary",
            "data": {
                "cols": [
                    {
                        "id": "NotRegistered",
                        "label": "Not Registered",
                        "type": "number"
                    },
                    {
                        "id": "Registered",
                        "label": "Registered",
                        "type": "number"
                    }
                ],
                "rows": [
                    {
                        "l": "9608",
                        "c": [
                            {
                                "v": "5",
                                "f": null
                            },
                            {
                                "v": "4",
                                "f": null
                            }
                        ]
                    },
                    {
                        "l": "9641",
                        "c": [
                            {
                                "v": "2",
                                "f": null
                            },
                            {
                                "v": "5",
                                "f": null
                            }
                        ]
                    },
                    {
                        "l": "9650SIP",
                        "c": [
                            {
                                "v": "3",
                                "f": null
                            },
                            {
                                "v": "0",
                                "f": null
                            }
                        ]
                    }
                ]
            }
        }
    ]
}

And here’s an example of how it’s converted into a Sencha ListStore:

[
    {
        v1=4,
        label0=NotRegistered,
        v0=5,
        label1=Registered,
        id1=Registered,
        type1=number,
        id0=NotRegistered,
        type0=number,
        f1=null,
        l=9608,
        f0=null
    },
    {
        v1=5,
        label0=NotRegistered,
        v0=2,
        label1=Registered,
        id1=Registered,
        type1=number,
        id0=NotRegistered,
        type0=number,
        f1=null,
        l=9641,
        f0=null
    },
    {
        v1=0,
        label0=NotRegistered,
        v0=3,
        label1=Registered,
        id1=Registered,
        type1=number,
        id0=NotRegistered,
        type0=number,
        f1=null,
        l=9650SIP,
        f0=null
    }
...
]

Step 1: Define the ChartModel I/F to map to the JSON datatable format.

Here’s an example of a ChartModel java object that maps to JSON object.

public abstract class ChartModel {
	
	private static ChartFactory factory = GWT.create(ChartFactory.class);
	private static JsonListReader<ChartObjects, ChartObject> reader = new JsonListReader<ChartObjects, ChartObject>(factory, ChartObjects.class );
	
	/**
	 * These constants are fixed per the value retrieved from JSON
	 */
	public static final String POINT = "point";
	public static final String COL_LABEL = "label";
	public static final String COL_TYPE = "type";
	public static final String COL_ID = "id";
	public static final String CELL_VALUE = "v";
	public static final String CELL_FORMATTED_VALUE = "f";
	public static final String ROW_LABEL = "l";
	
	/*
	 * Define the data model
	 */
	
	public interface ChartObjects extends ModelList<ChartObject>{}
	
	public interface ChartObject {
		String getTitle();
		ChartData getData();
	}
	
	public interface ChartData {
		List<ChartCol> getCols();	//represents # series
		List<ChartRow> getRows();	//an array of array of cells, i.e. all the data
	}
	
	public interface ChartCol {
		String getId();
		String getLabel();
		String getType();
	}
	
	public interface ChartRow {
		List<ChartCell> getC();	//an array of chart cells, one per column
		String getL();	//label for this group of chart cells (e.g. a timestamp)
	}
	
	public interface ChartCell {
		String getV();	//get value
		String getF();	//get formatted value
	}

	public abstract ListStore<DataPoint> getListStore(ChartObject chartObject) throws Exception;
...
}

You can see that I also added an abstract getListStore method that requires any consumer of the ChartObject to define how the ChartObject gets modeled to the ListStore.

Step 2. Create an Adapter that extends the ChartModel and implements the function getListStore.

Here’s an example of a StackedBarChart model (the models turned out to be different for multiple series in a stacked bar chart and for example, a pie chart. Line charts and stacked bar charts turned out to be the same.


public class StackedBarChartModel extends ChartModel {
	@Override
	public ListStore<DataPoint> getListStore(ChartObject chartObject) throws Exception {
		return new GXTChartModelAdapterImpl(chartObject).getSeries(ChartType.STACKED_BAR);
	}
}

You can see that this method instantiates a new adapter instance for GXT charts, passes in the chart object (modeled to the datatable JSON format) and returns the appropriate chart series (in this case a BarSeries GXT object).

Step 3. Map the chart object to a ListStore.

public class GXTChartModelAdapterImpl implements GXTChartModelAdapter {


	@Override
	public ListStore<DataPoint> getSeries(ChartType type) throws Exception {
		switch(type) {
			case LINE:
				return getLineSeriesListStore();
			case STACKED_BAR:
				return getStackedBarSeriesListStore();
			case BAR:
				return getLineSeriesListStore();
			case PIE:
				return getPieSeriesListStore();
			default:
				throw new Exception("Unknown data series type.");
		}
	}

/**
	 * Returns a ListStore for a BarSeries GXT object to add to a chart
	 * @return
	 */
	private ListStore<DataPoint> getStackedBarSeriesListStore() {
		ListStore<DataPoint> _listStore = new ListStore<DataPoint>(new ModelKeyProvider<DataPoint>() {
			@Override
			public String getKey(DataPoint item) {
				return String.valueOf(item.getKey());
			}
		});
		
		int numPoints = this.getNumPoints();  //The number of rows in the rows obj in ChartObject
		int numSeries = this.getNumSeries();  //The number of rows in the col obj in ChartObject
		
		for (int i = 0; i < numPoints; i++) {	
			//This key must be unique per DataPoint in the store
			DataPoint d = new DataPoint(indexStr(ChartModel.POINT,Random.nextInt()));
			
			for (int index = 0; index < numSeries; index++) {
				d.put(indexStr(ChartModel.COL_LABEL, index), chartObject.getData().getCols().get(index).getLabel());
				d.put(indexStr(ChartModel.COL_TYPE, index), chartObject.getData().getCols().get(index).getType());
				d.put(indexStr(ChartModel.COL_ID, index), chartObject.getData().getCols().get(index).getId());
			}
			
			ChartRow row = chartObject.getData().getRows().get(i);	//get the i-th point
			d.put(ChartModel.ROW_LABEL, row.getL());
			for (int j = 0; j < numSeries; j++) {
				if (row.getC().get(j) != null) {	//if ith-point is not blank
					d.put(indexStr(ChartModel.CELL_VALUE, j), row.getC().get(j).getV());
					d.put(indexStr(ChartModel.CELL_FORMATTED_VALUE, j), row.getC().get(j).getF());
				} else {	//otherwise, assume 0 value
					d.put(indexStr(ChartModel.CELL_VALUE, j), "0");
					d.put(indexStr(ChartModel.CELL_FORMATTED_VALUE, j), "");
				}
			}
			_listStore.add(d);
		}
		
		return _listStore;
	}
...
}

A few things to note here.

  • A DataPoint is basically just a map of keys and value pairs. Since the DataPoint object is specific to the ListStore, the implementation is in the interface GXTChartModelAdapter. To do this, I used the instructions here: https://www.sencha.com/blog/building-gxt-charts/
  • Next, you can see that for each bar I have (not each stack), I create a new DataPoint object that contains a the corresponding value of the specific row in the ChartObject.rows object. I know that each column corresponds to a stacked bar, so I increment the “v” value by 1 and use this as the key / value pair to the DataPoint.
  • Finally, because the number of columns defines the number of stacks I expect in each bar, if there isn’t a value (because our backend didn’t provide one), I assume the value is 0. This enforces that the v(n) in the first bar will correspond to v(n) in any subsequent bars.
  • I added an “l” value to each row in the ChartObject.rows obj which corresponds to a label for the entire stacked bar. I know that in the datatables JSON format you can specify the first row in each row obj to be a String and correspond to the label for each bar. However, it’s harder to enforce this type of thing in java and we expected numeric data back each time.


Step 4. Pass in the data into a Sencha GXT Chart!

Now the easy part. Essentially the data is transformed from the datatables format to a ListStore with each DataPoint key: v0 – vN corresponding to a stack in a bar.

Thus when instantiating the chart, this is all I had to do:

private BarSeries<DataPoint> createStackedBar() {

		BarSeries<DataPoint> series = new BarSeries<DataPoint>();
		series.setYAxisPosition(Position.LEFT);
		series.setColumn(true);
		series.setStacked(true);
		
		//numSeries is the number of bars you want stacked, it conforms to the chartObject.cols().length
		for (int i = 0; i < numSeries; i++) {
			MapValueProvider valueProvider = new MapValueProvider(ChartModel.CELL_VALUE + i);
			series.addYField(valueProvider);
		}
		
		return series;
	}

When I instantiate the BarSeries, I just pass in a key for each the number of stacked bars in my response data.

Finally, the finished product.

Screen Shot 2014-04-09 at 5.03.44 PM

Advertisements

Custom axes with Sencha 3.1 GXT Charts

Here’s a short idiom for today. If you have a Sencha chart with multiple BarSeries, you can click on the legend by default, which will “exclude” certain values from the recalculation of the y-axis minimum and maximum. We found that there are certain times when Sencha will compute the max range of a chart as a value less than 10, while the axis has 10 tick marks. This leaves some ugly decimal points in the y-axis labels, which just doesn’t make sense in some instances (e.g. you can’t have a fraction of a whole object).

Bad! You can't have a fraction of a device!
Bad! You can’t have a fraction of a device!

In order to get whole numbers here, I added a legend handler which recomputes the axis and sets a minimum value if Sencha would compute a < 10 max for the y-axis. Here's the code snippet:

			legend.addLegendSelectionHandler(new LegendSelectionHandler() {

				@Override
				public void onLegendSelection(LegendSelectionEvent event) {
					if (axis.getTo() < 10) {
						axis.setMaximum(10.0);
					} else {
						axis.setMaximum(Double.NaN);
						axis.setInterval(Double.NaN);
					}
					axis.calcEnds();
					axis.drawAxis(false);
					_chart.redrawSurface();
				}
				
			});

And here's the result:

Screen Shot 2014-04-01 at 2.04.52 PM
Ahh, much better.

Mapping tooltip of different key onto a GXT BarSeries

Here’s a little idiom for when you’re using one key for the value of a stacked bar chart but want to map a different ValueProvider onto your tooltip (or label).

		//Tooltip Config - gets (f) value but if null, just return the int
	    SeriesToolTipConfig<DataPoint> config = new SeriesToolTipConfig<DataPoint>();
	    config.setLabelProvider(new SeriesLabelProvider<DataPoint>() {
	        @Override
	        public String getLabel(DataPoint item, 
	        		ValueProvider<? super DataPoint, ? extends Number> valueProvider) {
	        	String vindex = valueProvider.getPath();
	        	String findex = vindex.replace(ChartModel.CELL_VALUE, ChartModel.CELL_FORMATTED_VALUE);
	        	MapLabelProvider labelProvider = new MapLabelProvider(findex);
	        	String tooltip = labelProvider.getValue(item);
	        	return tooltip == null || tooltip.equals(_chartConstants.emptyLabel()) ? 
	        		  String.valueOf(valueProvider.getValue(item).intValue()) : tooltip;
	        }
	    });
series.setToolTipConfig(config);

By default, Sencha GXT charts only passes in the valueProvider and the data item used to render the stacked bar chart. So what if you want to provider a different value for the tooltip or label? How do you add text? Since be follow the google datatables JSON format, we use a key of “v” in a cell array to represent the numeric value used to generate the chart. We use a key “f” in the same cell array to represent the tooltip.

Here’s what our JSON looks like and here’s the resultant store (After the adapter transforms the data).

{
    "offset": 0,
    "totalCount": 1,
    "items": [
        {
            "title": "Registration Summary",
            "data": {
                "cols": [
                    {
                        "id": "Stacked bar one",
                        "label": "Hamburgers",
                        "type": "number"
                    },
                    {
                        "id": "Stacked bar two",
                        "label": "Fries",
                        "type": "number"
                    }
                ],
                "rows": [
                    {
                        "l": "McDonalds",
                        "c": [
                            {
                                "v": "5",
                                "f": "5 hamburgers sold!"
                            },
                            {
                                "v": "2",
                                "f": "2 fries sold"
                            }
                        ]
                    },
                    {
                        "l": "Burger King",
                        "c": [
                            {
                                "v": "0",
                                "f": "0 hamburgers sold :("
                            },
                            {
                                "v": "3",
                                "f": "3 fries sold"
                            }
                        ]
                    }
... /* More omitted */
                ]
            }
        }
    ]
}

And here’s the resultant store:

[
    {
        v1=2,
        label0=Hamburgers,
        v0=5,
        label1=Fries,
        id1=Stacked bar two,
        type1=number,
        id0=Stacked bar one,
        type0=number,
        f1=2 fries sold,
        l=McDonalds,
        f0=5 hamburgers sold!
    },
    {
        v1=3,
        label0=Hamburgers,
        v0=0,
        label1=Fries,
        id1=Stacked bar two,
        type1=number,
        id0=Stacked bar one,
        type0=number,
        f1=3 fries sold,
        l=9650SIP,
        f0=0 hamburgers sold 😦
    }
... /* more omitted */
]

As you can see, we use v0 – vN to as values to render per bar in our chart, where each of v0 – vN corresponds to 1 stack within the bar. Here’s the resulting tooltip:

Custom tooltip
Custom tooltip