Mapping Google Datatable JSON format to Sencha GXT ListStores for charting

By now, if you’ve been keeping up with my blog, you’ll notice that we make extensive use of charting in our application. We decided to go with Sencha GXT charting because we were under time pressure to finish our product for a release. However, we weren’t so crunched for time that I decided to model the data in an unintelligent way. Since charts essentially graphically display tabular data, I wanted to ensure that if we were to move to a different graphing package (which we are, btw) in the future, our data was set up for easy consumption.

After tooling around a bit, I decided on google datatable JSON formatting for our response data from queries made to our server. Google datatable formatting essentially just mimics tabular data in JSON formatting. It’s also easily consumable by google or angular charts. Having used angular charts in the past, I knew that the data format would map nicely to charts. However, the immediate problem still loomed, which is how to convert from google datatable JSON format to consumable ListStores for consumption by GXT charts?

Solution: In the end, I ended up writing an adapter with specific implementations for consuming our backend charting data. I needed to write a Sencha/GXT specific one. Here’s an example of the data we receive from our REST query.

//Note: some data omitted for readability
{
    "offset": 0,
    "totalCount": 1,
    "items": [
        {
            "title": "Registration Summary",
            "data": {
                "cols": [
                    {
                        "id": "NotRegistered",
                        "label": "Not Registered",
                        "type": "number"
                    },
                    {
                        "id": "Registered",
                        "label": "Registered",
                        "type": "number"
                    }
                ],
                "rows": [
                    {
                        "l": "9608",
                        "c": [
                            {
                                "v": "5",
                                "f": null
                            },
                            {
                                "v": "4",
                                "f": null
                            }
                        ]
                    },
                    {
                        "l": "9641",
                        "c": [
                            {
                                "v": "2",
                                "f": null
                            },
                            {
                                "v": "5",
                                "f": null
                            }
                        ]
                    },
                    {
                        "l": "9650SIP",
                        "c": [
                            {
                                "v": "3",
                                "f": null
                            },
                            {
                                "v": "0",
                                "f": null
                            }
                        ]
                    }
                ]
            }
        }
    ]
}

And here’s an example of how it’s converted into a Sencha ListStore:

[
    {
        v1=4,
        label0=NotRegistered,
        v0=5,
        label1=Registered,
        id1=Registered,
        type1=number,
        id0=NotRegistered,
        type0=number,
        f1=null,
        l=9608,
        f0=null
    },
    {
        v1=5,
        label0=NotRegistered,
        v0=2,
        label1=Registered,
        id1=Registered,
        type1=number,
        id0=NotRegistered,
        type0=number,
        f1=null,
        l=9641,
        f0=null
    },
    {
        v1=0,
        label0=NotRegistered,
        v0=3,
        label1=Registered,
        id1=Registered,
        type1=number,
        id0=NotRegistered,
        type0=number,
        f1=null,
        l=9650SIP,
        f0=null
    }
...
]

Step 1: Define the ChartModel I/F to map to the JSON datatable format.

Here’s an example of a ChartModel java object that maps to JSON object.

public abstract class ChartModel {
	
	private static ChartFactory factory = GWT.create(ChartFactory.class);
	private static JsonListReader<ChartObjects, ChartObject> reader = new JsonListReader<ChartObjects, ChartObject>(factory, ChartObjects.class );
	
	/**
	 * These constants are fixed per the value retrieved from JSON
	 */
	public static final String POINT = "point";
	public static final String COL_LABEL = "label";
	public static final String COL_TYPE = "type";
	public static final String COL_ID = "id";
	public static final String CELL_VALUE = "v";
	public static final String CELL_FORMATTED_VALUE = "f";
	public static final String ROW_LABEL = "l";
	
	/*
	 * Define the data model
	 */
	
	public interface ChartObjects extends ModelList<ChartObject>{}
	
	public interface ChartObject {
		String getTitle();
		ChartData getData();
	}
	
	public interface ChartData {
		List<ChartCol> getCols();	//represents # series
		List<ChartRow> getRows();	//an array of array of cells, i.e. all the data
	}
	
	public interface ChartCol {
		String getId();
		String getLabel();
		String getType();
	}
	
	public interface ChartRow {
		List<ChartCell> getC();	//an array of chart cells, one per column
		String getL();	//label for this group of chart cells (e.g. a timestamp)
	}
	
	public interface ChartCell {
		String getV();	//get value
		String getF();	//get formatted value
	}

	public abstract ListStore<DataPoint> getListStore(ChartObject chartObject) throws Exception;
...
}

You can see that I also added an abstract getListStore method that requires any consumer of the ChartObject to define how the ChartObject gets modeled to the ListStore.

Step 2. Create an Adapter that extends the ChartModel and implements the function getListStore.

Here’s an example of a StackedBarChart model (the models turned out to be different for multiple series in a stacked bar chart and for example, a pie chart. Line charts and stacked bar charts turned out to be the same.


public class StackedBarChartModel extends ChartModel {
	@Override
	public ListStore<DataPoint> getListStore(ChartObject chartObject) throws Exception {
		return new GXTChartModelAdapterImpl(chartObject).getSeries(ChartType.STACKED_BAR);
	}
}

You can see that this method instantiates a new adapter instance for GXT charts, passes in the chart object (modeled to the datatable JSON format) and returns the appropriate chart series (in this case a BarSeries GXT object).

Step 3. Map the chart object to a ListStore.

public class GXTChartModelAdapterImpl implements GXTChartModelAdapter {


	@Override
	public ListStore<DataPoint> getSeries(ChartType type) throws Exception {
		switch(type) {
			case LINE:
				return getLineSeriesListStore();
			case STACKED_BAR:
				return getStackedBarSeriesListStore();
			case BAR:
				return getLineSeriesListStore();
			case PIE:
				return getPieSeriesListStore();
			default:
				throw new Exception("Unknown data series type.");
		}
	}

/**
	 * Returns a ListStore for a BarSeries GXT object to add to a chart
	 * @return
	 */
	private ListStore<DataPoint> getStackedBarSeriesListStore() {
		ListStore<DataPoint> _listStore = new ListStore<DataPoint>(new ModelKeyProvider<DataPoint>() {
			@Override
			public String getKey(DataPoint item) {
				return String.valueOf(item.getKey());
			}
		});
		
		int numPoints = this.getNumPoints();  //The number of rows in the rows obj in ChartObject
		int numSeries = this.getNumSeries();  //The number of rows in the col obj in ChartObject
		
		for (int i = 0; i < numPoints; i++) {	
			//This key must be unique per DataPoint in the store
			DataPoint d = new DataPoint(indexStr(ChartModel.POINT,Random.nextInt()));
			
			for (int index = 0; index < numSeries; index++) {
				d.put(indexStr(ChartModel.COL_LABEL, index), chartObject.getData().getCols().get(index).getLabel());
				d.put(indexStr(ChartModel.COL_TYPE, index), chartObject.getData().getCols().get(index).getType());
				d.put(indexStr(ChartModel.COL_ID, index), chartObject.getData().getCols().get(index).getId());
			}
			
			ChartRow row = chartObject.getData().getRows().get(i);	//get the i-th point
			d.put(ChartModel.ROW_LABEL, row.getL());
			for (int j = 0; j < numSeries; j++) {
				if (row.getC().get(j) != null) {	//if ith-point is not blank
					d.put(indexStr(ChartModel.CELL_VALUE, j), row.getC().get(j).getV());
					d.put(indexStr(ChartModel.CELL_FORMATTED_VALUE, j), row.getC().get(j).getF());
				} else {	//otherwise, assume 0 value
					d.put(indexStr(ChartModel.CELL_VALUE, j), "0");
					d.put(indexStr(ChartModel.CELL_FORMATTED_VALUE, j), "");
				}
			}
			_listStore.add(d);
		}
		
		return _listStore;
	}
...
}

A few things to note here.

  • A DataPoint is basically just a map of keys and value pairs. Since the DataPoint object is specific to the ListStore, the implementation is in the interface GXTChartModelAdapter. To do this, I used the instructions here: https://www.sencha.com/blog/building-gxt-charts/
  • Next, you can see that for each bar I have (not each stack), I create a new DataPoint object that contains a the corresponding value of the specific row in the ChartObject.rows object. I know that each column corresponds to a stacked bar, so I increment the “v” value by 1 and use this as the key / value pair to the DataPoint.
  • Finally, because the number of columns defines the number of stacks I expect in each bar, if there isn’t a value (because our backend didn’t provide one), I assume the value is 0. This enforces that the v(n) in the first bar will correspond to v(n) in any subsequent bars.
  • I added an “l” value to each row in the ChartObject.rows obj which corresponds to a label for the entire stacked bar. I know that in the datatables JSON format you can specify the first row in each row obj to be a String and correspond to the label for each bar. However, it’s harder to enforce this type of thing in java and we expected numeric data back each time.


Step 4. Pass in the data into a Sencha GXT Chart!

Now the easy part. Essentially the data is transformed from the datatables format to a ListStore with each DataPoint key: v0 – vN corresponding to a stack in a bar.

Thus when instantiating the chart, this is all I had to do:

private BarSeries<DataPoint> createStackedBar() {

		BarSeries<DataPoint> series = new BarSeries<DataPoint>();
		series.setYAxisPosition(Position.LEFT);
		series.setColumn(true);
		series.setStacked(true);
		
		//numSeries is the number of bars you want stacked, it conforms to the chartObject.cols().length
		for (int i = 0; i < numSeries; i++) {
			MapValueProvider valueProvider = new MapValueProvider(ChartModel.CELL_VALUE + i);
			series.addYField(valueProvider);
		}
		
		return series;
	}

When I instantiate the BarSeries, I just pass in a key for each the number of stacked bars in my response data.

Finally, the finished product.

Screen Shot 2014-04-09 at 5.03.44 PM

Advertisement

Mapping tooltip of different key onto a GXT BarSeries

Here’s a little idiom for when you’re using one key for the value of a stacked bar chart but want to map a different ValueProvider onto your tooltip (or label).

		//Tooltip Config - gets (f) value but if null, just return the int
	    SeriesToolTipConfig<DataPoint> config = new SeriesToolTipConfig<DataPoint>();
	    config.setLabelProvider(new SeriesLabelProvider<DataPoint>() {
	        @Override
	        public String getLabel(DataPoint item, 
	        		ValueProvider<? super DataPoint, ? extends Number> valueProvider) {
	        	String vindex = valueProvider.getPath();
	        	String findex = vindex.replace(ChartModel.CELL_VALUE, ChartModel.CELL_FORMATTED_VALUE);
	        	MapLabelProvider labelProvider = new MapLabelProvider(findex);
	        	String tooltip = labelProvider.getValue(item);
	        	return tooltip == null || tooltip.equals(_chartConstants.emptyLabel()) ? 
	        		  String.valueOf(valueProvider.getValue(item).intValue()) : tooltip;
	        }
	    });
series.setToolTipConfig(config);

By default, Sencha GXT charts only passes in the valueProvider and the data item used to render the stacked bar chart. So what if you want to provider a different value for the tooltip or label? How do you add text? Since be follow the google datatables JSON format, we use a key of “v” in a cell array to represent the numeric value used to generate the chart. We use a key “f” in the same cell array to represent the tooltip.

Here’s what our JSON looks like and here’s the resultant store (After the adapter transforms the data).

{
    "offset": 0,
    "totalCount": 1,
    "items": [
        {
            "title": "Registration Summary",
            "data": {
                "cols": [
                    {
                        "id": "Stacked bar one",
                        "label": "Hamburgers",
                        "type": "number"
                    },
                    {
                        "id": "Stacked bar two",
                        "label": "Fries",
                        "type": "number"
                    }
                ],
                "rows": [
                    {
                        "l": "McDonalds",
                        "c": [
                            {
                                "v": "5",
                                "f": "5 hamburgers sold!"
                            },
                            {
                                "v": "2",
                                "f": "2 fries sold"
                            }
                        ]
                    },
                    {
                        "l": "Burger King",
                        "c": [
                            {
                                "v": "0",
                                "f": "0 hamburgers sold :("
                            },
                            {
                                "v": "3",
                                "f": "3 fries sold"
                            }
                        ]
                    }
... /* More omitted */
                ]
            }
        }
    ]
}

And here’s the resultant store:

[
    {
        v1=2,
        label0=Hamburgers,
        v0=5,
        label1=Fries,
        id1=Stacked bar two,
        type1=number,
        id0=Stacked bar one,
        type0=number,
        f1=2 fries sold,
        l=McDonalds,
        f0=5 hamburgers sold!
    },
    {
        v1=3,
        label0=Hamburgers,
        v0=0,
        label1=Fries,
        id1=Stacked bar two,
        type1=number,
        id0=Stacked bar one,
        type0=number,
        f1=3 fries sold,
        l=9650SIP,
        f0=0 hamburgers sold 😦
    }
... /* more omitted */
]

As you can see, we use v0 – vN to as values to render per bar in our chart, where each of v0 – vN corresponds to 1 stack within the bar. Here’s the resulting tooltip:

Custom tooltip
Custom tooltip

Custom styling and appearance classes on top of Sencha themebuilder GXT 3.1 (Part 1 – Using the Themebuilder)

I must find somewhere for my collected knowledge on the “joy” of using Sencha themebuilder. At my company, we’re currently working on building a custom style for our web application. Somewhere in the past, the decision was made to use Sencha GXT for their custom widgets and styling on top of GWT. Like most frameworks, things went smoothly as long as we were using the functionality out of the box. However, when we tried venturing outside, we were quickly reprimanded by the limitations of the framework.

In our scenario, we needed to brand our application differently than one of the built-in Sencha themes. Fortunately for us, or so we thought, Sencha created a nice tool in the knick-of-time called the “themebuilder” (currently in beta as of 2/25/2014). The themebuilder alleviates some pains of styling if you stay strictly within the guardrails, but quickly becomes inadequate if you need to do styling on top of it.

Part A: About the themebuilder and running it.

    Running it is easy (just make sure you are using java version 1.7+). On a mac:

  • You run ./themer.sh with certain command line options. The themer script reads in a custom Sencha file (*.theme) which it then uses to compile and build a set of appearance classes which GWT accepts. Here’s an example of the command I used:
  • ./themer.sh ../examples/mytheme/mytheme.theme
  • As of the beta 2/25/2014, the running the themer will by default override the .jar file.

  • The .theme file is a huge pain to work with. It’s Sencha’s own proprietary format. It’s currently not well documented, which makes it very difficult to know how certain values within the .theme file map to Sencha widgets and how these get mapped on top of GWT and then what eventually gets spat out in your browser. Here’s a snippet:
    theme {
      /* First, create a name for your theme, and define a package to place it in */
      name = "myCustomTheme"
      basePackage = "com.example"
    
      /* Next, configure these basic defaults, to be used throughout the file */
      text = util.fontStyle("Arial, Helvetica, sans-serif", "12px", "#000000", "normal")
      textWhite = util.fontStyle("Arial, Helvetica, sans-serif", "12px", "#eeeeee", "normal")
    
      borderRadius = 6
    
      bgColor = "#ffffff"
      headerBgColor = "#666666"
      menuHoverColor = "#e7e7e7"
      menuSelectColor = "#b7daff"
      headerBgColorLight = "#f9f9f9"
    
      iconColor = "#777777"
      ...
    }
    

    You can see that the file is not quite json, java, or css. It’s some mishmash of everything, but it allows you to define variables and reference them later on in the .theme. This is Sencha’s claim for the themebuilders utility. By simply changing a few values (e.g. the background color, font color, font text, etc.), you can change the entire look and feel of your web application. That is all fine and dandy, but what is not described is what things you actually can and can’t change. For example, here’s a list of things I found that the themebuilder is not capable of:

    • Styling a widget in one instance different from another
    • Adding custom fonts
    • Styling nested widgets differently from non-nested widget (e.g. a button in one panel looking different than a button in another)
    • Support for icons libraries like font-awesome or glyphicon
    • Styling text within a toolbar (the LabelToolItem widget)
    • Styling the body content various content panels widgets (ContentPanel, FramedPanel, AccordionPanelAppearance, Window, Dialog)
    • Stylizing a button when its menu is open vs. closed

    In the next series of posts, I’ll be showing you how I solved each of these issues. Stay tuned for more.

    A general tip: When modifying the .theme file, I used sublime text and had the syntax set to “Javascript.” This enabled some basic coloring and recognition of comments and strings. The coloring was very helpful.

    Part B: Applying the custom theme on top of your GWT project
    This part is more specific to GWT users, but may be helpful to some. I’m currently working in a build environment using Java 1.7, Eclipse Version: Kepler Service Release 1, Build id: 20130919-0819, GWT plugin version 1.8.9 to Eclipse, and maven. Assuming that you’re already configured and up and running with GXT and GWT, you should have an App.gwt.xml file that looks something like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.5.1//EN"
      "http://google-web-toolkit.googlecode.com/svn/tags/2.5.1/distro-source/core/src/gwt-module.dtd">
    
    <module rename-to='mymodule'>
    ...
    	<!-- GXT Theme -->
     	<inherits name='com.example.Theme' /> 
    ...
    </module>
    

    The key line is the GXT Theme line where GWT applies a theme on top of all of its widgets. This theme name should follow the convention set forth in your .theme file.

      name = "myCustomTheme"
      basePackage = "com.example"
    

    After you build your theme.jar file using the custom .theme file (I used the skeleton.theme file included which has all the values in it), you need to add the jar to your buildpath (or to maven as a dependency). I found that the fastest workflow for me was to modify the .theme file, use an html tool that Sencha includes to approximate the styles, and have my buildpath point directly to the .jar file where it’s created by the themebuilder. For more on this html tool (which helps speed up theme creation), see this post.

    Edit: The new beta themebuilder (released by Sencha on 2/25/2014) spits out some instructions on how to do this, along with the context name of your theme.jar file.

    Here’s a sample:

    The customTheme theme has finished generating.
    Copy the jar (/path/to/theme/customTheme.jar) into your project and copy the following line into your gwt.xml file:
    
        <inherits name="com.example.Theme" />
    

    Then, I run GWT Server through my eclipse in debug mode, which picks up the changes to the theme. Because GWT is trying to rapidly render and compile all of the javascript files and apply the custom theme, it tends to operate very slowly.

    Hopefully if everything went according to play, you should see a different style on top of your application. In the next posts, I’ll start getting into customizing the theme and some workarounds for its deficiencies.

Adding css sprites in GWT on top of GXT 3.1

I’m a GWT/GXT beginner, so I typically end up having to look up how to do every little thing online in terms of styling/theming. However, after looking online for how to add css sprites to my application on top of GWT/GXT, I couldn’t find a good guide explaining how to do this. So I investigated and tinkered on my own and thought I would share what I discovered. There are a few necessary components:

Step 1: Define your ClientBundle

//My resource interface 
public interface MyResources extends ClientBundle {
	
    //@Source("mysprites.png")
    @ImageOptions(repeatStyle = RepeatStyle.None)
    ImageResource my_sprites();
    
    @Source("mysprites.css")
    MyCssResource resource();
    
    interface MyCssResource extends CssResource { 
    	String myLogo();
    	
    	String anotherSprite();
    }
	
}

Here’s what my sprite image looks like: example

First, it’s necessary to define an interface that extends the GWT ClientBundle. This interface becomes the entry point to injecting the proper CSS sprite into your web application.

In the source above, you can see that I have a .png (this has to be in the same package as the interface), which is your sprite image. The way sprites are loaded are through a css class, which is defined in the CssResource which is also a part of the interface. You use the @Source annotations to define what the names of your files are.

Here’s an example of what the css file should look like: You’ll notice that the gwt-image maps onto the name of the ImageResource method that is your css sprite. The height and width css properties correspond to the chunk of the image to use as your sprite. Background-position corresponds to where to start in the image (pixels from top, pixels from left).

/* mysprites.css */
@sprite .myLogo {
  gwt-image: "my_sprites";
  width: 86px;
  height: 36px;
  background-position: 0 0;
}

@sprite .anotherSprite {
  gwt-image: "my_sprites";
  width: 86px;
  height: 36px;
  background-position: 36 0;
}

Next, you define the interface of your CssResource, whose methods (String methodName();) get mapped to css classes defined in your css file.

Step 2: Get an instance of your ClientBundle using GWT.create

//My widget class
private static MyResources myresource = GWT.create(MyResources.class);

This is the instance that you will use to add sprites to your widget. I was doing this for a while, but noticed that after trying to call the methods made available through my CssResource interface, the css class was not being injected into my application. However, I did find that my sprite image was correctly injected. After some more online research, I found that you need to call this in order to ensure that the css class is actually injected. Add this line after you instantiate the class in order to actually put your css classes in your application

//My widget class
myresource.resource().ensureInjected();

Here, myresource is the GWT object created for the entire resources class. resources() is the CSSResource which has the method ensureInjected(); Finally, you need to actually add a widget that contains the correctly styling in order to render the sprite correctly. Since CSS sprites are just classes, I used a blank GWT HTML widget, which just gets rendered as a blank div in your application.

Step 3: Add the sprite to your application

//My widget class
HTML logo = new HTML();		
logo.addStyleName(myresource.resource().myLogo());
mybanner.add(logo, new HorizontalLayoutData(-1, -1, new Margins(0, 5, 5, 5)));

Finally, in your application if you inspect the element, you can see that the CSS class got rendered into the application and attached to the HTML widget

.

/* Inspected Javascript */
.GKTGO20BA3B-MyResources-MyCssResource-myLogo {
height: 492px;
width: 68px;
overflow: hidden;
background: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEQAAAHsCAYAAABrKwuFAAAPFUlE…Als9kMoqN9zlNMpAqDBOKWCBBGJ4BCICgySCoQhAEzbD+D1IzkMByojJyAAAAAElFTkSuQmCC") -0px -0px no-repeat;
width: 68px;
height: 36px;
background-position: 0 0;
}

The end result:

Screen Shot 2014-02-26 at 12.24.31 PM
Let me know if you have any comments or questions.