I’ve been playing around with the live integration of large cloud based data sets with D3.js. Most visualizations with D3 use some kind of static data, but I became interested in pulling different source into d3. For example, here is d3 integrated with parse.com which is fine for trivial data, but not much good for large data tables. Here’s how to integrate d3 with a large data set hosted on Google Fusion tables.
Clicking on a city (as opposed to hovering over it) will ‘freeze that city’, meaning that only routes involving that city will be shown until you unfreeze it. You can release the freeze by clicking on the city again, or by clicking on another city. In the example below, Tuscon has been frozen, and Delta Air is hovered over – so you can see where delta flies to Tuscon from.
This means that the backoff process has kicked in and another attempt at retrieving the Fusion data will be made shortly. In my tests so far, the backoff has worked fine and always been successful in eventually getting all the data.
Multiple queries on large datasets can certainly benefit from asynchronicity, and in fact each of the 5 separate data sets used in this visualization are kicked off at the same time . I’m using jQuery promises to orchestrate this, as I’ve done in previous examples such as Common query code. How to implement jQuery promises along with exponential backoff is covered in this Excel Liberation Blog post and you’ll see the code for it covered below.