How to Read Data From a Mongo Database in D3

I consider interactive data visualization to exist the critical tool for exploration of loftier-dimensional data.

That's led me to spend a adept amount of fourth dimension in the last few years learning some new skills (D3 and CoffeeScript) and developing some new tools, peculiarly the R packet R/qtlcharts, which provides interactive versions of the many information visualizations in R/qtl, my long-in-development R parcel for mapping genetic loci (called quantitative trait loci, QTL) that underlie circuitous trait variation in experimental organisms.

R/qtlcharts is rough in spots, and while it works well for moderate-sized data sets, information technology can't well handle truly big-scale data, as it only dumps all of the data into the file viewed by a spider web browser.

For large-scale data, one needs to dynamically load slices of the data based on user interactions. It seems all-time to have a formal database backside the scenes. But I think I'm not unusual, amidst statisticians, in having well-nigh no experience working with databases. My collaborators tend to go along things in Excel. Even for quite large problems, I go on things in flat files.

So, I've been trying to come to understand the whole database business organisation, and how I might utilise 1 for larger-scale data visualizations. I think I've finally made that final little conceptual step, where I tin encounter what I need to do. I made a minor illustration in my d3examples repository on GitHub.

Learning MongoDB

I idea mongoDB seemed like a good database to try:

• storage of general documents seems a adept match for the sort of information I work with

• utilise of JSON/BSON seems a good friction match for D3.js

So I've been taking an online-course though MongoDB University – the one for Node.js developers. I should have done the more general 1, using Python, but I was thinking it'd be interesting to larn Node, which is basically command-line scripting in javascript. I gauge the advantage of node is that you lot can just write javascript. Merely personally I prefer to switch to the language appropriate to the job, so I'm not as well hot on node. But peradventure I simply haven't come up to sympathise how best to make utilise of the whole callback function business.

MongoDB is non all that hard. It'due south just kind of boring. The MongoDB Uni course is quite well done, though a few of the lectures were frustratingly out-of-engagement. I have the MongoDB volume, only information technology's been rough going; the class has been expert for that. (The weekly deadlines have been adept for motivating me to become through the dull – for me – cloth.)

The obvious $.25 with a database: how to get stuff into information technology and so back out of information technology. And and so I approximate deleting or irresolute stuff. Then that'due south all fine. I as well learned about making indexes, so that you can more quickly get stuff out later. I don't think my needs for these things are all that complicated. And so I think I empathise those bits.

Getting the information for the data vis

The last key step that I was pondering last dark: how do I, based on some user interaction with a information visualization (clicking on a point, or some such), fire off a request to the database and get the data back?

I guessed some sort of RESTful API blazon thing. Simply while I tin write that down, I don't really understand what the hell it ways.

But and so I thought, what I need is to apply some sort of HTTP GET request. I kind of know what that means, merely I don't really understand what happens on the back end.

I'd been poking effectually in pymongo, the python-based mongoDB commuter. (I like it much ameliorate than the ruby or node drivers.) And as office of the Mongo University course, nosotros used express with Node to practise some HTTP Go and Push button. If I want to grab some bit of data, I could take a python script running on the back finish, looking for an http request, and and so based on some fundamental that was passed, grabbing data from the database and sending information technology back as JSON.

So I looked around for how to do this with mongoDB, found sleepy mongoose, but that seemed like more than I wanted, and then came to bottle, which seemed like what I needed to handle the http fleck. The example on the front folio is just what I want: take a name from a URL and use information technology as input to some function, and give results back.

MongoDB/D3 illustration

The goal of all this: as part of some information visualization, the user will hover over or click a bespeak, and this volition lead to a request to the database, and so a plot of the data that comes back.

It seemed similar a unproblematic test case would be to use the data for Anscombe's quartet. Put the iv series into a MongoDB database, and and then have iv buttons that, when clicked, cause a request to the database to grab the corresponding serial, which is then plotted.

So first I wrote an R script to grab the information from the table on the wikipedia page and write it to a JSON file. I'chiliad certain there was an easier way to get the data in a suitable form, but I was really procrastinating from writing a newspaper at this signal, and then I idea I'd exam my web scraping skill in a elementary case.

I and then wrote a python script that reads in the JSON file and inserts the information into a mongoDB database. You make a database connectedness with pymongo.MongoClient() and insert the documents with the insert_one() method to a collection. I after added an index with create_index(); pretty silly here, merely for large problems it would be useful.

And then, the data's in the database. How to get it back out? I wrote a second python script (just 8 lines, really), that uses bottle to create a server that connects a URL to a function call. The function then uses pymongo to observe the respective document and send it back as JSON. I tricky bit that needed some extra googling was getting the browser to allow the asking.

The D3 code (in coffeescript) is pretty standard. On clicking a button, I telephone call d3.json with the appropriate URL that corresponds to a request from the python script; the data come up back equally JSON, and are and so passed to a callback role to make the actual plot.

This MongoDB/D3 illustration is a toy case, merely it represents an of import footstep for me, personally. I'm rather proud of it.

I don't actually have a alive example of this. That'south the next thing: how to have this running on a proper spider web server, rather than simply locally.

shirleyraimad.blogspot.com

Source: https://kbroman.org/blog/2015/06/22/mongodb-with-d3-js/

Share this post