Boy, I can’t stop working on this. This morning I thought about a dynamic visualization with tornados. It was a 5 minutes work from the database to a shapefile to CartoDB visualization. That visualization was great but not really awesome. Then I thought I have thunders in the very same...

Posted on Tuesday, 05/08/14 at 01:02 in data , design , web

Read more

Never ending story

Yesterday I went to bed without telling how it went during the evening. Well I didn’t do much more coding apart from minimal touches to the table and the climogram. I wrote more on documenting about the project, so I put in order some ideas that I have for this...

Posted on Monday, 04/08/14 at 17:13 in web , design

Read more

First diagrams and interactions

After some attempts I’ve discarded using Crossfilter and dc. It’s a pity because they seem quite powerful but I don’t have time to invest on learning them. I’ll use plain d3 code that I already have some knowledge of. Nothing really fancy or powerful but at least it works. One...

Posted on Sunday, 03/08/14 at 16:04 in web , design

Read more

Small fixes on the website and crossfilter reading

This morning apart from more OSGeo Election work (voting are going well), I’ve fixed some small things regarding the Jekyll templates I’m using to produce the site. Also simplified the index page reordering the code to a new JavaScript file and moving all the externall calls to an outer template...

Posted on Sunday, 03/08/14 at 10:59 in web

Read more

First map on the web site and interactions

It’s been a long day but I have the foundation to work tomorrow on the visualization the whole day. As reported on the previous post, I’ve created a visualization on CartoDB for four layers. Next step has been to place this visualization into the front page of the web site....

Posted on Sunday, 03/08/14 at 01:09 in design , data

Read more

Putting the data into CartoDB

Woah, long afternoon. I’ve struggled with CartoDB as never used it so hard but I have something. Firstly, what I have at CartoDB are two tables: The table stations with the stations that appeared on 2008 records with the combined id for USAF and WBAN. The table observations with data...

Posted on Saturday, 02/08/14 at 21:32 in design , data

Read more

Filling the database with observation data

Time to write a post while the database is busy. Basically I’ve set up a table on the db with the proper fields for weather data, using a combined station_id that is created from both identifiers on the text files, USAF and the WBAN. Then I’ve created a class in...

Posted on Saturday, 02/08/14 at 16:45 in data

Read more

Setting up a local database

I have to handle like 3 million records to join with a stations table so I decided to setup a local database before doing anything on the web so I can explore my data in a more convenient way to do aggregations and so on. I’ve installed Postgres+PostGIS and created...

Posted on Saturday, 02/08/14 at 11:48 in data

Read more

Improving the web site

After some OSGeo Elections work and not too much sleeping (summer here is so hot it’s hard to sleep after 7am or so) I I’ve come back to the project. First thing this morning I’ve done is to try Darkly a different Bootswatch theme, I like flatly styles, let’s see...

Posted on Saturday, 02/08/14 at 09:16 in web

Read more

Processing all the stations

A python script has been developed to process the ish-history.csv using Fiona, Shapely and native CSV library to load the file and generate a shapefile of stations. Some of them doesn’t have codes or elevation data but all of them are geolocated on a valid position. On the next simple...

Posted on Saturday, 02/08/14 at 00:24 in data

Read more

Reviewing the data

After uncompressing the 2008 folder I see that I have: Country codes text file (fixed width): country-list.txt Database of stations as a CSV, TXT and XLS: ish-history.{csv,txt,xls} A folder with 2008 observation files On the 2008 folder there are 11.785 files with a total of 3.414.183 records (headers non counted)....

Posted on Friday, 01/08/14 at 22:57 in data

Read more

Setting up the EC2 vm and getting the data

Next step is accessing the dataset. I’ve never accessed data stored on a third party EBS volume. Anyway after reading the docs about EBS mapping I’m able to start a t2.micro instance. Once I had access to the server, I’ve mounted the EBS volume and downloaded the small general files...

Posted on Friday, 01/08/14 at 22:25 in sysadmin , data

Read more

Setting up the container website

Well this is my first post on this project. The idea behind the creation of this log is well, register the main steps done over this project. Don’t expect a meticulous wording, or many links and careful crafted images. This is more a dump of what’s going on so I’ll...

Posted on Friday, 01/08/14 at 22:03 in web

Read more