:::: MENU ::::

Creating spatial data analytics dashboards in Cartoframes

With the strength of Carto in terms of spatial science and location intelligence ; and the easy access to data science packages in Python, Carto’s new project ‘Cartoframes‘ has a lot of potential to provide excellent mapping dashboards for data-hungry workflows.

Below is a quick tutorial I have made which will hopefully help new users figure out how to use it. It is in no way comprehensive, and there are probably some pieces missing ; but it should be enough to go off to get started! The tutorial covers some of the elements of creating a ‘live’ weather data dashboard for New South Wales in Australia.

What is Cartoframes ? (from https://github.com/CartoDB/cartoframes)
A Python package for integrating CARTO maps, analysis, and data services into data science workflows.

Python data analysis workflows often rely on the de facto standards pandas and Jupyter notebooks. Integrating CARTO into this workflow saves data scientists time and energy by not having to export datasets as files or retain multiple copies of the data. Instead, CARTOframes give the ability to communicate reproducible analysis while providing the ability to gain from CARTO’s services like hosted, dynamic or static maps and Data Observatory augmentation.

Features

Write pandas DataFrames to CARTO tables
Read CARTO tables and queries into pandas DataFrames
Create customizable, interactive CARTO maps in a Jupyter notebook
Interact with CARTO’s Data Observatory
Use CARTO’s spatially-enabled database for analysis

Step 1 – Install libraries
Install all of the relevant libraries. For me I’m using Canopy. Canopy provides Python 2.7 and 3.5, with easy installation and updates via a graphical package manager of over 450 pre-built and tested scientific and analytic Python packages from the Enthought Python Distribution. These include NumPy, Pandas, SciPy, matplotlib, scikit-learn, and Jupyter / IPython. You can get Canopy for free here.

Once installed, open the console and install the packages:
pip install cartoframes
pip install pandas

Step 2 – Import libraries

In a new Jupyter notebook, start by importing the libraries in the first block, these are the ones you’ll generally need (though you can go to town with other numerical / statistical packages here!):

import cartoframes
import pandas as pd
import numpy as np

Step 3 – Set up a Carto account and register for an API key

Start by going to Carto.com and signing up through the prompts.

Once you have signed up, in the top-right of your home page there should be setting toggle which show you:

View your public profile
Your account
Your API keys
Close session

Click on ‘Your API keys’ and copy what shows up on the next page. It should be a long string of text, looking something like this:

31b453f27c085747acc6a51a9e5717beae254ced

Step 4 – Connecting to your Carto account in Python
Try the following line of code in your next Jupyter code block, where xxxxxxxxxxxx is your new API key. This key allows Cartoframes to communicate directly with the data in your Carto account.

Where it says ‘oclock’ you should put your own username.

cc = cartoframes.CartoContext(base_url='https://oclock.carto.com',api_key='xxxxxxxxxxxx')

When you run this code and call ‘cc’ it should provide you with a message such as this:
cartoframes.context.CartoContext at 0x1ea3fa2c518

This means that cartoframes has successfully accessed your Carto account and you can call ‘cc’ to reference accessing this account from now on. Make sure you keep your API key safe!

Step 5 – Upload some data to Carto
For this task, I downloaded the shapefile components of weather locations from the Australian Bureau of Meteorology. This is all of the spatial files (.shp, shx, .dbf etc) for IDM 13 from:
ftp://ftp.bom.gov.au/anon/home/adfd/spatial/

These are all the files prefixed by IDM000013 and suffixed by .dbf,.prj,.sbn,.sbx,.shp,.shx,.shp.xml. Carto will need these all in a .zip file before you upload them.

The metadata for this dataset can be found here:

IDM00013 – point places (precis, fire, marine)
http://reg.bom.gov.au/catalogue/spatialdata.pdf

Once you have downloaded these you can upload the shapefile and it should give you a series of geolocated dots covering all of Australia, with many attributes as described in the metadata above. For this I called the dataset ‘idm00013’.

Step 6 – Read the data in jupyter
Let’s test if everything is working. The following should display a dataframe of all of the aspatial information stored in each weather location:

carto_df = cc.read('idm00013')
carto_df

The following should give you a list of all of the variables available to you to access and change:
list(carto_df.columns.values)

Step 7 – Making a map
Now for the exciting bit – creating a Carto map inside the Jupyter notebook.
Here I’ve picked the elevation column with a brown colour scheme, try:


from cartoframes import Layer, BaseMap, styling
cc.map(layers=[BaseMap('light'),Layer('idm00013',color={'column': 'elevation','scheme': styling.brwnYl(7)},size=5)],
interactive=True)

The following map should display, with light brown showing where the weather points are a low elevation, and high points shown in a darker brown.

Extension – Accessing and parsing a live data feed

The code below retrieves the latest weather forecasts for the weekend ahead from the Bureau of Meteorology’s API. It is stored in a dataframe ‘df’.

I’ll leave the indentation as part of this tutorial!

import xml.etree.ElementTree as ET
import csv
import pandas as pd
import urllib.request
req = urllib.request.Request('ftp://ftp.bom.gov.au/anon/gen/fwo/IDN11060.xml')
with urllib.request.urlopen(req) as response:
xml_data = response.read()
list_dict = []
root = ET.XML(xml_data)
for element in root.findall('forecast'):
for area in element:
for forecast in area:
min_temp = ''
max_temp = ''
aac_id = area.get('aac')
forecast_date = forecast.get('start-time-local')
for element in forecast:
if element.attrib['type'] == 'air_temperature_minimum':
min_temp = element.text
elif element.attrib['type'] == 'air_temperature_maximum':
max_temp = element.text
list_dict.append({'aac':aac_id, 'forecast_date':forecast_date, 'low_temp': min_temp, 'max_temp':max_temp})
df = pd.DataFrame(list_dict)
df

Extension Part 1 – Joining in a live data source

We now want to join the geographical data from the first exercise with this live data feed.
This is done with a ‘left’ join, so we keep all of the weather forecast records and add the geographic data to them.

merged_data = pd.merge(df,carto_df,on='aac')
merged_data

Extension Part 2 – Selecting some data

Now we filter out all records to get one particular day’s forecast (you will need to change the date here to current date).
The filtered data is then written to a new dataset in Carto called ‘merged_weathermap’.

one_forecast = merged_data[merged_data['forecast_date']=='2018-01-16T00:00:00+11:00']
cc.write(one_forecast, 'merged_weathermap',overwrite=True)

Extension Part 3 – Putting it all together
#Step 10

Now let’s add the data from the Weather feed API to a Cartoframes map. The following reads in the merged_weathermap dataset we just created and colours
in the maximum temperature for the forecast data for each weather point in New South Wales. Pink being a high temperature, and blue being a lower temperatue.

from cartoframes import Layer, BaseMap, styling
cc.map(layers=[BaseMap('light'),Layer('merged_weathermap',color={'column': 'max_temp','scheme': styling.tropic(10)},size=10)],
interactive=True)

That’s it! From here, it is feasible to see with a bit of extra work and some scripts that continuously ping the APIs etc that we are only a few steps away from creating live dashboards which integrate other statistical and mathematical packages, such as even including machine learning.

Looking forward to seeing developments in this space and if you have any feedback or ideas let me know!

For more information on Cartoframes have a look at their documentation.


International Research – Lessons from Shanghai

Tongji CAUP International Doctoral School of Future City and Architecture 2017

Over the past two weeks I’ve had the pleasure of attending the Tongji University International Doctoral School of Future City and Architecture, hosted by their College of Architecture and Urban Planning (CAUP) as a representative PhD student from the University of New South Wales (UNSW) in Sydney. This is an intensive two-week program held in Shanghai which invites 100 students from Tongji University, other Asian universities and countries around the world (most of which hold dual-degrees with the university).

The program is aimed to build professional research relationships and friendships, invite doctoral students to extend their training in research design, invite talks from a variety of academic and professional guest lecturers from around the world and to develop ideas for research proposals which can be applied to Shanghai. The content of these (over 30) lectures was broad – from emerging materials science, to new ways of preserving heritage, artificial intelligence in architecture, mobility, vertical urbanism, research methods and simply inspiration for different types of research thinking.

**Left: ‘Vertical Urbanism & Urban Bowls’ – Martin Felsen, Associate Professor / Director of Architecture Program, Illinois Institute of Technology (IIT)

** Top-Right: ‘Research is fun, make it yours!’ – Luca Fabris, Associate Professor, Department of Architecture and Urban Studies, Politecnico Milano

** Bottom-Right: ‘Visualization of Urban Planning Regulation in Japan Using City Engine’ – Shen Zhenjiang – Vice-Head of Graduate School of Environmental Design, Kanazawa University. An excellent talk on how to codify complex planning zoning and building laws in ESRI CityEngine to evaluate both the existing city and future planning proposals. Continue Reading


Finding the centre point of Sydney

During this year’s GovHack our team was interested in visualising open economic data of Sydney. There were several outputs of the project ; and this was the least sophisticated but ended up the most interesting.

If you were to try and pick the centre of Sydney, where would you say it was? Probably first instinct is just to go where it says ‘Sydney’ on Google Maps, right?

Well we found it. This is the centre of Sydney! Recognise it?

2017-09-21 11_59_49-Durham St - Google Maps

No? Ok. What about this?

2017-09-21 11_58_38-PhotoMaps by nearmap

Still no? Well it is important as this is the forecast central location of Sydney’s projected population in 2040.

We calculated the average location of population and employment in Sydney from 2011 – 2040. This was based on the location of people and jobs in the Bureau of Transport Statistics’ forecasts, using a weighted average location based on the centre of travel zones and population/employment within them. This is what we ended up with (an interactive version below). The blue houses represent population, and the green buildings represent employment. The size of the icon denotes year, from 2011 to 2040.

2017-09-21 11_05_33-Economic Centres Map _ CARTO

As you can see, the forecasts show the centre of each moving west at about 50 metres each year, and it is definitely not very close to the ‘City of Sydney’ LGA as an outsider may expect.

In 2040, sprawling Sydney’s population centre will have moved from Halvorsen Park to Rose Hill, and the employment centre from Rhodes to Wentworth Point.

Some interesting further analysis would be in applying the same method to different cities. While it is a simple calculation it is something worth thinking about – particularly when trying to provide equal transport and social services or all.

Interactive map below:

Datasets: Bureau of Transport Statistics, Transport for NSW – Population and Employment Projection

Here’s an example of a different method applied to London:

London’s Real Centre Point


PopWeighted1


Pages:1234567...13