:::: MENU ::::
Browsing posts in: Programming

Highlights from 2nd International Conference on Urban Informatics, Hong Kong

Hong Kong Polytechnic University (PolyU) Campus (Background: Jockey Club Innovation Tower, Zaha Hadid Architects)

This week I had the pleasure of attending the 2nd International Conference on Urban Informatics, run by the Hong Kong Polytechnic University. The conference had an impressive scope of keynotes from various universities such as MIT, UCL, Shenzhen University and presentations of research in conference streams. It was also an ambitious event – further announcing a new international Informatics society, a book series, a set of new courses at PolyU and new journal to follow up the week’s activities.

One of the conference keynotes (Carlo Ratti, MIT Senseable City Lab)

The conference stream presenters ranged from Masters students to experienced Professors and professionals with a very large breadth of topics in ‘Urban Informatics’ spanning themes of AI, Urban Models, Emerging Technologies, Environmental Monitoring and Remote Sensing techniques. In particular, themes from the sessions I attended seemed to often focus on using both supervised machine learning methods and unsupervised clustering methods to understand variety urban systems – from mobility to health.

Some impressive LiDAR technology on show – creating extremely detailed live 3D models… very science fiction! (SkyNet anyone?)

The following presentations were my personal highlights.

Take a Look Around: Geographic Knowledge Discovery for the Case Study of London Streetviews by Dr. Stephen LAW, The Alan Turing Institute and University College London, UK

Stephen presented a comprehensive suite of research on using Google Streetview data and AI (convolutional neural networks) towards automated ways to quantify urban design elements at a large, detailed scale across cities. In particular, it was really impressive to see aesthetic / human experiential notions of active and passive frontages able to be identified en-masse across cities – which could potentially have great impacts as a tool for designers and planners to create more vibrant, happy and healthy environments.

Quantifying aspects of urban design on a massive scale is crucial to help develop a deeper understanding of urban designs elements that contribute to the success of a public space. In this study, we further develop the Street-Frontage-Net (SFN), a convolutional neural network (CNN) that can successfully evaluate the quality of street frontage as either being active (frontage containing windows and doors) or blank (frontage containing walls, fences and garages).

More articles by this author:

Law, S., Seresinhe, C. I., Shen, Y., & Gutierrez-Roig, M. (2018). Street-Frontage-Net: urban image classification using deep convolutional neural networks. International Journal of Geographical Information Science, 1-27.

Law, S., Shen, Y., & Seresinhe, C. (2017, November). An application of convolutional neural network in street image classification: the case study of london. In Proceedings of the 1st Workshop on Artificial Intelligence and Deep Learning for Geographic Knowledge Discovery (pp. 5-9). ACM.

A 4D Spatio-Temporal Approach to Modelling Land Value Uplift from Rapid Transit in High Density and Topographically-Rich Cities by Dr. Christopher Donald HIGGINS, The Hong Kong Polytechnic University, HK

Christopher proposes new approaches to accessibility analysis and predicting land-value uplift which is more sensitive to volumetric and topographically complex cities, such as Hong Kong. As well as showing relationships between access to stations and property value, this work provides innovative toolbox in order to tackle understanding travel time which typically does not thoroughly consider both slope and building height of first and last mile travel to stations.

Read more:
Higgins, C. D. (2019). A 4D spatio-temporal approach to modelling land value uplift from rapid transit in high density and topographically-rich cities. Landscape and Urban Planning, 185, 68-82.

“The present research utilizes spatio-temporal methods in a quasi-experimental research design to examine changes in property values associated with pedestrian accessibility to the West Island Line heavy rail extension in Hong Kong. Several innovations in methods and techniques are proposed that respond to the econometric challenges involved in conducting research in high density, topographically-rich cities. Of these, the paper incorporates landscape topography throughout its estimation process, including the calculation of slope-aware measures of walkable accessibility on a 3D pedestrian network and proposes a new Spherical Distance Weights method for capturing horizontal and vertical spatial association among observations in 3D space. Finally, these weights are combined with measures of temporal distance for a 4D approach that accounts for relations among observations in space and time.”

See the author’s full article at here

See also: Higgins, C.D. & P.S. Kanaroglou. A Latent Class Method for Classifying and Evaluating the Performance of Station Area Transit-Oriented Development in the Toronto Region. Journal of Transport Geography. 52, 61-72. DOI: 10.1016/j.jtrangeo.2016.02.012

A Systematic Tracing of Temporal and Spatial Boundaries of Places with High Concentration of Human Activity to Identify Popular Non-Work Destinations by Prof. Roberto Ponce LOPEZ, Tecnologico de Monterrey, Mexico

Roberto presented an impressive suite of analysis forming part of his Doctoral work ways to mine and classify large social data sets to more meaningfully understand what people do in cities outside of work (which still remains a primary function of many urban models!). Singapore formed an interesting Case Study.

“Research uses data from Google Place API to characterize the built environment of Singapore, building a typology of place that differentiates between shopping malls, strips of retail, food markets and large commercial areas. The method of analysis relies on spatial clustering and k-medoids clustering.”

See his personal blog for more detail on Roberto’s work.

The City

Hong Kong itself was a highlight of the conference. For any urbanist, Hong Kong is one of the most unique vertical cities in the world. Its form and structure become even more interesting the more one walks around and attempts to comprehend how it has all been stacked together in such a small space. The unique mix of cultures living in the city over time makes for a fascinating exploration of art, food, museums and architecture .. indeed complementing the many physical levels the city has on offer.

Yick Cheong Building – five incredibly dense residential buildings stacked closely to one another

Duk Ling – traditional Chinese Junk boat ride on the Harbour

View from Victoria Peak, highest hill on HK Island

Murakami vs Murakami Exhibit, Tai Kwun Contemporary

Didier Jaba Mathieu’s work at the Tung Wah Centenary Garden, Queen’s Road East in Wan Chai

Vegetarian restaurant, Nan Lian Garden

Pavilion of Absolute Perfection, Nan Lian Garden

Thanks for having us Hong Kong PolyU – looking forward to seeing where you can take Urban Informatics in future… 🙂

ICUI 2019 Conference Banner

ICUI 2019 Conference Pack


How to scrape data from a website in 10 lines using Beautiful Soup and Python

Have you ever wanted to scrape data from a webpage where their open data isn’t available? It is actually quite straightforward for people with a little coding knowledge to retrieve a lot of data with the power of Python and libraries such as Beautiful Soup.

Beautiful Soup is a Python package for parsing HTML and XML documents. It creates a parse tree for parsed pages that can be used to extract data from HTML, which is useful for web scraping. It is available for Python 2.6+ and Python 3.

You can perform research with this data scraped over time, or simply use it for personal use.

Below are two common data sets that everyday people would find useful – Property and Jobs!

Real Estate – Finding sold history of properties

  1. Navigate to webpage – for example http://realestate.com.au/sold

2) Perform a search ; for example here we have searched for all properties sold in Newcastle (you can do this for big areas, or specific streets .. it’s up to you).

3) Extract the URL of the search results to see if you can loop over the results:


For example, as above, the value ‘1’ returns us the first page, if we replace that with ‘2’ we get the second page


This means we can perform a simple loop over the data.

4) Right click over the element you want to retrieve and click inspect. This will tell you what part of the site’s HTML you would like to retrieve. For example, by right-clicking the price we can see that prices are stored in a <span> tag with the class ‘property-price’.

Run this script (>>>> is a tab):

from bs4 import BeautifulSoup
import requests
>>>>for num in range(0,20):
>>>>url = str(‘www.realestate.com.au/sold/in-olivers+hill,+vic+3199%3b/list-‘+str(num))
>>>>r = requests.get(“http://” +url)
>>>>data = r.text
>>>>soup = BeautifulSoup(data)
>>>>mydivs = soup.findAll(“span”, {“class”: “property-price”})
>>>>for line in mydivs:

This will print the first 20 pages of results for all properties. You can give even more information here to extract particular features relevant to your property search – such as number of bedrooms, parking spaces and bathrooms. You can modify this to save it to a text file or csv, or even collect it over time for historical property sales by type.

Sites like ‘Inside AirBnB’ do these kind of scraping exercises (note that like AirBnB itself;  whether this work is allowed to be performed is a grey area) :



We’ve all been there – looking for a new job or seeing how much your skills are worth in the market.

Here’s an example of scraping Indeed.com for jobs data (>>>> is a tab)::

from bs4 import BeautifulSoup
import requests
for num in range(0,2000,10):
>>>>url = str(‘au.indeed.com/jobs?q=python+data+analytics&l=australia&start=’+str(num))
>>>>r = requests.get(“http://” +url)
>>>>data = r.text
>>>>soup = BeautifulSoup(data)
>>>>mydivs = soup.findAll(“a”, {“data-tn-element”: “jobTitle”})
>>>>for line in mydivs:

This will return all jobs related to ‘python data analysis’ in Australia. Have a go at trying to change it to search by salary, or to retrieve additional information when printed.

Happy scraping! Always make sure you read a website’s terms of service before performing any of the above ; and do so at your own risk!