BostonCracks - bostonhacks 2018 submission

My team's bostonhacks 2018 submission which won the data science track. We submitted our solution to a problem proposed by the City of Boston.

Code

Github repo is here. Also be sure to check out our devpost!

Tools

Our Pitch to the Judges: The Problem - The Solution

The Problem

Challenge 1: Match Boston’s Ground Truth Sidewalk Data - Use Google Maps to try to replicate our sidewalk condition findings from 2014, to see if using this way is a viable option in the future.

In 2014, the City of Boston's team in charge of sidewalk repairs walked every inch of Boston's sidewalks. As they walked, they marked the degree to which the sidewalk was damaged and the precise location of the damaged sidewalk. While this method establishes a ground truth, it comes at a high cost - it's just not realistic for a team to consistenly examine every sidewalk in Boston. Because of this, they asked us to find them a solution.

The Solution & BostonCracks

Sidewalk Segmentation

If we want to be able to use Google Maps in order to replicate the city's 2014 findings we must be able to separate the sidewalk from any other aspects of a city scene. When using google street view this problem of separation becomes one very similar to the semantic image segmentation problem that self-driving cars face. With this realization, we decided to use a pre-trained DeepLabV3 model to help us solve this problem. This pre-trained model fits directly with this problem as it was trained on the Cityscapes Dataset which translates very well to the urban environment of Boston. What DeepLabV3 allows us to do is to take an image from google maps and get rid of everything but the sidewalk. For example:

First, we provide an image from google maps:

First, we provide an image from Google Maps

Then the DeepLabV3 model segments the image:

Then the DeepLabV3 model segments the image

Finally, we apply one last image pre-processing step and crop out everything but the sidewalk:

Finally, we apply one last image pre-processing step and crop out everything but the sidewalk

Now we can feed these isolated sidewalks into a classifier to determine if the sidewalk needs repairs!

Sidewalk Damage Classification

The Dataset: The possibility of finding a dataset containing damage scores for semantically segmented sidewalk was never even close to probable. Because of this, we were forced to make our own dataset, but because of Boston's StreetCaster project, we weren't running around blind. By extracting the latitude and longitude from the StreetCaster dataset we knew where to look for damaged sidewalks in Google Maps, and just about how damaged they should be. With this approach, we were successful in creating a small dataset (albeit tiny for ML) of labeled semantically segmented sidewalks.

The Architecture and Transfer Learning: At the center of any meaningful machine learning model is the data, but once the data is collected we need an architecture to best utilize such data. We decided to choose ResNet18 as our model. Because we were limited by both time and amount of data we opted to use a pre-trained ResNet18 model and transfer its already learned features to this task of sidewalk damage scoring. This process, known as transfer learning allows us, in a sense, to teach a model that already knows how to see to simply make meaning of the things it detects in our dataset's images.

The Results: After training for 100 epochs (iterations over the dataset) we saw the model successfully learn. However, it is questionable if the model was actually picking up on relevant features as the dataset the model was trained on is, as previously mentioned, tiny for an ML task. We do believe that our results lead to the conclusion that this is a worthwhile approach given additional time and resources.

The loss curve during training - this does not show generalization to new data.

Summary Of Our Model's Design

DeepLabV3 segmentation model's output fed into a fine tuned ResNet18 model. Ultimately classifies an image as damaged or not.

The team