screen capture of the BigDataVoie website showing the rail network and interactive graphics

Mapping railways maintenance

Over the course of 6 months, with a small Quantmetry team, I worked on a project for SNCF - the operator for the French railways. The goal of the project was to create a web application that allowed SNCF to better understand the effort dedicated to infrastructure maintenance:

  • cost of maintenance
  • amount and type of operations performed
  • evolution of the rail health over time (number and type of defects)

The website first shows an overview as a map of the network. On this map several indicators can be projected; each kilometer of rail has a color defined by the value of the chosen indicator. Furthermore, a detailed view can be accessed by clicking on a portion of rail several graphics are plotted that show the available data at the finer level of detail.

The project was well received by our client. It is probably one of my favourite projects while working at Quantmetry.

Challenges

The main challenges for this project were data variety and specifications.

We used data coming from several systems used by different business units at SNCF. Unfortunately the infrastructure is described differently by each business unit according to their need. For instance operators that repair local defects on the rails measure the defects position with a high accuracy whereas the financial department mostly cares about which account is in charge of repairing this section of rail. Merging together these very different data sources was a big part of the project but also one of the main results: it allows people from different backgrounds to share information in a single interface whereas they used to work with distinct applications.

Deciding what would be shown in the application was not easy. Firstly SNCF uses very specific indicators to measure the health of a given infrastructure. We had to make sur that the way we computed these indicators from raw data was conforming to their specifications. The second difficulty was making sure that we created easy to understand graphics showing detailed information. We decided to split the graphics in several categories which roughly correspond to different business units. We used existing graphics used in operations by our client as a reference to work from.

Technical details

The project is composed of different parts which work together as an application.

  • a data-processing pipeling was built (Python, Pandas) that loads, cleans and transform raw datasets in something that can be used by the rest of the application
  • some of the resulting datasets are exposed in a Postgre SQL database (PostGIS pluggin)
  • we used Geoserver to translate the PostgreSQL dataset into raster layers that show several indicators projected onto the railways network. Note that the Geoerver configuration files are dynamically generated from the Python pipeline
  • we built a simple Python API with bottle (I would use a more complete framework if I was doing this project again) to return data for the detailed graphics. As a matter of fact the API returns the data as well as a full description of the graphics as a JSON object using the Plotly specification.
  • the front-end was built with Angular 1.5; we used Leaflet to handle the map and Plotly to show the graphics.