The first in the series of Find Data blog posts - VMD FME Update

Welcome to the first in the series of Find Data blog posts! Here we will take an in-depth look at the many and varied datasets which Find acquires, processes, loads and supplies in various ways to our many excellent and demanding (in a good way!) clients. We are of course partnered with the best and biggest data suppliers in the UK (OS, BGS, EA, Land Registry etc.) but just how do we take their often quite raw data and easily create consumable data feeds for our wonderful customers? Read on to find out more....

OS VectorMap District - 1:25 000 backdrop mapping

1. About the data 

OS VMD (Ordnance Survey VectorMap District) is a free mapping dataset providing contextual backdrop mapping. The data can be displayed as vector or raster covering the whole of Great Britain. It is updated biannually in May and November. 

The dataset is available to download online in its raw format as shapefile, GML or TIFF: 

https://www.ordnancesurvey.co.uk/commercial/index.html

Styling is also provided with 2 different options, backdrop or full color:

https://github.com/OrdnanceSurvey/OS-VectorMap-District-stylesheets

VMD is provided, as with all OS mapping products, in OSGB36 (EPSG: 27700) coordinate referencing system. 

OS VMD data are divided for whole UK into squares 100km by 100km with unique two-letter reference. Each tile is then split into 20km by 20km tiles. 

Technical documentation can be find here: https://www.ordnancesurvey.co.uk/business-government/tools-support/vectormap-district-support 

2. VMD used by FIND mapping

FIND downloads the full UK OS VMD dataset as GML feature type format and then loads it into a PostGIS enabled Postgres database. 

OS VMD data is provided as WMS using Geoserver. Customer can then access to OS VMD WMS as contextual mapping background. 

3. FME workspace process 

To create a WMS (Web Mapping Service) feed of the OS VMD dataset from the raw GML files requires multiple loading and processing steps. Read on for the full step by step explanation of how we load OS VMD and make it available for all our customers. 

To load OS VMD into our database, we made the choice to use FME. It is a powerful software to load, transform and display huge datasets such as OS VMD. OS VMD is converted from GML files to be loaded into the FIND Postgres database using FME.

First, we save all the GML files in the same folder to be able to load it in FME. 

• Step 1

Load OS VMD data as one GML single merged feature reader in FME

 

• Step 2 

Filter attribute by geometry (point, line, polygon) with GeometryFilter transformer

• Step 3 

 Filter attribute by feature type for each geometry to create 21 output tables with AttributeFilter transformer

• • Administrative Boundary - Railway Tunnel
  • Building - Road
  • Electricity transmission Line - Road Tunnel 
  • Foreshore - Roundabout
  • Functional Site - Spot Height 
  • Glasshouse - Surface Water Area 
  • Motorway Junction - Surface Water Line
  • Named Place - Tidal Boundary 
  • Ornament - Tidal Water
  • Railway Station - Woodland 
  • Railway Track

• Step 4

Add Counter transformer to create ID column with unique number 

• Step 5 

Connect reader and writer attribute with AttributeRenamer transformer to create the right output table with right column, column name and attribute.

• Step 6 

Load the output writer with the right column names and types into Postgres. This step is important. Column names and types must be same as recommended in OS documentation to be showed correctly the WMS. 

• Step 7

Run the FME workspace. This takes around 9 hours. 

4. Load OS VMD in Geoserver 

Once OS VMD dataset has loaded we do a manual check in the Postgres database. Geoserver can then be used to display data as WMS to be accessible for customers. 

We always check the styling, because OS periodically update it, it is highly recommended to load the new VMD styling whenever OS VMD is updated: https://github.com/OrdnanceSurvey/OS-VectorMap-District-stylesheets

• Step 1 

Open the .SLD style file in notepad and have a look at the column name to confirm that is the same as your new OS VMD layers (be careful with capital letters and data types) 

Load all 21 new stylings in Geoserver. 

• Step 2 

Load 21 OS VMD layers from Postgres. Tick “compute from native bounds” to update the coordinate system and load relevant style previously load. 

• Step 3

Visualize each OS VMD layers with OpenLayers in Layer Preview menu. This step allows to check if layers are displayed properly using the relevant style. 

• Step 4

Update or create a Layer Group with OS VMD layers and styles following OS recommended layer ordering.

OS VMD is now ready to go as a WMS feed and available to stream in the normal fashion using a simple URL.

Halloween Antics @Find HQ

Things were quite spoooooky here at Find HQ yesterday

The day started off well, all colleagues were joining in the fun...

 But things took a terrifying turn when all the chocolates were eaten...

• We witch you a Happy Halloween, and hope it doesn’t suck like a vampire!!