Welcome to the first in the series of Find Data blog posts! Here we will take an in-depth look at the many and varied datasets which Find acquires, processes, loads and supplies in various ways to our many excellent and demanding (in a good way!) clients. We are of course partnered with the best and biggest data suppliers in the UK (OS, BGS, EA, Land Registry etc.) but just how do we take their often quite raw data and easily create consumable data feeds for our wonderful customers? Read on to find out more....
OS VectorMap District - 1:25 000 backdrop mapping
About the data
OS VMD (Ordnance Survey VectorMap District) is a free mapping dataset providing contextual backdrop mapping. The data can be displayed as vector or raster covering the whole of Great Britain. It is updated biannually in May and November.
The dataset is available to download online in its raw format as shapefile, GML or TIFF:
FIND downloads the full UK OS VMD dataset as GML feature type format and then loads it into a PostGIS enabled Postgres database.
OS VMD data is provided as WMS using Geoserver. Customer can then access to OS VMD WMS as contextual mapping background.
FME workspace process
To create a WMS (Web Mapping Service) feed of the OS VMD dataset from the raw GML files requires multiple loading and processing steps. Read on for the full step by step explanation of how we load OS VMD and make it available for all our customers.
To load OS VMD into our database, we made the choice to use FME. It is a powerful software to load, transform and display huge datasets such as OS VMD. OS VMD is converted from GML files to be loaded into the FIND Postgres database using FME.
First, we save all the GML files in the same folder to be able to load it in FME.
Load OS VMD data as one GML single merged feature reader in FME
Filter attribute by geometry (point, line, polygon) with GeometryFilter transformer
Filter attribute by feature type for each geometry to create 21 output tables with AttributeFilter transformer
Administrative Boundary- Railway Tunnel
Electricity transmission Line - Road Tunnel
Functional Site- Spot Height
Glasshouse- Surface Water Area
Motorway Junction- Surface Water Line
Named Place- Tidal Boundary
Ornament - Tidal Water
Railway Station- Woodland
Add Counter transformer to create ID column with unique number
Connect reader and writer attribute with AttributeRenamer transformer to create the right output table with right column, column name and attribute.
Load the output writer with the right column names and types into Postgres. This step is important. Column names and types must be same as recommended in OS documentation to be showed correctly the WMS.
Run the FME workspace. This takes around 9 hours.
Load OS VMD in Geoserver
Once OS VMD dataset has loaded we do a manual check in the Postgres database. Geoserver can then be used to display data as WMS to be accessible for customers.