CycleStreets blog

As previously announced, we are working with the UK’s Department for Transport to make advanced cycling data attributes available for incorporation into OpenStreetMap.

Rather than organising this along the lines of a bulk import, we are taking advantage of new technologies in Potlatch 2 and have commissioned Andy Allan, creator of OpenCycleMap, to develop new features to allow volunteers to collaborate on inspecting and merging the information into OSM.

This merging tool will also be of use for other external data that could be manually inspected and merged into OpenStreetMap.

Background

The DfT commissioned survey work in various cities around the UK for their Transport Direct product. In 2011 they released the results of the surveys as Open Data, in a complex GML format based on Ordnance Survey ITN data – unsuitable for use with OSM. However, in addition, they have funded work to convert the survey data to be based on OSM geometries suitable for incorporation. This has been done through brilliant work by Ralph of CCG.

The kinds of things surveyed include cycle routes, cycle parking, cycle lanes and their widths, surfaces widths and lighting conditions of cycleable paths, and so on. We are working to add support for these attributes into CycleStreets, so that routes are further improved.

In the UK wide areas of the cycling infrastructure have been mapped in OpenStreetMap, often more recently than the data from the DfT. Also, with the development of Vector Background layers in Potlatch 2, there was an opportunity to create an improved process for dealing with external datasets.

Further background information is available in blogs and on the mailing lists.

• Initial announcement of the project
• Update on the progress of the tool
• Talk-gb thread on the project

The demo

We’re pleased to announce that a demo is now available, and we’d like people to test it.

A demo is now available. It contains sample data for Nottingham and Cambridge, but it’s deliberately unable to save the data back to the main OSM server. When the final version of the data conversion is complete and available, this will be updated and fully able to work.

Two test areas are currently loaded:

• Cambridge
• Nottingham

How to use the merging tool

The merging process works as follows:

1. Click ‘Map style’ > ‘Wireframe’ to make things much easier to work with.
2. The background data is highlighted either orange (needs attention) or blue (already processed).
3. Click the background features to select them.
4. Ctrl+click (or cmd+click on a Mac keyboard) the relevant OSM feature (line) to see a side-by-side comparison.
5. Click on ‘Advanced’ in the left panel to see the merging controls.

Feel free to play around with this – the snapshot data is being reloaded from time to time as we get better imports, although we think we’re almost at a stage where the data conversion is fairly bug-free.

The merging tool is currently a beta and further improvements are planned. See the main Merging tool page on the OSM wiki.

The first screenshot shows the thick gray line (DfT data, as a background layer) highlighted. It shows the attributes it has:

The second screenshot shows what happens if we now control-click (or cmd+click on a Mac) on the OSM line – we now get a merging interface where we can accept/reject each attribute, and click the button at the end to accept all the changes:

Feedback on the data

We would really welcome feedback as to any errors you spot in the data conversion. The aim is that the data is pre-processed and snapped to the OSM geometry as effectively as possible, so that merging is merely a case of manual confirmation of each attribute according to your local knowledge.

Issues we have fed back so far on are:

• Alignment. The data was originally snapped to an OS Open Data, and has been geographically aligned via advanced GIS techniques to OSM. It’s already well over 90% matching and further improvements are being made.
• The issue of streets being broken up but having the same data. Our GIS contact plans to merge when the street name and data matches.

The software

A number of software components are used to make all this work

• Potlatch 2 is used as the editor, and can load data from both OSM and the DfT data. The splash pages and other resources are available on github
• Snapshot Server is used to serve the DfT data for each user, saving them from having to load the whole country at a time
• Some scripts are used for loading data in and out of the server. These use Osmosis to read/write between XML and Postgres.

License

The data is expected to be released under the Open Government License. We have been seeking an early letter of confirmation from the DfT on this and will update this page and the OSM Wiki accordingly. (The ITN-referenced dataset is released under the OGL already.)

Feedback

We’d really appreciate it if you could try out the beta and add comments below, or contact Andy Allan with any feedback you have. Did you figure out how to use the tool? Did you manage to merge some data? What doesn’t work? How could the tool be improved?

If you have local knowledge of the areas in question, it would be great to hear back from you on the datasets themselves – do they match reality? Are the tags appropriate?

Back in July we wrote about our work to assist the DfT with the opening up of their cycling data, a dataset featured in the Telegraph this weekend. Over the summer, much progress has been made.

Andy Allan, who is perhaps best known for his excellent OpenCycleMap project, has been working for us to create a merging tool (within Potlatch 2) for helping to combine external datasets into OSM on a street-by-street basis. This tool will be useful not only for the DfT data that is becoming available soon, but also for other datasets.

The DfT data, which has mostly been collected by surveyors on bicycles, has the potential to significantly improve the quality of routing in some areas of England. We are well aware, however, that data collected by other agencies can undermine the work of OSM volunteers in the area if not handled sensitively, and so we've stressed that automated, bulk imports would not be accepted by the OSM community.

Instead, the approach taken is a method for OSM volunteers to inspect and merge the information on a street-by-street and attribute-by-attribute basis via the simplest and quickest means possible, using their local knowledge to enhance the end result.

Surface type, cycle lane widths and other data are amongst the attributes in the dataset, and these will shortly be supported in the routing engine.

The code for the beta release of the tool [included in this diff] has been reviewed and merged into the main Potlatch 2 codebase, and is already in the latest builds. We expect further changes to be made as we get feedback from initial testers.

OSMers should be able to try out the beta and the data soon as we have confirmation of the Open Government License for the data. Here are screenshots of the beta:

We're pleased to announce the launch of a major new section of our website:
Points of interest that you can click on to view and select for your journey.

Points of interest

We've integrated OpenStreetMap location data so that you can now click on points on the map. Just go to our journey planner tab and you can use the presets like bike shops, cafés, pubs, and many more.

(Mobile app developers: This data is also available through our API – see below.)

For instance, stations:

If you select a point, it is clickable. A Google Street View picture of the location will be shown, if it's a roadside location:

There's a link in the popup to the place's website if it has one.

There are lots of different POI types available:

You can browse locations anywhere the UK, for instance to find these independent bike shops in London:

Please add locations to the map!

If a place you know of doesn't appear in the map, please contribute your knowledge to OpenStreetMap by using the Edit section of our website. Warning: it can be quite addictive!

Or perhaps no-one's added a location's website yet? Click on the 'add it' link in the popup shown above. Follow the link, click on the icon, click on 'Advanced' and then enter 'website' on the left and the URL on the right, and click Save. You'll need to create an OpenStreetMap account if you don't have one already.

You must not copy things from other people's maps, however – additions and edits must be based on your local knowledge of an area.

API

This data is now all available through our API so that it can be integrated into your cycle routing app.

For full details, see our API documentation.

(We hope to have this functionality in our own apps shortly. If you can help patch it in, please branch our app repo and pitch in!)

CycleStreets makes heavy use, as many people will know, of OpenStreetMap data as the basis of our street/path network for route planning. For us, the ability of the community of cyclists and others to contribute to this data is an important part of the effectiveness of CycleStreets.

But, lesser-known, is that we also make use of several OS OpenData datasets for various parts of the site.

OS OpenData is part of the many datasets offered by the Ordnance Survey. But unlike many commercial datasets, they are open data, allowing not-for-profit groups like ourselves to use them. The first release of OS OpenData was announced on 1st April 2010, a decision by the government and the OS that has been widely welcomed.

We thought we'd outline the datasets we use in various ways.

Firstly, the postcode database, Code-Point Open®. This enables us to translate postcodes to a point on the map. We were the first users of this open data – implementing the postcodes within hours of it being released. MySociety have helpfully converted the Code-Point Open dataset into latitude/longitudes rather than grid co-ordinates.(Code-Point Open is not to be confused with the Postcode Address File (PAF), which is a commercial product of Royal Mail, and which we do not have funds to purchase. The PAF enables individual house/address locations to be accurately located.)

Secondly, we use the Boundary-Line™ dataset to enable photos added to the Photomap to be assigned automatically to London boroughs and soon other boundaries around the UK. This dataset will also come in useful soon for our Cycle campaigner toolkit project.

Next, the OS Street View® dataset is being used by some OpenStreetMap volunteers to augment areas of OSM in the UK whose level of completion of street coverage is lacking.

Lastly, we're using a projection of the Street View dataset in our Android app and forthcoming mobile web site.

The OS OpenData datasets have enabled us to improve the facilities on CycleStreets. The postcode data in particular has been invaluable. It has helped us as a community organisation create a useful facility at low cost.

Update, Sunday 31st July: The Street View tileset is also now available as an option on the main site – just click on the blue (+) button in the top-right corner of the map panel:

PS Steven Feldman: here's one for you: OS OpenData™ :)

We're aiming with CycleStreets to provide the highest possible quality cycle routing, to give people trust in routes they plan. We've heard from many users how our routing is helping them give the confidence to use a bike for their journeys, and from people who've discovered cut-throughs and safer, easier routes for their existing journeys.

Increasing the quality of the routes found by CycleStreets means using more sources of good quality data. For instance, a cycle lane can improve a planned cycle journey, but not if the cycle lane is too narrow. On the other hand if the cycle lane is wide and has a good surface, it can be better than a shorter route on a busier road.

The information that CycleStreets uses to base its route recommendations comes primarily from the OpenStreetMap (OSM) project. Over time that data has become more detailed in both depth and breadth, and it continues to do so.

Over the last 18 months, the UK's Department for Transport (DfT) has undertaken a GPS-based survey of cycling infrastructure in towns and cities around England. This has been used for a related project, the Transport Direct multi-modal journey planner.

The DfT is keen to see this data used more widely and we've been talking to them about using it in our routing, by making it available as open data that could be merged into OpenStreetMap.

We're delighted now to announce that we're helping the DfT with its laudable objective to make this data more widely available. We’re working with its contractor, CycleCity Guides, who are well-known for producing a wide range of Local Authority cycle maps. The release of this data is one of a number of other datasets that the Cabinet Office has recently announced will be made available.

Rather than merely dump the data on data.gov.uk, the DfT is going a step further to help it be used, a development it should be highly commended for.

Respecting the way the way the OpenStreetMap community works, the DfT is planning to:

• Make the data available in a fully OSM-compatible format, aligned to OSM geometry with converted attributes.
• Simultaneously publish a dataset aligned to Ordnance Survey's (OS) Open data
• Use a standard, OSM-compatible license (the Open Government License), with the data unencumbered by OS derivative data issues.

This data, which has mostly been collected by surveyors on bicycles, has the potential to significantly improve the quality of routing in some areas of England. We are well aware, however, that data collected by other agencies can undermine the work of OSM volunteers in the area if not handled sensitively, and so we've stressed that automated, bulk imports would not be accepted by the OSM community.

Instead, useful data needs two things if it is to be used in OSM. Number one is a way of inspecting and accepting/rejecting the data on a street-by-street basis via the simplest and quickest means possible. Secondly encouraging routing engines and renderers to use the data. Therefore:

• Funding we've obtained will pay for a month or two of solid work on Potlatch 2, the default editor on the OSM website. We've engaged Andy Allan, one of Potlatch 2's core developers, for this. The funding will lead, amongst other improvements, to a generic tool to enable donated data to be merged in (or rejected), street-by-street via manual inspection and approval. A range of general usability improvements (such as those in the P2 buglist) will also be funded.
• We'll be implementing support for many more advanced routing attributes, which Andy and hopefully other OSMers will be helping with. This will demonstrate the difference that really detailed data can make to the quality of cycle routes found by engines like CycleStreets when the community merges in (by inspection) this type of data.
• A range of other improvements will also be made, for instance, changes to our feedback system so that errors in OpenStreetMap, found as a result of people using the routing, can be more easily discussed and fixed in OSM.

We hope the OSM community will react positively to these developments.

With community support, this data should help get lots more useful data into OSM and help it become a superbly detailed dataset ever more quickly.

We've been particularly impressed at the way that our contacts at the DfT have been open to learning about the way the OSM community works. We particularly hope that the success of this project will act as a demonstration and lead to more trailblazing open data initiatives where government learns from existing communities to 'do open data the right way'.

We've added a few new features to our www.LondonCycleHire.org website, a version of the journey planner that includes the Barclays Cycle Hire ('Boris-bike') points.

We've added live availability data, thanks to TfL's new data feed – thanks TfL for making this data open! (Disclaimer: as a third-party site, it is not endorsed by TfL).

We've also added Street View images from Google so you can familiarise yourself with the area before making a trip.

The popup links for each location enable you to 'Choose this point' as a start or finish location.

And as with our main website, you get a choice of directions and photos-en-route, brilliantly detailed data from OpenStreetMap, plus other features.

Check out www.LondonCycleHire.org for your next borisbike journey!

We’re pleased to announce that our bid, for a comprehensive online campaigning toolkit to assist cycle campaign groups around the UK, is a winner in the GeoVation contest!

It brings £27,000 for the development of a toolkit which, in the words of one supporter, should be “a hugely important step forward for all cycle campaigning groups”.

Our bid was one of 155 ideas submitted to the GeoVation challenge, on the theme of “How can we improve transport in Britain?”. Our bid was shortlisted, and we attended the GeoVation Camp in March to help develop the proposal amongst a total of 30 ideas invited. We were one of the final ten proposals, and took part in a Dragon’s Den -style pitch on Wednesday.

We were delighted to be picked as one of the winners who share the £150k pot of funding.

   

Photos by Ordnance Survey, licenced CC BY-NC 2.0

Martin Lucas-Smith, who presented the bid alongside CycleStreets’ routemaster, Simon Nuttall, said:

“We were delighted to be picked by the Ordnance Survey’s judges as one of the winners. The £27,000 of funding will enable us to get this much-needed project off the ground.

“As a member of one of the many local cycle campaign groups who will benefit, I’m all too aware of the large number of issues on the street network that need improvement, and the difficulty of managing this deluge of problems.

“The new system will help campaigners around the country convert these problem reports into prioritised, well-evidenced solution proposals. It should help them work more productively with local councils to see changes implemented.”

We’d like to thank all the groups who provided quotes of support for our bid, including the CTC, Cyclenation, London Cycling Campaign, and a variety of groups around the country. We’re working to provide you with a really great, useful and user-friendly system that will save a lot of time and effort.

Some of the things the new system will be able to do are:

• Enable members of the public and campaigners easily to pinpoint where cycling is difficult
• Help groups prioritise what to work on
• Pull in planning application data automatically, so that potential issues needing attention are readily accessible
• Automatically notify and involve people who cycle through an area – who therefore have an interest in seeing issues fixed
• Make geographical data such as collision data and accessibility analysis easily available, to provide context
• Enable simpler and more focussed discussion based on specific issues, groups of issues, or themes
• Enable best practice to be ‘pulled-in’ to discussions, by providing off-the-shelf examples shared from elsewhere in the UK
• Enable groups to include LA contacts in these discussions if they wish
• Enable groups to assemble ‘solution’ resources so that problems can be resolved on the ground
• Give groups a variety of ways of publishing their activity on their website easily.