CycleStreets blog

About three weeks ago we opened up Cyclescape to a test group, Cambridge. Their Committee have been using the system instead of the e-mail list they used to use.

On the right, you can see a planning application being discussed, with the location of the site marked out, and a list of the discussions.

In this three-week period there have been:

• 78 issues (problem reports and things to work on)
• 108 discussion threads (discussions connected to the issues)
• 789 messages (individual replies within the threads)
• 61 user locations (areas that people have set to be informed about changes in)
• 22 campaigning deadlines and dates
• 34 library items
• 151 pieces of feedback about the system, some of which are about tiny fixes, others about more substantial issues (most of which we know about)

Although there are unfinished areas (which have caused some user confusion) and some missing features, the testing has given us plenty of useful feedback, which we’re working through before we open up the system for wider use.

How are people using the system?

We’ve seen discussions on, for instance:

• Various local planning applications – on which it’s been useful to see the location on the map
• Smaller issues around a city
• The Times “Cities Fit For Cycling” campaign
• Administrative discussions (organising day-to-day things)
• Organisation of a cycle ride – which again the geographical location provides useful context
• Addition of some items to the resource ‘library’ which will be useful for future discussions

There are signs that the ability for people to subscribe only to things they’re interested in, based on geographical location(s), should work well – so that people can be involved in things they care about – but not overwhelmed with issues.

We’ve seen some confusion over the difference between ‘issues’ (problems) and ‘threads’ (discussions of those issues), which further interface work will address.

Key themes

Some of the key themes, on things we need to fix, are:

• The need to reorganise the various listings screens, which are currently not in their final positions. This is an area we knew would need quite a bit of iteration work.
• Making sure that maps are viewable in a few places where they’re currently omitted.
• Reducing the ‘friction’ between issues and threads. We want to encourage people to locate issues geographically when possible. To some extent this will be improved naturally as the listing screens get reorganised and as we fix up parts of the design that were not finished when we went live.

• Various layout fixes to add some more polish and improve usability.
• Further work on e-mail integration. Many areas work, but some kinds of notifications can’t be replied to by e-mail yet, and it’s not yet possible to start a thread by e-mail.
• Very active campaigners have asked for an ‘opt-in to everything by default’ setting, so that they get told about everything, and can opt-out when not interested (rather than merely getting told about something new in their area). This would save them having to subscribe to each issue or discussion thread as it comes in.
• There have also been some compatibility problems with a certain web browser… Internet Explorer!

An example problem that we need to fix – city-wide issues are overlapping and obscuring others, so a fundamentally different approach for city-wide issues might be needed, e.g. a separate listing screen or maybe a button to show these:

Usability testing

We’ve also done some individual usability testing, which has thrown up 35 actionable issues. Around half of these were things known about and planned to be fixed, but the others provided useful insight.

Next steps

We’re busy working on fixes for the above, as well as creating more guidance on installing the system for coders who want to get involved. Technical people can watch progress in our Github repository.

Stay tuned to the blog in the coming month as we talk about the latest changes and showcase the system’s features. We’re working as fast as we can, within the available funds, to get the system ready for wider use!

We’re holding a CycleStreets for Android Hack Day on Saturday 25th February 2012. Come along, because it’ll be fun.

Dive in

So what might you do with the Android app? Well, anything you like, but here are some suggestions – export a CycleStreets route as GPX, integrate a vector mapping library like MapsForge, add speech output, or simply poke around in the code and see what catches your eye. Learn some new things, ask questions, maybe have a beer in the evening.

What you’ll need

• A Laptop. Linux, Windows, or Mac, whatever your preference.
• The CycleStreets for Android source code. The source code lives up on GitHub, so grab a copy by forking it. If you don’t already have a Github account, you will need to create one. You’ll also need need a copy of Git itself. If you’re unfamiliar with Git, don’t worry because we’ll give you a primer. If you’re unfamiliar with source code control in general, you still don’t need to worry, we’ll teach you a valuable life skill.
• The Android SDK. To build the code, you’ll need the Android SDK. Strictly speaking the SDK and a text editor are more or less all you need, but you will probably find this a little easier using it with the Eclipse IDE. The integration with Eclipse is very good, the code completion support will help you work your way through an unfamiliar API, and the debugging support is excellent. See the image below for the components you’ll need.
• An Android device. Actually this is optional, as the Android SDK supports Virtual Devices you can run on your laptop, but it’s quicker and easier to run your code on a real device.
• 3G dongle if you have one – there’ll be internet, but bring a dongle just in case we have problems.

What you won’t need

• Previous Android Experience. Programming for Android is actually pretty straightforward, so you don’t need any specific experience. Familiarity with event-driven user interfaces is helpful, but if you’re keen you should be able to pick it up pretty easily.
• Java Expertise. The language of Android is Java, but you don’t need deep, deep Java expertise. You can get by with a smattering.

When and where?

Saturday 25th February 2012 from 12 noon – 10pm.

Ross Street Community Centre. [Map showing the route from Cambridge station - green route is best.]

It’d be great if you could let us know if you can come so we can order enough fresh pizza!

The bits of the Android SDK that you need, shown ticked (click to enlarge):

A new version of our Android app has been released, bringing with it our most-requested feature – waypoint routing!

The full list of new additions is:

• A to B to C routing (waypoint/via-point routing)
• Changes to Add Photo to prevent duplicate uploads
• Routing finding should now work on all 3G networks

Thanks as ever to volunteer Jez Higgins who has done great work on the app. Keep your ideas for new features coming!

Get it now – it’s free!

We’re working heavily at present on our open-sourcing initiative.

Already our mobile apps and the Cyclescape project are open-source:

• iPhone (iOS) app
• Android app (on which we’ll be having a hack day soon – possibly in February)
• HTML5 mobile web app
• Cyclescape campaign group toolkit (plus deployment)

If you can help with any of those, or find the code useful, please do get involved!

The main site is not yet open-sourced, but work is in progress.

Our key aim here is to make it much easier to get other people involved. Over the last year we’ve had a number of enquiries from fellow cycling programmers about getting involved, but there have been very confusing aspects of the codebase – mainly very old code that had not been upgraded, and the setup has been difficult to say the least.

So our work at present is focussed on:

• Modularising the code much more, so that – for instance – people wanting to work on the interface don’t need to get a Python/C++ -based routing engine installed
• Setting up and securing a journey planner endpoint for the same reason (so that a repository download can work without the routing engine installed)
• Untangling key bits of code that have long been very problematic, even for us
• Splitting up classes further to reduce dependencies (currently there are over 150 classes and this is growing by the day – see screenshot below)
• Re-scripting various installation routines
• Making things more self-documenting
• Parametrising site-specific switches  in the code related to specific non-core sites that we are paid to run
• Migrating to Git and then Github (we currently use SVN)
• Deciding whether any sections should be closed-source

We’re doing these before open-sourcing because – realistically – we simply do not have the time resource at present to hand-hold people through an installation process that is currently more difficult than it should be. We know from our experience from our hack day back in 2010, that any barriers here will stifle involvement. This is also forcing us to deal with some of the nastier architectural design flaws that have constantly held back some work.

Much of the code is actually in a reasonably good state, however. It runs on Windows, Linux or Mac OSX without changes (though getting the core routing component running on Windows might be interesting!). It is mostly in PHP5 (error_reporting=2047), written in a traditional class-interaction format (not MVC), though there are many established design patterns such as front controller pattern, a central application resource, common data formats, common class extension principles, all database interactions through an abstraction layer, etc.

A few key issues for us, once open-sourced, are:

• Whether we should aim to push people towards running a single, central installation, and encourage people from abroad to contribute funds and code towards that
• The extent to which we can realistically give any support for other sites (given that we’re pushed as it is to get everything done)
• The use of the name ‘CycleStreets’ if we give any support for sites we don’t control
• Maintaining quality-control of hosted sites that we are paid to run
• Ensuring that the Photomap isn’t closely-coupled with the journey planner
• Addressing the lack of test coverage
• Internationalisation work

We’d welcome any feedback (in the comments below) on these points. For us the key problem is time – and funding if the system is to expand to other countries!

We don’t yet have a timescale, but the code cleanup is going quite fast now, and in fact is accelerating, as the removal of hacks is making things quicker to deal with.

Screenshot of our current classes, where most of the code is:

As previously announced, we are working with the UK’s Department for Transport to make advanced cycling data attributes available for incorporation into OpenStreetMap.

Rather than organising this along the lines of a bulk import, we are taking advantage of new technologies in Potlatch 2 and have commissioned Andy Allan, creator of OpenCycleMap, to develop new features to allow volunteers to collaborate on inspecting and merging the information into OSM.

This merging tool will also be of use for other external data that could be manually inspected and merged into OpenStreetMap.

Background

The DfT commissioned survey work in various cities around the UK for their Transport Direct product. In 2011 they released the results of the surveys as Open Data, in a complex GML format based on Ordnance Survey ITN data – unsuitable for use with OSM. However, in addition, they have funded work to convert the survey data to be based on OSM geometries suitable for incorporation. This has been done through brilliant work by Ralph of CCG.

The kinds of things surveyed include cycle routes, cycle parking, cycle lanes and their widths, surfaces widths and lighting conditions of cycleable paths, and so on. We are working to add support for these attributes into CycleStreets, so that routes are further improved.

In the UK wide areas of the cycling infrastructure have been mapped in OpenStreetMap, often more recently than the data from the DfT. Also, with the development of Vector Background layers in Potlatch 2, there was an opportunity to create an improved process for dealing with external datasets.

Further background information is available in blogs and on the mailing lists.

• Initial announcement of the project
• Update on the progress of the tool
• Talk-gb thread on the project

The demo

We’re pleased to announce that a demo is now available, and we’d like people to test it.

A demo is now available. It contains sample data for Nottingham and Cambridge, but it’s deliberately unable to save the data back to the main OSM server. When the final version of the data conversion is complete and available, this will be updated and fully able to work.

Two test areas are currently loaded:

• Cambridge
• Nottingham

How to use the merging tool

The merging process works as follows:

1. Click ‘Map style’ > ‘Wireframe’ to make things much easier to work with.
2. The background data is highlighted either orange (needs attention) or blue (already processed).
3. Click the background features to select them.
4. Ctrl+click (or cmd+click on a Mac keyboard) the relevant OSM feature (line) to see a side-by-side comparison.
5. Click on ‘Advanced’ in the left panel to see the merging controls.

Feel free to play around with this – the snapshot data is being reloaded from time to time as we get better imports, although we think we’re almost at a stage where the data conversion is fairly bug-free.

The merging tool is currently a beta and further improvements are planned. See the main Merging tool page on the OSM wiki.

The first screenshot shows the thick gray line (DfT data, as a background layer) highlighted. It shows the attributes it has:

The second screenshot shows what happens if we now control-click (or cmd+click on a Mac) on the OSM line – we now get a merging interface where we can accept/reject each attribute, and click the button at the end to accept all the changes:

Feedback on the data

We would really welcome feedback as to any errors you spot in the data conversion. The aim is that the data is pre-processed and snapped to the OSM geometry as effectively as possible, so that merging is merely a case of manual confirmation of each attribute according to your local knowledge.

Issues we have fed back so far on are:

• Alignment. The data was originally snapped to an OS Open Data, and has been geographically aligned via advanced GIS techniques to OSM. It’s already well over 90% matching and further improvements are being made.
• The issue of streets being broken up but having the same data. Our GIS contact plans to merge when the street name and data matches.

The software

A number of software components are used to make all this work

• Potlatch 2 is used as the editor, and can load data from both OSM and the DfT data. The splash pages and other resources are available on github
• Snapshot Server is used to serve the DfT data for each user, saving them from having to load the whole country at a time
• Some scripts are used for loading data in and out of the server. These use Osmosis to read/write between XML and Postgres.

License

The data is expected to be released under the Open Government License. We have been seeking an early letter of confirmation from the DfT on this and will update this page and the OSM Wiki accordingly. (The ITN-referenced dataset is released under the OGL already.)

Feedback

We’d really appreciate it if you could try out the beta and add comments below, or contact Andy Allan with any feedback you have. Did you figure out how to use the tool? Did you manage to merge some data? What doesn’t work? How could the tool be improved?

If you have local knowledge of the areas in question, it would be great to hear back from you on the datasets themselves – do they match reality? Are the tags appropriate?

We’ve had a busy year, and we’d like to thank everyone who has helped out with the CycleStreets project – whether that’s helping with development, design, promotion and word-of-mouth, mobile coding, and more.

As you can see, since our launch in March 2009, usage has grown very considerably this year.

We’ve planned almost 1.4 million cycle journeys, covering some 23 million km as the crow flies, equivalent to cycling to the moon 60 times!

It’s been incredibly heartening to see so many great comments from users of CycleStreets on how it’s helped them find better routes or start cycling.

Major projects this year

The year for us has been dominated by a number of funded major projects.

Work for the bicycle industry to create a Leisure Routing mode for the great cycle satnav app, Bike Hub (which uses our routing). Leisure routing is a new kind of routing, exclusive to the app, which enables you to plan a circular route lasting a specified time or distance. Read more about this great new addition to an already brilliant app.

Work with the Department for Transport to open up their cycling data. We’ve been helping the DfT with their very laudable objective to open up the cycling data they collected as part of their Transport Direct system. This detailed cycling attribute data has the potential to enhance the quality of cycle routing, by adding things like surface quality, cycle lane widths, missing pieces of data. We’ve funded some development work on a merging tool for the main OpenStreetMap editor, Potlatch 2, which we hope will be of wider benefit for other datasets also. The data is now almost ready for release following a lot excellent work by the DfT and its main contractor – stay tuned.

Cyclescape, a comprehensive online campaigning toolkit to assist cycle campaign groups around the UK. We’re able to create this thanks to funding from GeoVation. Earlier this year we won a Dragon’s Den -style contest to create a website that would build on our Photomap of 30,000 images, and provide a way for cycle campaign groups around the UK to work more effectively to solve cycling infastructure problems in their area. The website will be launched shortly, and we’re moving into beta with it after months of solid work. You can read more about this on the Cyclescape blog.

Work with Cycling Scotland to improve cycle journey planning in Scotland. We’ve been undertaking a range of projects to help motivate improved data collection and usage. This work will be finished shortly, and will see a journey planner on the Cycling Scotland website and a new guide to assist local communities with data collection for OpenStreetMap.

Hosted journey planners for London Cycling Campaign, West Sussex County Council, Bike Hub and others soon to launch! Read more about our offering of customised cycle journey planner sites and see the brochure for Local Authorities we created.

Other projects

As well as these funded projects, we’ve been working on several other areas of our core offering:

Continual improvements in route quality: We’re continually tweaking the routing engine to make the routes better and better, partly in response to the great feedback we get from users. In the summer we added partial support for surface quality data (something we’ll be building on soon), and earlier we undertook some work to make routes more ‘sticky’ to the Sustrans and Local Authority cycle networks where they exist.

Faster routing: The routing engine is now much faster than a year ago. Even when we’ve had a load spike, the hosting has barely flinched!

Mobile apps on all the major platforms have been released. As well as our free iPhone app, which has seen many improvements over the year, we’ve released a well-received and fully-featured Android app and a Mobile web site which works on Blackberry and other platforms. All of these are open source projects, and we encourage people to join our brilliant volunteers who have been working on the apps so far.

A lot of work on resilience: Simon has worked extremely hard since the summer to improve the resilience of the system in many ways: Backup, fallback, easier importing of data, scripting various administrative tasks and so on.

Points of Interest: We’ve got a new set of webpages and an API for points of interest (shops, cash machines, leisure locations, etc.). We hope to integrate this much more closely with other parts of our offerings in coming months.

Lots of other little changes: Addition of CO2 and calorie counts, an increase in the journey planning limit to 200 miles, little usability fixes, reworking of the Photomap pages, railway station codes in searches, new Photomap listing modes, promotional flyers you can order, clickable icons for the London Cycle Hire website, a ‘cycle to us’ link facility, an integrated map editor installed, etc.

Photo of the day: We hope you’ve enjoyed our Photo of the day feature, if you’ve been following us on Twitter!

What about next year?

The main areas we’re going to be working on are:

• Support for advanced routing data (work in progress) according to an advanced ruleset we’ve been considering
• Drag routing (waypoints in the interface)
• Expanding to other countries
• Open sourcing (the main barrier at the moment being lack of modularity in older parts of the code)
• Reworking the feedback system so that we can get lots of OpenStreetMap volunteers involved

OpenStreetMap – a solid foundation

CycleStreets wouldn’t have been possible without the brilliant OpenStreetMap (OSM) project, whose data we use. OSM has gone from strength to strength, with many areas of the country seeing far more detailed data than this time last year.

It’s a volunteer project which anyone can contribute to. So whether there’s a newly-added cycle lane near where you live, a bike shop has opened, or whether the surface of a path is unsuitable for cycling, you can contribute that information – just edit away.

If you’ve appreciated the work of the people who’ve created the map, please donate to OpenStreetMap to help keep it running fast.

This is a guest post from Carlton Reid, executive editor of BikeBiz.com and the editor of BikeHub.co.uk. He writes here about the great new facilities in the Bike Hub app, which uses our routing and OpenStreetMap POIs feed.

The latest version of the Bike Hub cycle satnav app is 2.2; I’m currently testing 3.0, due for release soon. The new version will be packed with extra features, many of them suggested by app users.

I’m conflicted by all the additions. On the one hand I’ve always wanted to keep the app plain, simple, focussed. But users have said they want it to be feature rich.

2.2 – released at the end of October – was enriched with a GPX download feature so plotted routes could be emailed to a user for later use, perhaps in a route visualisation programme, such as Google Earth.

I’ve used this feature a couple of times but it’s not what I’d consider the app’s core function: this core function is to be a cycle-specific satnav. We have also been asked to include other ‘performance monitoring’ features but there are many other apps out there that do this well.

It’s a fine balancing act: making the app feature-rich but not so complex that it becomes hard to use or worse, buggy.

The app is a standard satnav in that it routes from A to B. Version 3.0 adds what could be a world exclusive and that’s A to A routing. This makes the app into a touring tool: arrive at a railway station with your bike, fire up the app to guide you on a three hour trip sticky to country pubs and off you go.

Part of this functionality was added to 2.2. You can navigate to and from Points of Interest: ATMs, places of worship, parks, castles and many other PoIs.

Version 3.0 uses this growing database as a tour suggester.

Also in version 3.0 – and a much requested feature – is dynamic route recalculation. Stray from the route and the app will suggest you make a u-turn. Stray a bit further and the app will create a new route, on the fly. This is standard on car satnavs, and it’s finally on Bike Hub version 3.0, thanks to app coding work from app developer Tinderhouse and routing tweaks by CycleStreets.

Version 3.0 also has tighter integration with CycleStreets, adding a box to input a CycleStreets journey number. This will be useful if you prefer to pre-plan your routes via desktop and then want to transfer the same route to your smartphone.

The most requested feature from users is added in version 3.0. This is map cacheing. Map tiles can be downloaded to a smartphone when in range of wifi or a good 3G signal. Users will still require a phone signal to call down the route from CycleStreets but maps – which can take a while to download in a poor signal area – can now be stored locally. (And deleted later, if wished).

I know I’ll use this feature for cycle tours in Northumberland.

The app has evolved greatly since launch and will continue to evolve, thanks to funding from the Bike Hub levy fund. If you have other features you’d like to see added, get in touch. editor at bikehub.co.uk

Carlton Reid

Jez Higgins is the lead developer of our Android app and has been working on lots of useful new features. He writes:

The new update to the CycleStreets Android app was released to the Android Marketplace this week. If you already have the app installed, it should update itself automatically. If you don’t have it installed, you can read about it in an earlier blog post and download it from the Android Market place here – for free.

This updates adds:

• New Points of Interest main menu option. Show points of interest, including cycle parking, pubs, cafes, post boxes, and London Cycle Hire points, on the route map. Choose the icon size from the settings menu.
• New main menu option Open route number. Want to load a journey you planned on the website? Just type in the journey number and off you go. Journey number is also shown on the itinerary tab.
• Saved routes can be renamed. Press and hold on a saved route to bring up the menu, then choose Rename.
• CO₂ and calorie savings: Itinerary now includes the estimated CO₂ saving from cycling rather than driving, along with the calories used on the journey.

Please do give a review of the app in the Android Marketplace! (Click on ‘User reviews’ then ‘Write a review’).

The Points of Interest menu option brings up the POI selection menu. You can choose as many different categories as you like, and they will be overlaid on the route map.

Tapping on a POI icon brings up a bubble with more information. Tap the icon again to dismiss the bubble, or tab bubble itself to start or end your route at the POI.

You can choose the size of the onscreen icons from the Settings screen.

The Points of Interest facility uses our POIs API available to mobile apps. It’s basically a convenience function replicating the same data in OpenStreetMap. We’ve focussed on practical location types in this app, as our main focus is on everyday, practical journeys.

Every route planned by CycleStreets is given a unique number. Journey number 1332563, for example, guides you from Euston Station to the Natural History Museum. Perhaps you planned a route on the website you now want to open in the app, or maybe someone emailed the route to you? Whereever that number came from the Open route number menu option lets you type it in open the route with the app.

If you’d like to help out with our app, perhaps to add new features, do grab the code and hack away!

We’ve recently completed a switch over of the whole CycleStreets system to upgraded hosting. Our previous primary server has been extremely reliable and, having just checked, I find it has been up for 795 days, and has only been rebooted once in the last two-and-a-half years.

The new primary server is needed to support expansion of the journey planner, to handle the growing amount of data we manage in the CycleStreets system and to provide increased reliability of our services,  as well as for our new project, Cyclescape. We now also have additional redundancy.

The switchover has turned out to be a rather more complex procedure than expected as there are quite a number of different components that are required to make CycleStreets all work smoothly. This recent switchover it has helped to tighten up a number of areas. This will make future switchovers easier to handle, and a major benefit has been a restructuring of our documentation.

We have tighted up how we handle file permissions and this means that developers no longer need sudo access to rollout code onto the live server. The structure we’ve got for that feels a lot cleaner and has improved the security of the system.

The hardest part of all this was the moment of switchover itself. As the system is live and being used to generate thousands of routes per day we aimed to have as little downtime as possible. The data on both systems was synchronised before turning off the live server and switching over the DNS. This was done in the small hours and because of careful prepartion was completed in about 15 minutes. When the new server became live it was impossible to tell that anything had changed. Only a few things were missed out in the switchover – the automatic tweeting of our photo-of-the-day, and the updating of a missed DNS record – but both of these are now fixed.

The system does feel more responsive now, and we’ve noticed that some of our scripts run twice as quick as previously.

We are very grateful to our hosts, Mythic Beasts, for their continued support and technical advice.

Funding for this upgrade has been partly helped by a grant from Cycling Scotland and from donations.

We are seeking a talented designer with strong user interface abilities to undertake design work for the new Cyclescape website. Timescales are tight. Cyclescape is to be a website (currently under construction) aimed at helping cycling groups around the UK. The work will involve the creation of an overall design concept, plus user interface templates for each of the user pages, and an icon set.

Read more on the Cyclescape website …