CycleStreets blog

This is a tech post, so if you’re not a techy person, feel free to stop reading now!

We’ve been reorganising the CycleStreets codebase to get ready for open-sourcing the remainder of our software.

This has been a challenging task involving learning how to write scripts that automate the installation of the CycleStreets website. The job has taken rather longer than expected, but has resulted in a much clearer structure to the highest level of the CycleStreets system and given us greater control of how the system is deployed.

It is fair to say that this aspect of our system had become rather complex with a lot of hidden dependencies. This would have been a real barrier for new coders to our project and so it had to be sorted out. The work now allows our codebase to separate out into these scenarios:

• Live deployment – provides the main CycleStreets website
• Failover deployment – holds a replicated copy of all CycleStreets data that can be used, should the live site fail for some reason
• Import – processes the latest OpenStreetMap extracts into a routing system

An immediate benefit of the work will be that we can much more easily deploy each of these roles. So for instance we could have multiple import machines, and each could be configured to serve different regions of the world.

We still have a little further work to clean and clarify the codebase before we can open the core code. This has been our main priority for the last few months, and will remain so for the next few weeks. It won’t be perfect, but we’ve aimed to get rid of the really bad dependency gremlins.

We’ve also been doing various cleanups, such as removing this blog installation from the main repository, daemonizing the routing process (optionally), and adding SMS monitoring.

The ongoing work on the scripts part of our system can be viewed at our public CycleStreets setup repository.

We’re pleased to announce a major new version of the CycleStreets Android App (version 1.4), our fourth update in just over a year. It brings a great new feature … Offline maps!

 Download the app update and install the map pack now!

Up until now, the CycleStreets app has had to download maps as it needed them. As you move around the map, or zoom in and out, the downloads new parts of the map – the tiles – as it goes. The different map styles it offers are provided by simply downloading the tiles from different servers. Generally, this works fine. But if you’re on you’re out and about and have a poor signal, or want to try and eke out your battery life, it can be a problem.

This new update provides a solution – a free downloadable map pack covering the whole of Great Britain and Ireland. The map pack is available from Google Play as a separate download, and once installed, you need never download another tile again. All the map data CycleStreets needs is there, on your phone, and it will render the maps it needs as it needs them. (Routing itself still uses the network, but this is a tiny amount of data by comparison.)

To set up the offline maps download the app update and install the map pack. Start the map pack app and it will set up the offline maps automatically. You can always switch between online and offline maps using the Settings screen.

Plan cycle-friendly routes from A to B, via C, anywhere in the UK!

• Plans routes through the full street and path network, including Sustrans routes and other networks
• Innovative & quick “three taps” system: Set current location, tap the map to set destination, and plan!
• Or search for any location in the UK, including full postcode support and local/national placefinder
• Choice of map styles (including the great OpenCycleMap showing contours, and Ordnance Survey Street View, and offline style)
• Downloadable maps for offline use
• Turn-by-turn itinerary view
• Choose from different types of routing – fastest/quietest/balanced/shortest
• Easily switch between route types (e.g. quietest to fastest) having planned a route
• Takes account of hills automatically
• Routes automatically saved for later use
• UK-wide (NB some areas of OpenStreetMap are better than others)
• Share your route by Twitter, Facebook, or email

CycleStreets for Android is an open source project and the code is available on GitHub. If you’d like to get involved with the code – and Android development is easier than you might imagine – do get in touch.

Please do add a review of the app too!

- Jez

We’re making good progress on this, after a stalled attempt to sort things out back in March when a busy period then struck. Our previous update has some background on this overall initiative.

Firstly, we’re planning a Developer Day in late summer – more news soon.

The last month or so has seen significant refactoring of some of the least clean parts of the codebase. In particular:

• The XML API generation framework has been tidied considerably, with a long-standing and obstructive dependency finally removed. There is more to be done later.
• The Photomap code has been restructured. For a long time it has been shackled by a wrong class hierarchy that saw all access (e.g. viewing photos, adding them, browsing on the map) going through an overarching class that really should just have been a resource class. Over a several-day refactoring this class hierarchy was reversed.
• The code for addition of photos in particular has been rather a mess for a while because there are many different GUIs writing to it. Work to move the model-related code to a separate model class is seeing this cleaned up, and various lurking bugs have been removed in the process.

Some of the work has been very difficult even for us as the original authors(!), with some of it quite painful due to the age of the code and the number of hacks it had acquired. It is code areas like these which we know would have put people off even considering helping because the risk of breakage was very high to an unfamiliar author.

A Git server has just been installed (thanks Wookey!), so we are about – at long last – to move away from SVN which will enable us to move to a multi-branch model and, more importantly, then to move to GitHub.

We’ve thought long and hard about whether to retain the SVN history during the migration to Git. I’m pleased to say that we do intend to retain this (unless any big problems arise), with over 9,000 commits therefore being carried forward into the new Git repo to be created. In coming to this conclusion we took the view that, despite presenting some risks, it means that the history of the development of the system – which is effectively its documentation – is retained. This is particularly important because of the code was never designed with unit tests in mind (unlike our new Cyclescape codebase).

We’ve identified most of the sensitive strings such as passwords and API keys that will be stripped out as part of this process.

Initially we planned to split out the split the codebase into around 7 separate repositories (e.g. Journey planner, Photomap, common, API launcher, etc.), but in retrospect we think this would cause more problems than it solves in terms of ease of installation. So instead we are planning to split the sections of the code more simply into directories at the top-level, and do this after putting the code on GitHub.

Stay tuned for imminent updates!

We’re holding a CycleStreets for Android Hack Day on Saturday 25th February 2012. Come along, because it’ll be fun.

Dive in

So what might you do with the Android app? Well, anything you like, but here are some suggestions – export a CycleStreets route as GPX, integrate a vector mapping library like MapsForge, add speech output, or simply poke around in the code and see what catches your eye. Learn some new things, ask questions, maybe have a beer in the evening.

What you’ll need

• A Laptop. Linux, Windows, or Mac, whatever your preference.
• The CycleStreets for Android source code. The source code lives up on GitHub, so grab a copy by forking it. If you don’t already have a Github account, you will need to create one. You’ll also need need a copy of Git itself. If you’re unfamiliar with Git, don’t worry because we’ll give you a primer. If you’re unfamiliar with source code control in general, you still don’t need to worry, we’ll teach you a valuable life skill.
• The Android SDK. To build the code, you’ll need the Android SDK. Strictly speaking the SDK and a text editor are more or less all you need, but you will probably find this a little easier using it with the Eclipse IDE. The integration with Eclipse is very good, the code completion support will help you work your way through an unfamiliar API, and the debugging support is excellent. See the image below for the components you’ll need.
• An Android device. Actually this is optional, as the Android SDK supports Virtual Devices you can run on your laptop, but it’s quicker and easier to run your code on a real device.
• 3G dongle if you have one – there’ll be internet, but bring a dongle just in case we have problems.

What you won’t need

• Previous Android Experience. Programming for Android is actually pretty straightforward, so you don’t need any specific experience. Familiarity with event-driven user interfaces is helpful, but if you’re keen you should be able to pick it up pretty easily.
• Java Expertise. The language of Android is Java, but you don’t need deep, deep Java expertise. You can get by with a smattering.

When and where?

Saturday 25th February 2012 from 12 noon – 10pm.

Ross Street Community Centre. [Map showing the route from Cambridge station - green route is best.]

It’d be great if you could let us know if you can come so we can order enough fresh pizza!

The bits of the Android SDK that you need, shown ticked (click to enlarge):

We’ve recently completed a switch over of the whole CycleStreets system to upgraded hosting. Our previous primary server has been extremely reliable and, having just checked, I find it has been up for 795 days, and has only been rebooted once in the last two-and-a-half years.

The new primary server is needed to support expansion of the journey planner, to handle the growing amount of data we manage in the CycleStreets system and to provide increased reliability of our services,  as well as for our new project, Cyclescape. We now also have additional redundancy.

The switchover has turned out to be a rather more complex procedure than expected as there are quite a number of different components that are required to make CycleStreets all work smoothly. This recent switchover it has helped to tighten up a number of areas. This will make future switchovers easier to handle, and a major benefit has been a restructuring of our documentation.

We have tighted up how we handle file permissions and this means that developers no longer need sudo access to rollout code onto the live server. The structure we’ve got for that feels a lot cleaner and has improved the security of the system.

The hardest part of all this was the moment of switchover itself. As the system is live and being used to generate thousands of routes per day we aimed to have as little downtime as possible. The data on both systems was synchronised before turning off the live server and switching over the DNS. This was done in the small hours and because of careful prepartion was completed in about 15 minutes. When the new server became live it was impossible to tell that anything had changed. Only a few things were missed out in the switchover – the automatic tweeting of our photo-of-the-day, and the updating of a missed DNS record – but both of these are now fixed.

The system does feel more responsive now, and we’ve noticed that some of our scripts run twice as quick as previously.

We are very grateful to our hosts, Mythic Beasts, for their continued support and technical advice.

Funding for this upgrade has been partly helped by a grant from Cycling Scotland and from donations.

Progress on our cycle campaigner toolkit is steaming (pedalling?) ahead! Coding work began on Monday, and will continue solidly for the next few months.

The development of the toolkit has been possible thanks to our GeoVation award, which secured us £27,000 of funding. GeoVation is an Ordnance Survey initiative and forms part of the Ideas in Transit project with funding from the Technology Strategy Board and the Department for Transport.

We're pleased to introduce our developer team, Andy Allan, Andrew France, plus myself (Martin Lucas-Smith) from CycleStreets as project manager. Andy and Andrew will be working in it on solidly for the next few months so that a large amount of development can be done.

We're also approaching various designers and design companies for the work on the design and information architecture aspects of the project.

Get involved in an exciting open source project!

The toolkit is to be developed as an open source project, with the code on Github. It will be written using the technologies of Ruby on Rails (v3.1), PostgreSQL, and jQuery.

Volunteers are needed to contribute to the code from the end of October. If you'd be interested, do let us know.

We're aiming to build a project team who will develop and look after the system from December onwards. By that stage, the grant-funded development work will mean that the system is already fully-featured and well coded so should be in a good state to add functionality to.

Timeline

We'll have:

• Screenshots of our 'alpha' available by 21^st October.
• A beta available to a few campaign groups from by 4^th November. If your group would like to be a 'guinea-pig', do let us know! We'll start to open it up to more groups in the month following.
• General availability of the site around 2^nd December, with bugfixes and additional features leading to the week before Christmas.

Stay tuned to this blog and our Twitter feed for updates.

We're also seeking an additional grant of around £10k to enable us to undertake solid coding on some of the more advanced features that groups have suggested, particularly do deal with more complex issues like cross-group sharing, which will increase the utility of the system considerably.

Designing the toolkit

We held our first developer meetings this week to develop the specification further and do some wireframing:

We're pleased to announce the launch of a major new section of our website:
Points of interest that you can click on to view and select for your journey.

Points of interest

We've integrated OpenStreetMap location data so that you can now click on points on the map. Just go to our journey planner tab and you can use the presets like bike shops, cafés, pubs, and many more.

(Mobile app developers: This data is also available through our API – see below.)

For instance, stations:

If you select a point, it is clickable. A Google Street View picture of the location will be shown, if it's a roadside location:

There's a link in the popup to the place's website if it has one.

There are lots of different POI types available:

You can browse locations anywhere the UK, for instance to find these independent bike shops in London:

Please add locations to the map!

If a place you know of doesn't appear in the map, please contribute your knowledge to OpenStreetMap by using the Edit section of our website. Warning: it can be quite addictive!

Or perhaps no-one's added a location's website yet? Click on the 'add it' link in the popup shown above. Follow the link, click on the icon, click on 'Advanced' and then enter 'website' on the left and the URL on the right, and click Save. You'll need to create an OpenStreetMap account if you don't have one already.

You must not copy things from other people's maps, however – additions and edits must be based on your local knowledge of an area.

API

This data is now all available through our API so that it can be integrated into your cycle routing app.

For full details, see our API documentation.

(We hope to have this functionality in our own apps shortly. If you can help patch it in, please branch our app repo and pitch in!)

A new release of our iPhone app is now out. Go to the app store updates page and update today, or download the app if you don't already have it (it's free!).

Version 1.5.1 fixes various reported bugs (including the 'freezing map' bug that affected some installations).

So what's new?

Firstly, we've made various improvements to the itinerary listing page. There are more icons on the listing view to help distinguish the type of paths you'll be cycling on. We've also rearranged the itinerary map view to put the left/right buttons nearer your thumb, and enabled the Photos-en-route to be disabled. In areas like London and Cambridge these sometimes obscured the road names.

We've also added a new feature for Ordnance Survey fans – the OS Street View (open data) map is now available experimentally. You can switch the app over to this OS style (or to OpenCycleMap) on the settings page.

Next, the OpenCycleMap map view works much better. This is the second map option which highlights cycle paths and shows hills.

The app was previously using an older and slower map server (entirely our fault, not Andy who runs it!). We've updated to the recommended server, and ensured that you can zoom all the way in. As a result, OpenCycleMap now loads much quicker and in all its beautifully detailed glory!

Station codes have been added to into the search results – a mini-feature which some people will find useful.

If you take your bike on trains, or perhaps travel to London before hopping on a boris-bike (or indeed now to Newcastle on a ScratchBike!), you can enter the station code (e.g. KGX for King's Cross, CBG for Cambridge), and the app will find that directly. This is particularly useful if you regularly make cycle journeys to/from particular stations.

The full list of changes is:

• OpenCycleMap map style now loads faster and to a higher zoom level.
• New Ordnance Survey map style option (OS open data Street View).
• Can now search for stations using the station code (e.g. KGX for King's Cross)
• Better support for journey road types.
• Ability to toggle route photos on/off in the itinerary view.
• Freezing map bug fixed.
• Bug fixes and UI enhancements.

An open source project – developers wanted!

This update to the app is thanks again to the hard work of Neil Edwards, who has done a few more days' work on it in his spare time. We can really recommend Neil if you need any iPhone app development doing.

The app is open source, and we'd really welcome volunteers to join our team. The code is on GitHub is available, and there are a range of new features and fixes we'd like to add – both small and big. So if you know Objective-C and would like to contribute to an app that's planned over 2 million km of cycle routes, do get in touch. Neil will be doing a tech posting soon, but don't let that hold you up.

Other platforms

Users of other types of phones need not feel left out … Our Android app will see another update soon. And our mobile web site is almost ready for release, meaning Blackberry/Windows/Symbian owners should have access to our routing on the go shortly – stay tuned for a blog post soon.

This is a technical post – you can stop reading now if you're not into web development!

We've been continuing to develop the spec that we recently published for the toolkit.

Several options for development have emerged.
We outline these in our Technical Options Report.

We'd welcome any comments.

CycleStreets has secured £27,000 of funding as a winner of Geovation – GeoVation is an Ordnance Survey initiative and forms part of the Ideas in Transit project with funding from the Technology Strategy Board and the Department for Transport.

Jez Higgins is the lead developer of our Android app. Here he writes about getting involved in the project.

Until Christmas last year, I'd never even thought of writing software for an Android phone. Yet within a month of googling "Android development quick start" I was committing code to the CycleStreets Android app. This indicates two things – writing code for Android is pretty easy to get in to, and the chaps at CycleStreets are pretty welcoming.

I don't remember quite how that happened. I'd used the website for a while and probably registered in the back of my mind that an Android app was in the works. When I wanted something more substantial than a toy example to play with, there it was on GitHub just asking to be forked. The basic shape of the code was there, laid down by Theodore Hong and incorporating pieces from Jono Gray's BikeRoute and Christopher Fraser's Cycloid. I made a little change and submitted a patch, something GitHub makes very easy, and that was that.

Because I have a job, and kids, and a dog, and the other joys and encumbrances of modern life I worked on the app in my space in between. Half an hour in the evening perhaps, an hour on the train. Bit by bit, piece by piece, it came together and really quite quickly. Because the heavy lifting of mapping and routing, provided respectively by the osmdroid library and by the CycleStreets website via its API, work could concentrate on wiring things up and making things happen. It wasn't without incident, of course, but the Android platform is well documented and surprisingly few quirks. Most of the time it was more a question of "how do I do this?" rather than "why is this happening?"

The app went live in the Android Market last month. At time of writing it's installed on around 3,250 people's phones. It might not (yet) be the most widely used piece of software I've worked on, but it's by far the most useful, and one of the most fun and satisfying.

Android developers: come on in

If you feel like dipping your toe into Android development but didn't know where to start, come and have go. If you're already swimming around in there, but fancy working on something that really does make a difference to people's lives this could be the project for you.

1. Install the app so you know what it does
2. Grab the code from GitHub: git@github.com:cyclestreets/android.git
3. Have a read through the current issues list or just get in and start working the code.
4. and .. Say hello [info@] – we'd love to hear from you!

Jez