Version 2.0.1 16 December 2016
Today we have released version 2.0.1 of FixMyStreet, a bugfix release. The issues fixed are:
- strftime output (e.g. used to display dates) was sometimes being double decoded, giving incorrect output in some languages.
- If a category was filtered, it was not being properly carried through to the new report form.
- The body/area inputs in the admin were being fixed to a too-small height.
Some bugfixes and improvements have also been made to the admin inspector report view, along with a couple of development improvements – the tests coping better if run on a network that intercepts NXDOMAINs, and a better way of showing the git version in the admin config.
It also let us add the Content-Security-Policy header to FixMyStreet, which is a method in browsers to prevent cross-site scripting (XSS), clickjacking and other code injection attacks by specifying the valid sources for script execution.
Separate scripting and styling
When you report an issue on FixMyStreet on mobile, the map
css()calls when setting up (or ending) the mobile view. It was straightforward to move this CSS to a
only-mapclass to prevent the map being scrolled until it is clicked, when the rest of the form can then be shown.)
Now that I had a
mobile-reporting-mapclass, I wanted this class activated as soon as possible as the page is loading, not only when the document had been parsed. There were also a couple of site-wide variables,
page(the type of page, e.g.
To achieve all this, I created a
header.jsfile that performed the above three tasks, setting a class on <html>, setting two variables on our global
fixmystreetvariable, and if we’re on a small width (using Modernizr) and perhaps a map page, setting the appropriate classes. I then minimized and inlined this script in the header of each page, so that we don’t have to wait for any external script to load.
“nonce” was only added in the second version of the CSP spec, so you may note our header also specifies
unsafe-inline. Any browser that supports version 2 will ignore this when it sees the nonce header, but it is needed in order for the inline script to still run in any browser only supporting version 1.
In Google Page Speed Insights, with manual minification of the main JS files, this moves the front page from 68/84 to 85/92ish (filmstrip from webpagetest.org, top is live site, bottom is my dev site):
These are requests from the US: most of the initial delay is in that initial download. Now here’s a report page going from 58/77 to 85/86ish (“ish” because e.g. live site will have analytics that my dev site doesn’t):
Version 2.0 – testing improvements 15 December 2016
FixMyStreet has a large and hopefully comprehensive test suite that runs through all aspects of the codebase, checking everything is working. This makes it easier to change code and add new features, safe in the knowledge that any breakages will be quickly highlighted.
Speeding up the tests
Every time someone commits code to our GitHub repository, or opens a pull request, the tests are automatically run for us by Travis CI. We’re alerted to success or failure with little green ticks or red crosses on GitHub, and by notice in IRC.
The tests seemed to have slowed down considerably in recent times, but we couldn’t identify any changes at the FixMyStreet side which might have caused this.
However, there had recently been some spam scraping of Gaze, our web service that provides population density information to FixMyStreet (so that e.g. the maps can try and guess an appropriate zoom level, and so alerts can try and guess an appropriate radius), and rate limiting had been added to try and help combat it.
Dave spotted that this was being triggered by FixMyStreet test runs, leading to pauses as the suite waited for the rate limiting to ease. Thankfully, all Gaze calls were being routed through one function (that had been created in order to cope gracefully with a Gaze failure) and so it was a simple matter for this function to be stubbed out if being run as part of a test.
There are many tests that still rely on the internet (e.g. for some MapIt lookups) and eventually it would be good to get to the point where they are all stubbed out and the test suite can run completely offline, probably even more quickly.
Multiple test running
When running the tests, the suite creates a test database (in PostgreSQL terms, it actually creates a temporary cluster) so that anything it does won’t affect your development database. Theoretically, this means you should be able to run the test suite multiple times simultaneously – perhaps it’s doing a full run, but you want to try and fix (and retest) the first error while it carries on. However, this was not working, and after some investigation it turned out that each run was creating (and overwriting) a test configuration
.ymlfile, which meant the existing runs got all confused. Adding a process ID to the test configuration file meant that each run is independent and can successfully coexist with each other.
Lastly, you used to have to run the full suite with
bin/run-tests t, but now if you run
bin/run-tests, it will assume you meant
t. A small thing, but it might save a few seconds over the years. ;-)
FixMyStreet has had a nice multiple image uploader since version 1.8. This uses multiple
input type=filefields, progressively enhanced to add drag’n’drop, image preview, and uploading in the background whilst you fill in the rest of the form.
For this new version, we had a different bug to fix. If the user had uploaded a picture, submitted the form, and was shown the form again due to a server side error of some sort (some validation not caught by client-side validation, for example, or because you were logging in during the reporting process), the image for the preview was then being loaded from the server (where it had already been uploaded), not the client, and not displaying. We patched the exif-js library., Now, if it is given a URL rather than a data: string, it will go off and fetch the image so that it can read out the orientation data.
Version 2.0 – Improved forms 13 December 2016
The new release of FixMyStreet includes a number of improvements to various forms on the site. In this post, we will take a brief look at the notable changes.
Public reporting form
This form has been rejigged, in order to more obviously split out details that will be public (e.g. photos, details) from those that will not be published on the site (e.g. the user’s email address and phone number). The category selector has also moved to the top, and if the category chosen requires the display of extra questions or information (e.g. through Open311 attributes or a custom built asset layer), they will be shown immediately.
If a report is made in an area that is covered by more than one body, the category the user selects will normally dictate which one the report is sent to. Now, when the category is selected, we update the list of bodies given at the top of the report page, if we know that the report will be sent there.
Talking about custom built asset layers, this is a good place to show how the FixMyStreet codebase can be put to other uses, with a bit of development.
Angus Council in Scotland provide a WFS layer (that is, vector format geographic information) containing the locations of all their streetlights, which we display if the street lighting category is selected within Angus on FixMyStreet.
Importantly, it can display which lights Angus already knows are broken. If the user is able to identify precisely which street light is affected, they can click on it. But picking a street light isn’t mandatory: we don’t want to put people off who aren’t certain, or who are unable to select an individual light. You can see an example of what this looks like on the Angus cobrand of FixMyStreet.com below.
Admin report editing
Previously, the form for administrators to edit a report was functional, but certainly nothing more than that! Due to the work we’ve done on the new user system, more admin users may well be accessing this form in future, and so we’ve taken the opportunity to make it much tidier.
It now looks much more like the front end of the site. We’ve added a map that lets you move the location of the report, tidied up the various functions an admin can perform, and so on. If a category change means the report should have been sent to a different body, it will be re-sent.
Admin category editing
This was a historical oddity, in that the Add category form and the Edit category form were completely separate creatures, though both contained the same fields, and were used for basically the same purpose:
You can see how they looked different in the screenshots above. They now share an HTML template, which also makes it easier for us to update should it need changing in future.
Version 2.0 – HTML5 History 9 December 2016
When viewing a list of reports on FixMyStreet, you might want to look at a few reports one after the other, much as you can on a Google Maps or OpenStreetMap results screen.
This feature uses HTML5’s History API, ie.
popState, to update the URL as the page changes. That means that when you share it, the page that loads will always be the one that the user intended it to be.
This improvement did not come without problems, however. Overriding the browser’s own behaviour when it comes to history and navigation means you have to think carefully, and I’m sure we’ll need to make further refinements to ensure that everything works as the user would expect.
There were small issues: for example,
pushStatestores the document title at the point when it’s called, for the ‘Back’ button list, meaning we had to make sure any title change happened after that. Some browsers have a
popstateon page load, which can cause an issue if you assume it’s only fired due to history events.
There was the complication of needing to tell the difference between someone clicking back to the ‘initial state’ of the page, and an internal hashchange or other less crucial event – as well as using a
replaceStateon page load, we store the original URL and title for use in such a situation.
Then we forgot that the code would be running on
/reportslists as well as
We also used
Version 2.0 – area highlighting 8 December 2016
FixMyStreet’s report pages for a particular body have always highlighted the area of the body covered, by fetching the KML shape from the associated MapIt and plotting it on a map. In version two we have made it look much nicer:
Until now we have coloured in the shape in a light orange, as pictured above left. This is fine for an overall view of a body, but if you want to zoom in to a particular area, it makes it quite hard to see the underlying map.
It would be preferable if every part of the map except the body’s area could be shaded, highlighting the correct area without losing any clarity from the part you’re interested in, as in the new image above right.
Polygons in KML and other similar formats are made up of multiple linear rings – the first ring is the polygon’s outline, and any subsequent entries are holes within the polygon. You can therefore ‘invert’ a polygon by having the outer ring be something surrounding the polygon, then having the polygon be a hole within that.
So the way we have implemented this in our new version is to fetch the area from MapIt, but before plotting it preface it with a giant rectangle covering the whole world. (Thankfully, any holes within the original shape are inverted too, though that is implementation dependent and it would have been a lot harder if they weren’t!).
There were a couple of issues along the way. MapIt may return either a polygon or multiple polygons, so we needed to deal with each differently (we turn multiple polygons into one polygon, which also thankfully Just Works). And OpenLayers has a hardcoded maximum pixel co-ordinate for SVG rendering, dating from 2007 and an issue in Firefox 2. We hope that any such issues have been fixed by now. We haven’t had any reports of crashes yet, anyway…
Version 2.0 – New user system 7 December 2016
Version 2 launches a new user system for admins, including more granular permissions and a variety of new features.
Admins can now use the same login system as the main front end. We highly recommend running your site over HTTPS in order to keep credentials secure; LetsEncrypt can supply free 3-month certificates.
The main admin user type is the ‘superuser’; a user with this flag set (which can be done in the database, or by running the
createsuperusercommand) has full access to all areas of the admin, just like admins under the old system. By default, these are the only types of user allowed to log in to
/admin. (It is possible to change this, e.g. in the UK, non-superuser admin users associated with a body can log in to
/adminon their own cobrand.)
Users associated with a body (called ‘body users’) can be given a variety of different feature-based permissions; the screenshot here shows the list of different options.
Categories: You can associate a user with a list of categories, which e.g. pre-selects those categories when the user visits the All Reports page.
Response priorities: This allows you to set a list of different priorities for a body, or again for a particular category in a body, letting you note different priorities for different reports.
Response templates: You can create and edit templates associated with your body, or with a particular category in that body, and then when leaving an update you can select one of these templates to allow easy updating of reports.
You can give a user access to a front-end report ‘inspect’ view, which lets a user edit a report’s category, state, or location. If the category change moves the report to a different body, it will be re-sent. Alternatively, a user can be given only category edit or priority edit permission. Here is a screenshot of the top of an inspect form view:
Create as another/body
This permission gives a user the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user. Below is a short animation showing this in action on the Oxfordshire cobrand of FixMyStreet.com:
A user with the shortlist permission gains a shortlist button on each report; clicking this adds the report to your own personal shortlist of reports, which you can view in a section of Your Account. This may be useful for an ‘inspector’ type of admin user, who wishes to compile the day’s list of reports before going out and investigating them. You can also see if a report is on someone else’s shortlist, and take it off them if you need to.
Version 2.0 – Multi-select dropdown jQuery plugin 6 December 2016
There are several types of ‘list page’ on FixMyStreet: for example, when you view all reports from a specific body, or when you log into your account to see reports you have made.
For quite a while now, users have been able to filter these by state (eg ‘fixed’ or ‘in progress’) or category (eg ‘pothole’, ‘streetlight’, etc), but a recent suggestion from Oxfordshire County Council prompted us to look again at this functionality and improve it.
So now it’s simple to filter by multiple states or categories: want to see all unfixed potholes? All streetlights or flytipping reports that are fixed? No problem.
<select multiple>is not the nicest tool in the box, especially when it comes to actually selecting multiple options. It usually involves holding down a key, but that key is different depending upon your operating system.
By default, it converts a multiple select into a dropdown with checkboxes:
The options can be wrapped within a container to stop the dropdown overlapping other content:
With options in the constructor, you can specify text for when all/no options are selected, and specify groups of options as presets that will be listed at the top of the dropdown:
On FixMyStreet, we have used this for both the state and category filters on list pages:
Let us know if you find the plugin useful!
Version 2.0 – email validation 5 December 2016
When someone enters an email address on FixMyStreet, we try and do some simple validation before sending a confirmation email, to catch typos and the like. We do the following:
- Check the address is correctly formed (e.g. that it contains an @ sign and a domain, and doesn’t include irregular characters, etc);
- Check that the email address’s domain is a valid top-level domain (TLD) (it’s not going to be delivered anywhere if it’s not); and
- Check that the email address’s domain has a valid DNS entry (so we’ll be able to try and deliver an email there).
The middle point means using a Perl module, Net::Domain::TLD, that contains a list of all valid TLDs. This list has grown since we last updated it, and we found ourselves incorrectly rejecting an email address that was perfectly valid (a @domain.cymru address). It was at least easy to update our version of Net::Domain::TLD to ensure that our list of top-level domains is current.
Changing your email address
The confirmation email is itself another form of validation, and the only one that can actually confirm that the user entering the email address has access to that email inbox.
Version 2 of FixMyStreet now allows users to change their own email address, which threw up a complication – we can’t update the user’s table to their new email address until it’s been confirmed.
So we have to store their old email address with the token that is sent to the new one. When the new address has been confirmed, we deal with it differently depending on whether or not that address already exists in the database. If it does, it’s a matter of merging the two accounts.