Version 2.0 – testing improvements 15 December 2016
FixMyStreet has a large and hopefully comprehensive test suite that runs through all aspects of the codebase, checking everything is working. This makes it easier to change code and add new features, safe in the knowledge that any breakages will be quickly highlighted.
Speeding up the tests
Every time someone commits code to our GitHub repository, or opens a pull request, the tests are automatically run for us by Travis CI. We’re alerted to success or failure with little green ticks or red crosses on GitHub, and by notice in IRC.
The tests seemed to have slowed down considerably in recent times, but we couldn’t identify any changes at the FixMyStreet side which might have caused this.
However, there had recently been some spam scraping of Gaze, our web service that provides population density information to FixMyStreet (so that e.g. the maps can try and guess an appropriate zoom level, and so alerts can try and guess an appropriate radius), and rate limiting had been added to try and help combat it.
Dave spotted that this was being triggered by FixMyStreet test runs, leading to pauses as the suite waited for the rate limiting to ease. Thankfully, all Gaze calls were being routed through one function (that had been created in order to cope gracefully with a Gaze failure) and so it was a simple matter for this function to be stubbed out if being run as part of a test.
There are many tests that still rely on the internet (e.g. for some MapIt lookups) and eventually it would be good to get to the point where they are all stubbed out and the test suite can run completely offline, probably even more quickly.
Multiple test running
When running the tests, the suite creates a test database (in PostgreSQL terms, it actually creates a temporary cluster) so that anything it does won’t affect your development database. Theoretically, this means you should be able to run the test suite multiple times simultaneously – perhaps it’s doing a full run, but you want to try and fix (and retest) the first error while it carries on. However, this was not working, and after some investigation it turned out that each run was creating (and overwriting) a test configuration
.ymlfile, which meant the existing runs got all confused. Adding a process ID to the test configuration file meant that each run is independent and can successfully coexist with each other.
Lastly, you used to have to run the full suite with
bin/run-tests t, but now if you run
bin/run-tests, it will assume you meant
t. A small thing, but it might save a few seconds over the years. ;-)
FixMyStreet has had a nice multiple image uploader since version 1.8. This uses multiple
input type=filefields, progressively enhanced to add drag’n’drop, image preview, and uploading in the background whilst you fill in the rest of the form.
For this new version, we had a different bug to fix. If the user had uploaded a picture, submitted the form, and was shown the form again due to a server side error of some sort (some validation not caught by client-side validation, for example, or because you were logging in during the reporting process), the image for the preview was then being loaded from the server (where it had already been uploaded), not the client, and not displaying. We patched the exif-js library., Now, if it is given a URL rather than a data: string, it will go off and fetch the image so that it can read out the orientation data.
Version 2.0 – Improved forms 13 December 2016
The new release of FixMyStreet includes a number of improvements to various forms on the site. In this post, we will take a brief look at the notable changes.
Public reporting form
This form has been rejigged, in order to more obviously split out details that will be public (e.g. photos, details) from those that will not be published on the site (e.g. the user’s email address and phone number). The category selector has also moved to the top, and if the category chosen requires the display of extra questions or information (e.g. through Open311 attributes or a custom built asset layer), they will be shown immediately.
If a report is made in an area that is covered by more than one body, the category the user selects will normally dictate which one the report is sent to. Now, when the category is selected, we update the list of bodies given at the top of the report page, if we know that the report will be sent there.
Talking about custom built asset layers, this is a good place to show how the FixMyStreet codebase can be put to other uses, with a bit of development.
Angus Council in Scotland provide a WFS layer (that is, vector format geographic information) containing the locations of all their streetlights, which we display if the street lighting category is selected within Angus on FixMyStreet.
Importantly, it can display which lights Angus already knows are broken. If the user is able to identify precisely which street light is affected, they can click on it. But picking a street light isn’t mandatory: we don’t want to put people off who aren’t certain, or who are unable to select an individual light. You can see an example of what this looks like on the Angus cobrand of FixMyStreet.com below.
Admin report editing
Previously, the form for administrators to edit a report was functional, but certainly nothing more than that! Due to the work we’ve done on the new user system, more admin users may well be accessing this form in future, and so we’ve taken the opportunity to make it much tidier.
It now looks much more like the front end of the site. We’ve added a map that lets you move the location of the report, tidied up the various functions an admin can perform, and so on. If a category change means the report should have been sent to a different body, it will be re-sent.
Admin category editing
This was a historical oddity, in that the Add category form and the Edit category form were completely separate creatures, though both contained the same fields, and were used for basically the same purpose:
You can see how they looked different in the screenshots above. They now share an HTML template, which also makes it easier for us to update should it need changing in future.
Version 2.0 – HTML5 History 9 December 2016
When viewing a list of reports on FixMyStreet, you might want to look at a few reports one after the other, much as you can on a Google Maps or OpenStreetMap results screen.
This feature uses HTML5’s History API, ie.
popState, to update the URL as the page changes. That means that when you share it, the page that loads will always be the one that the user intended it to be.
This improvement did not come without problems, however. Overriding the browser’s own behaviour when it comes to history and navigation means you have to think carefully, and I’m sure we’ll need to make further refinements to ensure that everything works as the user would expect.
There were small issues: for example,
pushStatestores the document title at the point when it’s called, for the ‘Back’ button list, meaning we had to make sure any title change happened after that. Some browsers have a
popstateon page load, which can cause an issue if you assume it’s only fired due to history events.
There was the complication of needing to tell the difference between someone clicking back to the ‘initial state’ of the page, and an internal hashchange or other less crucial event – as well as using a
replaceStateon page load, we store the original URL and title for use in such a situation.
Then we forgot that the code would be running on
/reportslists as well as
We also used
Version 2.0 – area highlighting 8 December 2016
FixMyStreet’s report pages for a particular body have always highlighted the area of the body covered, by fetching the KML shape from the associated MapIt and plotting it on a map. In version two we have made it look much nicer:
Until now we have coloured in the shape in a light orange, as pictured above left. This is fine for an overall view of a body, but if you want to zoom in to a particular area, it makes it quite hard to see the underlying map.
It would be preferable if every part of the map except the body’s area could be shaded, highlighting the correct area without losing any clarity from the part you’re interested in, as in the new image above right.
Polygons in KML and other similar formats are made up of multiple linear rings – the first ring is the polygon’s outline, and any subsequent entries are holes within the polygon. You can therefore ‘invert’ a polygon by having the outer ring be something surrounding the polygon, then having the polygon be a hole within that.
So the way we have implemented this in our new version is to fetch the area from MapIt, but before plotting it preface it with a giant rectangle covering the whole world. (Thankfully, any holes within the original shape are inverted too, though that is implementation dependent and it would have been a lot harder if they weren’t!).
There were a couple of issues along the way. MapIt may return either a polygon or multiple polygons, so we needed to deal with each differently (we turn multiple polygons into one polygon, which also thankfully Just Works). And OpenLayers has a hardcoded maximum pixel co-ordinate for SVG rendering, dating from 2007 and an issue in Firefox 2. We hope that any such issues have been fixed by now. We haven’t had any reports of crashes yet, anyway…
Version 2.0 – New user system 7 December 2016
Version 2 launches a new user system for admins, including more granular permissions and a variety of new features.
Admins can now use the same login system as the main front end. We highly recommend running your site over HTTPS in order to keep credentials secure; LetsEncrypt can supply free 3-month certificates.
The main admin user type is the ‘superuser’; a user with this flag set (which can be done in the database, or by running the
createsuperusercommand) has full access to all areas of the admin, just like admins under the old system. By default, these are the only types of user allowed to log in to
/admin. (It is possible to change this, e.g. in the UK, non-superuser admin users associated with a body can log in to
/adminon their own cobrand.)
Users associated with a body (called ‘body users’) can be given a variety of different feature-based permissions; the screenshot here shows the list of different options.
Categories: You can associate a user with a list of categories, which e.g. pre-selects those categories when the user visits the All Reports page.
Response priorities: This allows you to set a list of different priorities for a body, or again for a particular category in a body, letting you note different priorities for different reports.
Response templates: You can create and edit templates associated with your body, or with a particular category in that body, and then when leaving an update you can select one of these templates to allow easy updating of reports.
You can give a user access to a front-end report ‘inspect’ view, which lets a user edit a report’s category, state, or location. If the category change moves the report to a different body, it will be re-sent. Alternatively, a user can be given only category edit or priority edit permission. Here is a screenshot of the top of an inspect form view:
Create as another/body
This permission gives a user the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user. Below is a short animation showing this in action on the Oxfordshire cobrand of FixMyStreet.com:
A user with the shortlist permission gains a shortlist button on each report; clicking this adds the report to your own personal shortlist of reports, which you can view in a section of Your Account. This may be useful for an ‘inspector’ type of admin user, who wishes to compile the day’s list of reports before going out and investigating them. You can also see if a report is on someone else’s shortlist, and take it off them if you need to.
Version 2.0 – Multi-select dropdown jQuery plugin 6 December 2016
There are several types of ‘list page’ on FixMyStreet: for example, when you view all reports from a specific body, or when you log into your account to see reports you have made.
For quite a while now, users have been able to filter these by state (eg ‘fixed’ or ‘in progress’) or category (eg ‘pothole’, ‘streetlight’, etc), but a recent suggestion from Oxfordshire County Council prompted us to look again at this functionality and improve it.
So now it’s simple to filter by multiple states or categories: want to see all unfixed potholes? All streetlights or flytipping reports that are fixed? No problem.
<select multiple>is not the nicest tool in the box, especially when it comes to actually selecting multiple options. It usually involves holding down a key, but that key is different depending upon your operating system.
By default, it converts a multiple select into a dropdown with checkboxes:
The options can be wrapped within a container to stop the dropdown overlapping other content:
With options in the constructor, you can specify text for when all/no options are selected, and specify groups of options as presets that will be listed at the top of the dropdown:
On FixMyStreet, we have used this for both the state and category filters on list pages:
Let us know if you find the plugin useful!
Version 2.0 – email validation 5 December 2016
When someone enters an email address on FixMyStreet, we try and do some simple validation before sending a confirmation email, to catch typos and the like. We do the following:
- Check the address is correctly formed (e.g. that it contains an @ sign and a domain, and doesn’t include irregular characters, etc);
- Check that the email address’s domain is a valid top-level domain (TLD) (it’s not going to be delivered anywhere if it’s not); and
- Check that the email address’s domain has a valid DNS entry (so we’ll be able to try and deliver an email there).
The middle point means using a Perl module, Net::Domain::TLD, that contains a list of all valid TLDs. This list has grown since we last updated it, and we found ourselves incorrectly rejecting an email address that was perfectly valid (a @domain.cymru address). It was at least easy to update our version of Net::Domain::TLD to ensure that our list of top-level domains is current.
Changing your email address
The confirmation email is itself another form of validation, and the only one that can actually confirm that the user entering the email address has access to that email inbox.
Version 2 of FixMyStreet now allows users to change their own email address, which threw up a complication – we can’t update the user’s table to their new email address until it’s been confirmed.
So we have to store their old email address with the token that is sent to the new one. When the new address has been confirmed, we deal with it differently depending on whether or not that address already exists in the database. If it does, it’s a matter of merging the two accounts.
Version 2.0 – HTML emails 29 November 2016
FixMyStreet sends a number of automated emails, both to users (confirmation emails, follow-up questionnaires) and to bodies (the reports themselves). Previously these were plain text, but we have now introduced HTML emails, with all the design possibilities that this implies.
One reason for this is to make the site’s communications look more professional; another is an attempt to minimise one of the most time-consuming admin tasks - dealing with users who reply to our automated emails.
Designing better emails
In Version 2.0 we wanted to afford FixMyStreet emails the same design and usability attention that we normally spend on the FixMyStreet website.
Incorporating feedback from our users, our support team, and our council partners, we quickly identified a few key places that HTML (graphical) emails could improve the FixMyStreet experience for everyone:
- Attaching a map and photo of each problem to our “Has your problem been fixed?” questionnaire email, to help people remember the report we’re asking about.
- Styling calls to action as attractive, clickable buttons, to make the emails easier to quickly scan and comprehend, and to reduce user support queries.
- Using photos and a clearer typographical hierarchy to make the area alert emails easier to scan, especially when there are multiple new problems in your chosen area.
- Letting our council partners and international reusers maintain their brand image across the website and emails, by easily customising the logo and colour scheme of emails sent to their users.
The best way to see how we’ve improved FixMyStreet’s emails is to give FixMyStreet a try, and receive the emails yourself! Try reporting a new problem near you, or subscribing to email alerts about new reports in your area.
If you’d like to know how we implemented some of our more technical changes, read on…
Attaching the static map image
One thing that we wanted to include in the email was a map showing the location of the report (be that the one you’ve just made, or the one you’re receiving an alert or questionnaire for). The map that you see on the website is made up of many 256x256px tiles stitched together in HTML/CSS to appear as one smooth map, with separate pin images superimposed in the correct location; for the email, we needed just one image covering whatever portion of the map was necessary, plus the pin.
This was perfect for the image to be used in an HTML email. The new static map function fetches the same data used by the front end, requests the tile data for those four tiles, stitches them together in one 512x512px image, composites the pin on top in the correct location, and then cuts off the bottom 128 and top 64 pixels – as the pin’s central location means those will always be pin-less. The image is then shrunk to 310px in width, providing output that looks like this, appearing in the top right hand corner of the email:
The new HTML email - whether they’re reports, alerts, confirmations or questionnaires - normally have upwards of three images: the static map image, an image of the report (or repots for alert emails), and a site logo. We wanted to include all these images within the email itself, rather than use remote images, as due to spam many people have remote images switched off.
We also wanted to keep things as simple as possible when including images in the email templates. Inline images in HTML emails use an image source of
cid:UNIQUE_IDwithin the HTML (or CSS), and then give a particular attachment of the email the same
UNIQUE_IDin its Content-ID header. Lastly, we may potentially also have normal attachments.
We settled on an
inline_imagefunction in the template, which can be provided with either a path to an image file (for the static logo), or a function that returns image data and a content type (for the static map or report image). As the email template is processed, each call to this function generates a unique ID for the image and stores the information in a list to be added after the plain text/HTML email parts.
There are a variety of ways to attach images to an email. Content types are used to identify what is contained by each part of the email. As well as the various types of image, and text/plain and text/html for the text and HTML parts, there are various containers:
multipart/alternativeas a container for parts containing the same information in different formats,
multipart/relatedas a container for parts that are linked in some way, and
multipart/mixedas a general container.
The setups we decided on were as follows:
If the HTML email has no inline images or other attachments, then we create a
multipart/alternativeemail, containing the two textual parts.
If the HTML email has inline images but no other attachments (the most common case), we create a
multipart/relatedemail, its first part being the
multipart/alternativeas above, the subsequent parts the inline images.
An alternative here would be to create a
multipart/alternativeemail, with its first part being text, and its second part being
multipart/relatedcontaining the HTML part and the inline images. This would mean that an email client that didn’t support HTML email might only see the textual part and not any of the images. If your inline images were not ‘important’ to the email (e.g. an email footer signature) then this might be a way to go; we thought that the map and image should be visible to all users if possible.
If the HTML email has attachments but no inline images, we create a
multipart/mixedemail, its first part being the
multipart/alternativeas above, the subsequent parts the attachments.
If the HTML email has both inline images and other attachments, then we create a
multipart/mixedemail. Its first part is a
multipart/relatedemail (that again contains
multipart/alternativeand the inline images), and then its other attachments form the later parts of the mixed email. If you imagine the parts as envelopes with brackets, it would look like this:
multipart/mixed ( multipart/related ( multipart/alternative ( text/plain text/html ) image/jpeg image/png ) application/pdf )
As part of this work, I discovered that the oldest open GitHub issue of the Rails framework was related to this topic – if you used Rails to create an email containing both inline images and normal attachments, the normal attachments were not accessible to most email clients (that support HTML email) as they had embedded all the normal attachments inside a
multipart/relatedpart. I have submitted a pull request to fix this structure, which I hope will be accepted in some way.
This work was also a good opportunity to move some text generation out of some code into the templates (necessary because the text being generated now needed some HTML around each entry), for all the alert emails.
Finally, this post wouldn’t be complete without a few words about email testing.
Any of you who have built HTML emails in the past will agree that they are like taking a time machine back to web development in the mid 1990s. Email clients like Outlook, Gmail, and iOS Mail have dramatically different capabilities and ways of rendering the same email code.
One way of avoiding cross-client complications is to keep your HTML layout as simple as possible; maybe add an
<img>tag in for your site logo, some
<em>tags for emphasis, and call it a day.
But our plans for FixMyStreet required much more complex email layouts than this. It was a hard requirement that the details of the report (the map, the name, the photo) in our questionnaire emails, were given equal priority to the introductory text and the call to action buttons. The requirements led us to a two-column layout, which, inevitably, required two or three layers of nested table cells. (Remember, we’re in our 1990s time machine!)
To help us test these layout changes in all the required email clients, we used Litmus, which is like Browserstack but for emails. You send a single email to Litmus, and it renders that email in dozens of different email clients, grabbing screenshots of each one, and presenting them to you in a handy dashboard. Building HTML emails this way still requires a good deal of trial and error, and obscure CSS knowledge, but at least with Litmus, the process of iterating on your design is made as short and fast as possible. It’s an expensive service, but well worth the cost for the peace of mind that your new layout works in even the most uncooperative email clients.
With the help of our Litmus checklists, we made a bunch of unintuitive discoveries, including:
<td>elements, so that the Android 4.x mail client can give them
blockstyling in the small screen media query.
- Defining our font settings on every table cell, rather than simply inheriting
body, so that sans-serif fonts are used in Outlook, rather than Times New Roman.
- Using a three-column wrapper table to create a 620px centred content area that also shrinks down on narrow screens. (Outlook doesn’t like max-width, so this is the simplest alternative.)
- Enforcing a sensible (500px) min-width for the main content area, on clients that don’t support media queries.
- Using giant borders on
<a>elements, to make them into Outlook-friendly buttons without resorting to less accessible alternatives like images.
- Aligning images with the deprecated
alignattribute, rather than CSS floats.
- Applying the email background colour to a wrapper element inside the
body, and thus leaving the
bodyto keep its default white background, so that replies sent from Outlook (which inserts the reply message inside the body of the original message) will have a white background.
Version 2.0 15 November 2016
We’re proud to announce the release of version 2.0 of FixMyStreet.
This version contains a wide array of new features, including HTML email and multiple state/category filtering, a new admin user system with graduated permissions, and various bugfixes and development improvements.
Over the next few days and weeks, we will be writing a series of blog posts, going into details about a number of the changes, which I hope will be of interest. But before then, do set up the code or update your installation, and ask us questions :)
New front end features
HTML email: There is now the option for all emails sent by FixMyStreet to be HTML formatted where previously they were plain text only. This includes confirmation and questionnaire emails to the user, and report emails to the public body. These emails include any image added to the report, plus a small static map of the problem’s location.
State/category filtering and sorting of list pages: When viewing a list of reports, you can now filter and sort them in pretty much any way you choose, including sorting by most- or least-recently updated, newest or oldest, or most commented. You can also select multiple categories or states (e.g. “fixed”).
Pretty area highlighting on body pages: The highlighting of areas on a body page has been inverted, so that the unimportant parts of the map are shaded and you can interact more easily with reports on the page.
Users can now update their own email address This was a frequent request from users and we’re glad to report that they can now do it themselves on their account page.
Performance improvements: When looking at reports from a list page, the other report pins stay visible so that it is easier to switch between them. The report itself is being pulled in behind the scenes, meaning the whole page does not need to reload. The map no longer extends underneath the sidebar and header, which makes things easier, and a scroll wheel can now zoom the map in and out.
Making privacy options clearer: The reporting form has been separated into public and private sections, to make it clearer which parts of what you provide will be made visible on the site.
Showing the relevant recipient: If you live in an area where there’s more than one body, the category you pick normally dictates which body we send your report to. Now, when you select the category we update the name of the body given at the top of the report page, if we know that the report will be sent there.
New admin user system
Admin users can now use the same log-in right across the site - whether they’re making a report like a standard user, or logging in to make edits and moderate the site.
In the past, the distinction between admin and other users was black and white. As an admin user, you had access to every part of the site, but users can now be given individual permissions for various layers of access. These include:
- Proxy users This layer grants the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user;
- Report editors Giving the power to edit a report’s category, state, or location. If the admin user changes the category, and that change means that a different body is now responsible for the report, it will be re-sent;
- List makers, who can compile their own shortlist of reports they wish to go and inspect. This may be useful for a contractor or team who wishes to compile the day’s tasks;
- Quick responders These users have access to response templates, allowing them to edit and publish templated updates;
- Prioritisers These users may set different priorities on reports;
- Trusted users A simple reputation system, which e.g. potentially lets reports from trusted users be actioned more quickly.
The admin report edit form has also been greatly improved, including a map to update a report’s location (and re-sending the report if the body changes), and much tidier layout.
Bugfixes and development changes
Bugfixes include updating the top-level domain (TLD) list for email validation, hiding authorities which don’t exist any more on the
/reportspage, and fixing the previously-broken photo preview display after form submission. We have dropped support for Internet Explorer 6.
If you’re a re-user of the codebase, there are a number of changes that will hopefully help you out. Geocoder results won’t be cached in development, the
GEO_CACHEvariables are now relative to the project root, we’ve added a server-side MapIt proxy, and you can add your own fields to the contact form. Open311 support has been tidied up and improved. If you run the tests, you can now run the test suite multiple times simultaneously, and the tests have been sped up quite a bit.
Plus quite a few other things; as always, see the full list of changes over on GitHub.