Blog

Version 2.0 – Multi-select dropdown jQuery plugin 6 December 2016

There are several types of ‘list page’ on FixMyStreet: for example, when you view all reports from a specific body, or when you log into your account to see reports you have made.

For quite a while now, users have been able to filter these by state (eg ‘fixed’ or ‘in progress’) or category (eg ‘pothole’, ‘streetlight’, etc), but a recent suggestion from Oxfordshire County Council prompted us to look again at this functionality and improve it.

So now it’s simple to filter by multiple states or categories: want to see all unfixed potholes? All streetlights or flytipping reports that are fixed? No problem.

HTML’s <select multiple> is not the nicest tool in the box, especially when it comes to actually selecting multiple options. It usually involves holding down a key, but that key is different depending upon your operating system.

So I looked at existing JavaScript plugins and finding nothing suitable created a very simple jQuery plugin which is available at https://github.com/mysociety/jquery-multi-select.

By default, it converts a multiple select into a dropdown with checkboxes:

The options can be wrapped within a container to stop the dropdown overlapping other content:

With options in the constructor, you can specify text for when all/no options are selected, and specify groups of options as presets that will be listed at the top of the dropdown:

On FixMyStreet, we have used this for both the state and category filters on list pages:

Let us know if you find the plugin useful!
Version 2.0 – email validation 5 December 2016

When someone enters an email address on FixMyStreet, we try and do some simple validation before sending a confirmation email, to catch typos and the like. We do the following:
- Check the address is correctly formed (e.g. that it contains an @ sign and a domain, and doesn’t include irregular characters, etc);
- Check that the email address’s domain is a valid top-level domain (TLD) (it’s not going to be delivered anywhere if it’s not); and
- Check that the email address’s domain has a valid DNS entry (so we’ll be able to try and deliver an email there).
The middle point means using a Perl module, Net::Domain::TLD, that contains a list of all valid TLDs. This list has grown since we last updated it, and we found ourselves incorrectly rejecting an email address that was perfectly valid (a @domain.cymru address). It was at least easy to update our version of Net::Domain::TLD to ensure that our list of top-level domains is current.

Changing your email address

The confirmation email is itself another form of validation, and the only one that can actually confirm that the user entering the email address has access to that email inbox.

Version 2 of FixMyStreet now allows users to change their own email address, which threw up a complication – we can’t update the user’s table to their new email address until it’s been confirmed.

So we have to store their old email address with the token that is sent to the new one. When the new address has been confirmed, we deal with it differently depending on whether or not that address already exists in the database. If it does, it’s a matter of merging the two accounts.
Version 2.0 – HTML emails 29 November 2016

FixMyStreet sends a number of automated emails, both to users (confirmation emails, follow-up questionnaires) and to bodies (the reports themselves). Previously these were plain text, but we have now introduced HTML emails, with all the design possibilities that this implies.

One reason for this is to make the site’s communications look more professional; another is an attempt to minimise one of the most time-consuming admin tasks - dealing with users who reply to our automated emails.

Designing better emails

In Version 2.0 we wanted to afford FixMyStreet emails the same design and usability attention that we normally spend on the FixMyStreet website.

Incorporating feedback from our users, our support team, and our council partners, we quickly identified a few key places that HTML (graphical) emails could improve the FixMyStreet experience for everyone:
1. Attaching a map and photo of each problem to our “Has your problem been fixed?” questionnaire email, to help people remember the report we’re asking about.
2. Styling calls to action as attractive, clickable buttons, to make the emails easier to quickly scan and comprehend, and to reduce user support queries.
3. Using photos and a clearer typographical hierarchy to make the area alert emails easier to scan, especially when there are multiple new problems in your chosen area.
4. Letting our council partners and international reusers maintain their brand image across the website and emails, by easily customising the logo and colour scheme of emails sent to their users.
The best way to see how we’ve improved FixMyStreet’s emails is to give FixMyStreet a try, and receive the emails yourself! Try reporting a new problem near you, or subscribing to email alerts about new reports in your area.

If you’d like to know how we implemented some of our more technical changes, read on…

Attaching the static map image

One thing that we wanted to include in the email was a map showing the location of the report (be that the one you’ve just made, or the one you’re receiving an alert or questionnaire for). The map that you see on the website is made up of many 256x256px tiles stitched together in HTML/CSS to appear as one smooth map, with separate pin images superimposed in the correct location; for the email, we needed just one image covering whatever portion of the map was necessary, plus the pin.

This is the point at which FixMyStreet’s progressively-enhanced base came in very useful. If JavaScript does not work for whatever reason, the site has always displayed an alternative: a small map made up of four tiles in a square, with the pin located appropriately on top. It also makes sure that the pin location (the point at the bottom middle of the pin) is contained within the central half of the tiled map (the red dashed area in our image here), so that there’s no chance the pin overlaps the edge of the map.

This was perfect for the image to be used in an HTML email. The new static map function fetches the same data used by the front end, requests the tile data for those four tiles, stitches them together in one 512x512px image, composites the pin on top in the correct location, and then cuts off the bottom 128 and top 64 pixels – as the pin’s central location means those will always be pin-less. The image is then shrunk to 310px in width, providing output that looks like this, appearing in the top right hand corner of the email:

Inline images

The new HTML email - whether they’re reports, alerts, confirmations or questionnaires - normally have upwards of three images: the static map image, an image of the report (or repots for alert emails), and a site logo. We wanted to include all these images within the email itself, rather than use remote images, as due to spam many people have remote images switched off.

We also wanted to keep things as simple as possible when including images in the email templates. Inline images in HTML emails use an image source of cid:UNIQUE_ID within the HTML (or CSS), and then give a particular attachment of the email the same UNIQUE_ID in its Content-ID header. Lastly, we may potentially also have normal attachments.

We settled on an inline_image function in the template, which can be provided with either a path to an image file (for the static logo), or a function that returns image data and a content type (for the static map or report image). As the email template is processed, each call to this function generates a unique ID for the image and stores the information in a list to be added after the plain text/HTML email parts.

There are a variety of ways to attach images to an email. Content types are used to identify what is contained by each part of the email. As well as the various types of image, and text/plain and text/html for the text and HTML parts, there are various containers: multipart/alternative as a container for parts containing the same information in different formats, multipart/related as a container for parts that are linked in some way, and multipart/mixed as a general container.

The setups we decided on were as follows:
- If the HTML email has no inline images or other attachments, then we create a multipart/alternative email, containing the two textual parts.
- If the HTML email has inline images but no other attachments (the most common case), we create a multipart/related email, its first part being the multipart/alternative as above, the subsequent parts the inline images.
  
  An alternative here would be to create a multipart/alternative email, with its first part being text, and its second part being multipart/related containing the HTML part and the inline images. This would mean that an email client that didn’t support HTML email might only see the textual part and not any of the images. If your inline images were not ‘important’ to the email (e.g. an email footer signature) then this might be a way to go; we thought that the map and image should be visible to all users if possible.
- If the HTML email has attachments but no inline images, we create a multipart/mixed email, its first part being the multipart/alternative as above, the subsequent parts the attachments.
- If the HTML email has both inline images and other attachments, then we create a multipart/mixed email. Its first part is a multipart/related email (that again contains multipart/alternative and the inline images), and then its other attachments form the later parts of the mixed email. If you imagine the parts as envelopes with brackets, it would look like this:
```
multipart/mixed (
  multipart/related (
    multipart/alternative (
      text/plain
      text/html
    )
    image/jpeg
    image/png
  )
  application/pdf
)
```
As part of this work, I discovered that the oldest open GitHub issue of the Rails framework was related to this topic – if you used Rails to create an email containing both inline images and normal attachments, the normal attachments were not accessible to most email clients (that support HTML email) as they had embedded all the normal attachments inside a multipart/related part. I have submitted a pull request to fix this structure, which I hope will be accepted in some way.

Templating

This work was also a good opportunity to move some text generation out of some code into the templates (necessary because the text being generated now needed some HTML around each entry), for all the alert emails.

Testing

Finally, this post wouldn’t be complete without a few words about email testing.

Any of you who have built HTML emails in the past will agree that they are like taking a time machine back to web development in the mid 1990s. Email clients like Outlook, Gmail, and iOS Mail have dramatically different capabilities and ways of rendering the same email code.

One way of avoiding cross-client complications is to keep your HTML layout as simple as possible; maybe add an <img> tag in for your site logo, some <strong> or <em> tags for emphasis, and call it a day.

But our plans for FixMyStreet required much more complex email layouts than this. It was a hard requirement that the details of the report (the map, the name, the photo) in our questionnaire emails, were given equal priority to the introductory text and the call to action buttons. The requirements led us to a two-column layout, which, inevitably, required two or three layers of nested table cells. (Remember, we’re in our 1990s time machine!)

To help us test these layout changes in all the required email clients, we used Litmus, which is like Browserstack but for emails. You send a single email to Litmus, and it renders that email in dozens of different email clients, grabbing screenshots of each one, and presenting them to you in a handy dashboard. Building HTML emails this way still requires a good deal of trial and error, and obscure CSS knowledge, but at least with Litmus, the process of iterating on your design is made as short and fast as possible. It’s an expensive service, but well worth the cost for the peace of mind that your new layout works in even the most uncooperative email clients.

With the help of our Litmus checklists, we made a bunch of unintuitive discoveries, including:
- Using <th> rather than <td> elements, so that the Android 4.x mail client can give them block styling in the small screen media query.
- Defining our font settings on every table cell, rather than simply inheriting font-family from the body, so that sans-serif fonts are used in Outlook, rather than Times New Roman.
- Using a three-column wrapper table to create a 620px centred content area that also shrinks down on narrow screens. (Outlook doesn’t like max-width, so this is the simplest alternative.)
- Enforcing a sensible (500px) min-width for the main content area, on clients that don’t support media queries.
- Using giant borders on <a> elements, to make them into Outlook-friendly buttons without resorting to less accessible alternatives like images.
- Aligning images with the deprecated align attribute, rather than CSS floats.
- Applying the email background colour to a wrapper element inside the body, and thus leaving the body to keep its default white background, so that replies sent from Outlook (which inserts the reply message inside the body of the original message) will have a white background.
Version 2.0 15 November 2016

We’re proud to announce the release of version 2.0 of FixMyStreet.

This version contains a wide array of new features, including HTML email and multiple state/category filtering, a new admin user system with graduated permissions, and various bugfixes and development improvements.

Over the next few days and weeks, we will be writing a series of blog posts, going into details about a number of the changes, which I hope will be of interest. But before then, do set up the code or update your installation, and ask us questions :)

New front end features
- HTML email: There is now the option for all emails sent by FixMyStreet to be HTML formatted where previously they were plain text only. This includes confirmation and questionnaire emails to the user, and report emails to the public body. These emails include any image added to the report, plus a small static map of the problem’s location.
- State/category filtering and sorting of list pages: When viewing a list of reports, you can now filter and sort them in pretty much any way you choose, including sorting by most- or least-recently updated, newest or oldest, or most commented. You can also select multiple categories or states (e.g. “fixed”).
- Pretty area highlighting on body pages: The highlighting of areas on a body page has been inverted, so that the unimportant parts of the map are shaded and you can interact more easily with reports on the page.
- Users can now update their own email address This was a frequent request from users and we’re glad to report that they can now do it themselves on their account page.
- Performance improvements: When looking at reports from a list page, the other report pins stay visible so that it is easier to switch between them. The report itself is being pulled in behind the scenes, meaning the whole page does not need to reload. The map no longer extends underneath the sidebar and header, which makes things easier, and a scroll wheel can now zoom the map in and out.
- Making privacy options clearer: The reporting form has been separated into public and private sections, to make it clearer which parts of what you provide will be made visible on the site.
  
  Showing the relevant recipient: If you live in an area where there’s more than one body, the category you pick normally dictates which body we send your report to. Now, when you select the category we update the name of the body given at the top of the report page, if we know that the report will be sent there.
New admin user system

Admin users can now use the same log-in right across the site - whether they’re making a report like a standard user, or logging in to make edits and moderate the site.

In the past, the distinction between admin and other users was black and white. As an admin user, you had access to every part of the site, but users can now be given individual permissions for various layers of access. These include:
- Proxy users This layer grants the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user;
- Report editors Giving the power to edit a report’s category, state, or location. If the admin user changes the category, and that change means that a different body is now responsible for the report, it will be re-sent;
- List makers, who can compile their own shortlist of reports they wish to go and inspect. This may be useful for a contractor or team who wishes to compile the day’s tasks;
- Quick responders These users have access to response templates, allowing them to edit and publish templated updates;
- Prioritisers These users may set different priorities on reports;
- Trusted users A simple reputation system, which e.g. potentially lets reports from trusted users be actioned more quickly.
The admin report edit form has also been greatly improved, including a map to update a report’s location (and re-sending the report if the body changes), and much tidier layout.

Bugfixes and development changes

Bugfixes include updating the top-level domain (TLD) list for email validation, hiding authorities which don’t exist any more on the /reports page, and fixing the previously-broken photo preview display after form submission. We have dropped support for Internet Explorer 6.

If you’re a re-user of the codebase, there are a number of changes that will hopefully help you out. Geocoder results won’t be cached in development, the UPLOAD_DIR and GEO_CACHE variables are now relative to the project root, we’ve added a server-side MapIt proxy, and you can add your own fields to the contact form. Open311 support has been tidied up and improved. If you run the tests, you can now run the test suite multiple times simultaneously, and the tests have been sped up quite a bit.

Plus quite a few other things; as always, see the full list of changes over on GitHub.
Version 1.8.4 6 July 2016

We’ve released version 1.8.4 of FixMyStreet, along with versions 1.7.2 and 1.6.3.

These releases include a security fix, closing two cross-site scripting (XSS) vulnerabilities. Please update your installation as soon as possible.

Version 1.8.4 also contains other minor bug fixes, such as correctly orientating preview images, wrapping emails better for differing screen sizes, a bug with filter redirect when JavaScript wasn’t available, a race condition when starting a new report, and a couple of display bugs in IE7.

See the full list of changes over on GitHub.
Version 1.8.3 3 June 2016

We’ve released version 1.8.3 of FixMyStreet.

For developers, the main improvement in this version is the great speed up to CSS compilation by moving from sass to libsass.

We’ve fixed some map issues, getting the Google Maps layer working again and dealing with tap sensitivity on some devices.

The admin interface gets a bit of love, adding common search boxes to the index page, allowing change of email to an existing address, and speeding up photo removal.

Upgrading notes

If your templates aren’t kept in the main repository, there are a couple of things to be aware of with this release:
- If you’ve used the cf class, you’ll need to rename it to clearfix.
- If you have customised any of the email templates that use <?=...?> style variables (e.g. alert emails, questionnaire, submit), you’ll need to rewrite them to use the standard [%...%] variables.
See the full list of changes over on GitHub.
Version 1.8.2 3 May 2016

We’ve released version 1.8.2 of FixMyStreet, along with versions 1.7.1 and 1.6.2.

These releases include an important security fix, whereby a malicious user could craft an image upload to the server that allowed them to run external commands as the user running the site. Please update your installation as soon as possible.

Version 1.8.2 also contains other improvements and additions to existing features:
- Twitter social login, alongside the existing Facebook login;
- PNG and GIF image upload support;
- Some development improvements, including the final merging of base and fixmystreet templates, storing any Open311 error in the database, and tidying up some unused cobrands;
- A few bug fixes, such as showing the right body user form value for fixed reports (thanks Jon Kristensen).
See the full list of changes over on GitHub.
Version 1.8.1 23 March 2016

We’ve released version 1.8.1 of FixMyStreet, which fixes some bugs and makes some improvements to the 1.8 release.

Now there is multiple photo support, the display of those photos on a report page is now a bit nicer, and if there’s an error message on photo upload it should now always be visible. Auto-scrolling of the sidebar when you hover over a pin has been removed as it was confusing, and the site now remembers a user’s last anonymous state. Fixes have been made for running on more recent versions of Perl, one of which was causing the geocoder to break.

For developers, an easier way of adding cobrand-specific custom reporting fields has been added.

There are a couple of other minor changes; see the full list of changes over on GitHub.
Version 1.8 2 March 2016

We’ve released version 1.8 of FixMyStreet.

The two main new features in this release are Facebook login – provide a Facebook app ID and secret in your configuration and it will smoothly fit into the creation and login flow – and multiple photo support, along with a modern interface for previewing and uploading photos whilst you create your report.

Smaller improvements include highlighting the pin when you hover over an item in the sidebar, and vice-versa; fixing some small display bugs such as how updates were displayed in Your Reports and preventing a chevron being stretched in Firefox; improving the look of the 404 page, and making sure you can see an update if you got to it via an in-page link.

Memory performance has been improved, meaning cron jobs can take up to half as much memory, and this release also fixes a number of small bugs, including an embarrassing swap of latitude and longitude in the Google geocoder, making sure you’re signed up for updates if you used the app and were logged in, and better internationalisation and display of numbers.

For developers, we’ve added a generic static route handler, so that adding new static HTML pages to your installation involves only creating a new file in your template directory and nothing more; improved bounce handling; and fixed the cobrand restriction handling on Your Reports and list pages.

Plus quite a few other things; as always, see the full list of changes over on GitHub.
Version 1.7 23 October 2015

We’ve released version 1.7 of FixMyStreet.

This version adds some new features. First off is that the FixMyStreet design is now bi-directional, providing an easy switch to flip the design to either left-to-right or right-to-left. This was done with the kind support of the National Democratic Institute.

We have added state and category filters to the list pages, letting users view only e.g. open reports in the potholes category, or all reports in the graffiti category. Various design improvements have been made, including the showing of the report on a questionnaire page and the email confirmation pages, and we’ve added a nicer default OpenGraph image.

Database performance has been improved in a number of areas, and the accessibility of the map pages has been improved.

This release also fixes a number of small bugs, including translating report states in the admin index, dealing with DMARC email issues, and fixes for Google Maps API users.

For developers, we’ve made it easier to run gettext-extract if you’re performing your own translations, removed some confusing warnings, finally removed the final few hardcoded “FixMyStreet” strings so it’s easy to rename your site, streamlined the navigation menu and list item CSS using a BEM style naming scheme so it is easy to change and override, and lastly fixed a long standing issue where errors were not always logged correctly.

Plus quite a few other things; as always, see the full list of changes over on GitHub.