Blog

• Version 2.0.3 31 March 2017

  

  Today we have released version 2.0.3 of FixMyStreet, a bugfix release along with some other improvements.

  The map on a mobile report page can now be made full screen, and if you are using Google Maps you can supply a custom map styling. There’s also now a loading indicator whilst data is being fetched for the map.

  Various missing translations have been added, for moderation, social login, and offline usage.

  We’ve upgraded our email sending to deal with issues sending with SSL, dealt with IE11 caching report Ajax calls too aggressively, and with Safari 5/ Android 4 not showing our questionnaire answer buttons.

  Performance improvements include moving admin-related JavaScript to its own file (so normal users don’t need to download it), and reducing the amount of disk stats the code performs.

  Lastly, all the test suite can now run offline, and amusingly I found a bug in a test that only happened if the test was run c. 55 hours before daylight savings time began :)

  Full changelog

• Version 2.0.2 3 February 2017

  

  Today we have released version 2.0.2 of FixMyStreet, a bugfix release along with some new admin improvements.

  The main bugfixes are to mark two missing strings for translation, to make sure email is lowercase when signing in via /auth, and to make sure a language subdomain can be included in calls to base_url_for_report. There are also some CSS fixes to the homepage and an improved print layout for report list pages. Fixes to the admin interface include fixing filtering on the shortlist page, and fixing the ‘save with public update’ toggle.

  For international users, there have been a few string renames, but we have maintained translations as the concept remains unchanged, so please only feel free to update if you consider it necessary:

  • ‘unable to fix’ has been renamed to ‘no further action’
  • ‘marked as a duplicate report’ has been changed to ‘closed as a duplicate report’

  An offline fallback page has been added using appcache, which inspectors can use for offline updating of reports in their shortlist.

  Admin improvements include allowing response templates to be associated with a state (so that template is used by default when the state is changed), allowing shortlist addition/removal from report lists, and shortlist reordering. Users with a new permission can see the body user who left a contribute_as_body report or update.

  Full changelog

• Version 2.0.1 16 December 2016

  

  Today we have released version 2.0.1 of FixMyStreet, a bugfix release. The issues fixed are:

  • Chrome 55 changes the way it handles mouse/touch events, unifying them as pointer events, and OpenLayers, the JavaScript mapping library we use, needed patching to deal with this. Thanks to the Chrome team for helping debug this.
  • strftime output (e.g. used to display dates) was sometimes being double decoded, giving incorrect output in some languages.
  • If a category was filtered, it was not being properly carried through to the new report form.
  • The body/area inputs in the admin were being fixed to a too-small height.

  Some bugfixes and improvements have also been made to the admin inspector report view, along with a couple of development improvements – the tests coping better if run on a network that intercepts NXDOMAINs, and a better way of showing the git version in the admin config.

• Version 2.0 – JavaScript performance 16 December 2016

  The JavaScript on FixMyStreet has gradually evolved over many years (we launched in 2007, remember!), and while we were working on other features in this area (such as HTML5 History) it was a good opportunity to tidy up the JavaScript, making it clearer and simpler. Below I’m going to go through most of the steps I took, not necessarily in the order I took them, which hopefully might prove useful to your own websites. And there are exciting pictures at the end, I promise!

  It also let us add the Content-Security-Policy header to FixMyStreet, which is a method in browsers to prevent cross-site scripting (XSS), clickjacking and other code injection attacks by specifying the valid sources for script execution.

  Separate scripting and styling

  When you report an issue on FixMyStreet on mobile, the map /around page is full-screen to make reporting easier. This was being done in JavaScript using some jQuery css() calls when setting up (or ending) the mobile view. It was straightforward to move this CSS to a mobile-reporting-map class and have the JavaScript do no more than add or remove this class, making both parts clearer. (Later during this process, I also added an only-map class to prevent the map being scrolled until it is clicked, when the rest of the form can then be shown.)

  Similarly, the JavaScript was requisitioning the desktop-width green banner to provide a different mobile banner; updating this to use a separate mobile banner made the code clearer and shorter.

  Tidy up the JavaScript set up

  The main setting up of our JavaScript was taking place, for historical reasons, in two files, both confusingly called fixmystreet.js. Whilst in the future we may want to split this up into separate files more tailored to the particular pages where the code is used (though our main use of JavaScript is our map page, which already has its JavaScript separate), for now it made most sense to combine these in one file, and tidy up all the setup functions into a list of feature-based functions each called in turn on page load.

  We were bundling a copy of Modernizr (for media query detection) that contained html5shiv and yepnope. But html5shiv is only needed in old versions of Internet Explorer, and yepnope is only used on FixMyStreet’s front page (to try and preload the map page JavaScript mentioned above), so I could move html5shiv into an IE conditional comment, and included yepnope.js only on the front page, reducing Modernizr to only Modernizr itself.

  Move vital JavaScript as early as possible

  Now that I had a mobile-reporting-map class, I wanted this class activated as soon as possible as the page is loading, not only when the document had been parsed. There were also a couple of site-wide variables, page (the type of page, e.g. around or new), and cobrand (the branding of the site you’re on). Lastly, I wanted to be able to set a class on the document if JavaScript was activated, so that CSS using that class would be instantly active, preventing a flash of style change or content.

  To achieve all this, I created a header.js file that performed the above three tasks, setting a class on <html>, setting two variables on our global fixmystreet variable, and if we’re on a small width (using Modernizr) and perhaps a map page, setting the appropriate classes. I then minimized and inlined this script in the header of each page, so that we don’t have to wait for any external script to load.

  Move JavaScript to the end of the HTML, remove inline JavaScript

  As recommended by Jake Archibald here, I moved all the JavaScript to the end of the HTML. To make this easier, and also to make adding a Content-Security-Policy header easier, I removed all the inline JavaScript from FixMyStreet (I didn’t think FixMyStreet had that much inline JavaScript, and it didn’t, but it still had more than I had remembered!). This was most commonly being used to set up some JavaScript variables with server-side data, so the easiest way to replace this was to place this data as attributes on HTML elements (preferably semantically related elements), and set up the JavaScript variables in an external script.

  Minify JavaScript

  I didn’t want to make this mandatory, as we have done with SCSS/CSS, but I wanted the option available, so I added an option to our templating code that means it will prefer an .auto.min.js file in preference to a .js file of the same basename. This lets you compile your JavaScript in a deploy process, for example, should you wish to. We do this with the standard Closure Compiler from Google; I haven’t yet been brave enough to try and check/get the JavaScript working with the advanced option of the Closure Compiler :)

  Activate Content-Security-Policy

  The Content-Security-Policy header lets you specify domains from which JavaScript will run, plus lets you choose to run inline JavaScript either en masse or only if you provide a specific unique nonce ID in the <script> tag that matches the same ID in the CSP header. That latter option is how we kept our inline header JavaScript running without having to externalise it again.

  “nonce” was only added in the second version of the CSP spec, so you may note our header also specifies unsafe-inline. Any browser that supports version 2 will ignore this when it sees the nonce header, but it is needed in order for the inline script to still run in any browser only supporting version 1.

  Conclusion

  In Google Page Speed Insights, with manual minification of the main JS files, this moves the front page from 68/84 to 85/92ish (filmstrip from webpagetest.org, top is live site, bottom is my dev site):

  

  These are requests from the US: most of the initial delay is in that initial download. Now here’s a report page going from 58/77 to 85/86ish (“ish” because e.g. live site will have analytics that my dev site doesn’t):

  

  Due to page elements displaying more quickly than might previously have been expected, there are still a few things to tidy up. For example, when an element displays before a bit of JavaScript kicks in and makes it look slightly different… but hopefully nothing major.

• Version 2.0 – testing improvements 15 December 2016

  FixMyStreet has a large and hopefully comprehensive test suite that runs through all aspects of the codebase, checking everything is working. This makes it easier to change code and add new features, safe in the knowledge that any breakages will be quickly highlighted.

  Speeding up the tests

  Every time someone commits code to our GitHub repository, or opens a pull request, the tests are automatically run for us by Travis CI. We’re alerted to success or failure with little green ticks or red crosses on GitHub, and by notice in IRC.

  The tests seemed to have slowed down considerably in recent times, but we couldn’t identify any changes at the FixMyStreet side which might have caused this.

  However, there had recently been some spam scraping of Gaze, our web service that provides population density information to FixMyStreet (so that e.g. the maps can try and guess an appropriate zoom level, and so alerts can try and guess an appropriate radius), and rate limiting had been added to try and help combat it.

  Dave spotted that this was being triggered by FixMyStreet test runs, leading to pauses as the suite waited for the rate limiting to ease. Thankfully, all Gaze calls were being routed through one function (that had been created in order to cope gracefully with a Gaze failure) and so it was a simple matter for this function to be stubbed out if being run as part of a test.

  Before:

  After:

  There are many tests that still rely on the internet (e.g. for some MapIt lookups) and eventually it would be good to get to the point where they are all stubbed out and the test suite can run completely offline, probably even more quickly.

  Multiple test running

  When running the tests, the suite creates a test database (in PostgreSQL terms, it actually creates a temporary cluster) so that anything it does won’t affect your development database. Theoretically, this means you should be able to run the test suite multiple times simultaneously – perhaps it’s doing a full run, but you want to try and fix (and retest) the first error while it carries on. However, this was not working, and after some investigation it turned out that each run was creating (and overwriting) a test configuration .yml file, which meant the existing runs got all confused. Adding a process ID to the test configuration file meant that each run is independent and can successfully coexist with each other.

  Keystroke saving

  Lastly, you used to have to run the full suite with bin/run-tests t, but now if you run bin/run-tests, it will assume you meant t. A small thing, but it might save a few seconds over the years. ;-)

• Version 2.0 – EXIF rotation in JavaScript 14 December 2016

  FixMyStreet has had a nice multiple image uploader since version 1.8. This uses multiple input type=file fields, progressively enhanced to add drag’n’drop, image preview, and uploading in the background whilst you fill in the rest of the form.

  

  In version 1.8.4, we patched the third party library we use, dropzone, to correctly orient photos in the image preview. We did this by including a cut-down version of exif-js to read in the EXIF orientation data, and then make sure we rotated the image as instructed in JavaScript before drawing the thumbnail preview. The rotation was accomplished by moving the image so its centre was over (0,0), rotating the appropriate amount, and then re-translating it back.

  

  For this new version, we had a different bug to fix. If the user had uploaded a picture, submitted the form, and was shown the form again due to a server side error of some sort (some validation not caught by client-side validation, for example, or because you were logging in during the reporting process), the image for the preview was then being loaded from the server (where it had already been uploaded), not the client, and not displaying. We patched the exif-js library., Now, if it is given a URL rather than a data: string, it will go off and fetch the image so that it can read out the orientation data.

• Version 2.0 – Improved forms 13 December 2016

  The new release of FixMyStreet includes a number of improvements to various forms on the site. In this post, we will take a brief look at the notable changes.

  Public reporting form

  

  This form has been rejigged, in order to more obviously split out details that will be public (e.g. photos, details) from those that will not be published on the site (e.g. the user’s email address and phone number). The category selector has also moved to the top, and if the category chosen requires the display of extra questions or information (e.g. through Open311 attributes or a custom built asset layer), they will be shown immediately.

  If a report is made in an area that is covered by more than one body, the category the user selects will normally dictate which one the report is sent to. Now, when the category is selected, we update the list of bodies given at the top of the report page, if we know that the report will be sent there.

  Talking about custom built asset layers, this is a good place to show how the FixMyStreet codebase can be put to other uses, with a bit of development.

  Angus Council in Scotland provide a WFS layer (that is, vector format geographic information) containing the locations of all their streetlights, which we display if the street lighting category is selected within Angus on FixMyStreet.

  Importantly, it can display which lights Angus already knows are broken. If the user is able to identify precisely which street light is affected, they can click on it. But picking a street light isn’t mandatory: we don’t want to put people off who aren’t certain, or who are unable to select an individual light. You can see an example of what this looks like on the Angus cobrand of FixMyStreet.com below.

  

  
  

  Admin report editing

  

  Previously, the form for administrators to edit a report was functional, but certainly nothing more than that! Due to the work we’ve done on the new user system, more admin users may well be accessing this form in future, and so we’ve taken the opportunity to make it much tidier.

  It now looks much more like the front end of the site. We’ve added a map that lets you move the location of the report, tidied up the various functions an admin can perform, and so on. If a category change means the report should have been sent to a different body, it will be re-sent.

  

  Admin category editing

  This was a historical oddity, in that the Add category form and the Edit category form were completely separate creatures, though both contained the same fields, and were used for basically the same purpose:

  
  

  You can see how they looked different in the screenshots above. They now share an HTML template, which also makes it easier for us to update should it need changing in future.

  
  

• Version 2.0 – HTML5 History 9 December 2016

  We’ve mentioned before that FixMyStreet is built on a progressively-enhanced base, a concept explained neatly in a nice blog post from the UK government.

  This means that e.g. the slippy map is, underneath it all, an old-style server-side image map that works out where you click; URLs are all shareable and pages are functional even if JavaScript is not available; the front page loads quickly and doesn’t need to preload an entire application.

  None of this means that we don’t use or like JavaScript, however. This post is about adding JavaScript to FixMyStreet to provide a quicker experience to users looking at reports on our map.

  When viewing a list of reports on FixMyStreet, you might want to look at a few reports one after the other, much as you can on a Google Maps or OpenStreetMap results screen.

  So now, when you click a report in the list or a pin on the map, the report page is pulled in via JavaScript. This updates the page, pin and URL in situ, rather than loading a new page. Other pins remain visible, plus the page feels (and probably is) a bit quicker as the page header doesn’t reload, and it is easy to switch back to the list view.

  This feature uses HTML5’s History API, ie. pushState and popState, to update the URL as the page changes. That means that when you share it, the page that loads will always be the one that the user intended it to be.

  This improvement did not come without problems, however. Overriding the browser’s own behaviour when it comes to history and navigation means you have to think carefully, and I’m sure we’ll need to make further refinements to ensure that everything works as the user would expect.

  There were small issues: for example, pushState stores the document title at the point when it’s called, for the ‘Back’ button list, meaning we had to make sure any title change happened after that. Some browsers have a popstate on page load, which can cause an issue if you assume it’s only fired due to history events.

  There was the complication of needing to tell the difference between someone clicking back to the ‘initial state’ of the page, and an internal hashchange or other less crucial event – as well as using a replaceState on page load, we store the original URL and title for use in such a situation.

  Then we forgot that the code would be running on /reports lists as well as /around which led to some confusion until we realised what was happening! And of course, you have to make sure everything JavaScript-wise is set up appropriately for content brought in via JavaScript.

  We also used pushState in the new report process, to update the URL as you select the report’s location, and on list pages when you select one of the filters or sort. This has worked well, and is certainly much more preferable to the ‘hash-bang’ technique used by some sites in previous years (and still now), which is reliant on JavaScript functioning.

• Version 2.0 – area highlighting 8 December 2016

  FixMyStreet’s report pages for a particular body have always highlighted the area of the body covered, by fetching the KML shape from the associated MapIt and plotting it on a map. In version two we have made it look much nicer:

  

  Until now we have coloured in the shape in a light orange, as pictured above left. This is fine for an overall view of a body, but if you want to zoom in to a particular area, it makes it quite hard to see the underlying map.

  It would be preferable if every part of the map except the body’s area could be shaded, highlighting the correct area without losing any clarity from the part you’re interested in, as in the new image above right.

  Polygon make-up

  Polygons in KML and other similar formats are made up of multiple linear rings – the first ring is the polygon’s outline, and any subsequent entries are holes within the polygon. You can therefore ‘invert’ a polygon by having the outer ring be something surrounding the polygon, then having the polygon be a hole within that.

  So the way we have implemented this in our new version is to fetch the area from MapIt, but before plotting it preface it with a giant rectangle covering the whole world. (Thankfully, any holes within the original shape are inverted too, though that is implementation dependent and it would have been a lot harder if they weren’t!).

  There were a couple of issues along the way. MapIt may return either a polygon or multiple polygons, so we needed to deal with each differently (we turn multiple polygons into one polygon, which also thankfully Just Works). And OpenLayers has a hardcoded maximum pixel co-ordinate for SVG rendering, dating from 2007 and an issue in Firefox 2. We hope that any such issues have been fixed by now. We haven’t had any reports of crashes yet, anyway…

• Version 2.0 – New user system 7 December 2016

  Version 2 launches a new user system for admins, including more granular permissions and a variety of new features.

  Logging in

  Admins can now use the same login system as the main front end. We highly recommend running your site over HTTPS in order to keep credentials secure; LetsEncrypt can supply free 3-month certificates.

  The main admin user type is the ‘superuser’; a user with this flag set (which can be done in the database, or by running the createsuperuser command) has full access to all areas of the admin, just like admins under the old system. By default, these are the only types of user allowed to log in to /admin. (It is possible to change this, e.g. in the UK, non-superuser admin users associated with a body can log in to /admin on their own cobrand.)

  User permissions

  

  Users associated with a body (called ‘body users’) can be given a variety of different feature-based permissions; the screenshot here shows the list of different options.

  • Categories: You can associate a user with a list of categories, which e.g. pre-selects those categories when the user visits the All Reports page.

  • Response priorities: This allows you to set a list of different priorities for a body, or again for a particular category in a body, letting you note different priorities for different reports.

  • Response templates: You can create and edit templates associated with your body, or with a particular category in that body, and then when leaving an update you can select one of these templates to allow easy updating of reports.

  • You can give a user access to a front-end report ‘inspect’ view, which lets a user edit a report’s category, state, or location. If the category change moves the report to a different body, it will be re-sent. Alternatively, a user can be given only category edit or priority edit permission. Here is a screenshot of the top of an inspect form view:

  

  Create as another/body

  This permission gives a user the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user. Below is a short animation showing this in action on the Oxfordshire cobrand of FixMyStreet.com:

  

  Shortlists

  A user with the shortlist permission gains a shortlist button on each report; clicking this adds the report to your own personal shortlist of reports, which you can view in a section of Your Account. This may be useful for an ‘inspector’ type of admin user, who wishes to compile the day’s list of reports before going out and investigating them. You can also see if a report is on someone else’s shortlist, and take it off them if you need to.