Version 2.1.1 3 August 2017
Today we have released version 2.1.1 of FixMyStreet, a bugfix release with a few small improvements.
The map in HTML emails is now a link through to the report page, and we’ve fixed a bug causing the wrong pin size to be shown there. We’ve also fixed a bug that could cause a different language from the requested one to be shown on about pages.
Admin improvements include resending reports if changing the category has changed the
send_methodand displaying the reporter’s phone number on the inspector form. Text deleted during moderation is no longer replaced with
Version 2.1 18 July 2017
Today we have released version 2.1 of FixMyStreet, with some new features and a variety of improvements for users and reusers.
New features include allowing users to hide their own name on reports/updates, and a new graph-based /reports page, which you can see in operation on our main UK site. We also now resize photos client-side before uploading, given the increase in very large images these days.
Admin improvements include adding an ‘inactive’ state to categories, so you can prevent a category being used for new reports but still have it available in filters, various improvements to the inspect form and the new report process for inspectors, and an easier way for inspectors to shortlist all reports in their view.
Geolocation is back on the alert page, lost sometime last year, and some random blank spaces are no longer clickable, hooray!
Development-wise, FixMyStreet now supports Debian stretch and perl 5.24, badly configured SMTP options should now be spotted, and we have refactored and simplified the CSS used for header/content/navigation. This may mean you can simplify overrides in your cobrand. If you wish to have a static front page rather than the normal report page, that’s now possible by creating an
about/homepage.htmltemplate in your cobrand, and cobrands can now easily change the new report pin colour.
The test suite now runs each file in a transaction which means it can be run in parallel and sped up dramatically; on the other hand, on Travis we added code coverage which slowed it down again.
Two backwards incompatible changes that may require changes to your own templates if you have customised them:
nav-wrapper-2class has been removed. If you have a custom footer template, replace that class with ‘container’.
/reportspage now uses different generated data. If you have a custom
reports/index.htmltemplate and wish to keep it, you may need to call
--tableargument to generate the old style data.
Version 2.0.4 13 April 2017
Today we have released version 2.0.4 of FixMyStreet, a bugfix release along with a couple of other improvements.
The bugfixes are updating our Facebook library so that Facebook login works again, stopping an error if you had a devolved body and its contact both lacking a send method, and an issue in the multi-select front end if you had characters such as brackets in your category name.
Strangely, while the backend has always allowed multiple email addresses for a contact, separated by commas, the admin interface hasn’t allowed them to be entered - this has now been corrected.
Lastly, body pages under
/reportswill now limit the reports shown to those within the visible map, which should make moving and zooming around much more intuitive.
Version 2.0.3 31 March 2017
Today we have released version 2.0.3 of FixMyStreet, a bugfix release along with some other improvements.
The map on a mobile report page can now be made full screen, and if you are using Google Maps you can supply a custom map styling. There’s also now a loading indicator whilst data is being fetched for the map.
Various missing translations have been added, for moderation, social login, and offline usage.
We’ve upgraded our email sending to deal with issues sending with SSL, dealt with IE11 caching report Ajax calls too aggressively, and with Safari 5/ Android 4 not showing our questionnaire answer buttons.
Lastly, all the test suite can now run offline, and amusingly I found a bug in a test that only happened if the test was run c. 55 hours before daylight savings time began :)
Version 2.0.2 3 February 2017
Today we have released version 2.0.2 of FixMyStreet, a bugfix release along with some new admin improvements.
The main bugfixes are to mark two missing strings for translation, to make sure email is lowercase when signing in via
/auth, and to make sure a language subdomain can be included in calls to
base_url_for_report. There are also some CSS fixes to the homepage and an improved print layout for report list pages. Fixes to the admin interface include fixing filtering on the shortlist page, and fixing the ‘save with public update’ toggle.
For international users, there have been a few string renames, but we have maintained translations as the concept remains unchanged, so please only feel free to update if you consider it necessary:
- ‘unable to fix’ has been renamed to ‘no further action’
- ‘marked as a duplicate report’ has been changed to ‘closed as a duplicate report’
An offline fallback page has been added using appcache, which inspectors can use for offline updating of reports in their shortlist.
Admin improvements include allowing response templates to be associated with a state (so that template is used by default when the state is changed), allowing shortlist addition/removal from report lists, and shortlist reordering. Users with a new permission can see the body user who left a
contribute_as_bodyreport or update.
Version 2.0.1 16 December 2016
Today we have released version 2.0.1 of FixMyStreet, a bugfix release. The issues fixed are:
- strftime output (e.g. used to display dates) was sometimes being double decoded, giving incorrect output in some languages.
- If a category was filtered, it was not being properly carried through to the new report form.
- The body/area inputs in the admin were being fixed to a too-small height.
Some bugfixes and improvements have also been made to the admin inspector report view, along with a couple of development improvements – the tests coping better if run on a network that intercepts NXDOMAINs, and a better way of showing the git version in the admin config.
It also let us add the Content-Security-Policy header to FixMyStreet, which is a method in browsers to prevent cross-site scripting (XSS), clickjacking and other code injection attacks by specifying the valid sources for script execution.
Separate scripting and styling
When you report an issue on FixMyStreet on mobile, the map
css()calls when setting up (or ending) the mobile view. It was straightforward to move this CSS to a
only-mapclass to prevent the map being scrolled until it is clicked, when the rest of the form can then be shown.)
Now that I had a
mobile-reporting-mapclass, I wanted this class activated as soon as possible as the page is loading, not only when the document had been parsed. There were also a couple of site-wide variables,
page(the type of page, e.g.
To achieve all this, I created a
header.jsfile that performed the above three tasks, setting a class on <html>, setting two variables on our global
fixmystreetvariable, and if we’re on a small width (using Modernizr) and perhaps a map page, setting the appropriate classes. I then minimized and inlined this script in the header of each page, so that we don’t have to wait for any external script to load.
“nonce” was only added in the second version of the CSP spec, so you may note our header also specifies
unsafe-inline. Any browser that supports version 2 will ignore this when it sees the nonce header, but it is needed in order for the inline script to still run in any browser only supporting version 1.
In Google Page Speed Insights, with manual minification of the main JS files, this moves the front page from 68/84 to 85/92ish (filmstrip from webpagetest.org, top is live site, bottom is my dev site):
These are requests from the US: most of the initial delay is in that initial download. Now here’s a report page going from 58/77 to 85/86ish (“ish” because e.g. live site will have analytics that my dev site doesn’t):
Version 2.0 – testing improvements 15 December 2016
FixMyStreet has a large and hopefully comprehensive test suite that runs through all aspects of the codebase, checking everything is working. This makes it easier to change code and add new features, safe in the knowledge that any breakages will be quickly highlighted.
Speeding up the tests
Every time someone commits code to our GitHub repository, or opens a pull request, the tests are automatically run for us by Travis CI. We’re alerted to success or failure with little green ticks or red crosses on GitHub, and by notice in IRC.
The tests seemed to have slowed down considerably in recent times, but we couldn’t identify any changes at the FixMyStreet side which might have caused this.
However, there had recently been some spam scraping of Gaze, our web service that provides population density information to FixMyStreet (so that e.g. the maps can try and guess an appropriate zoom level, and so alerts can try and guess an appropriate radius), and rate limiting had been added to try and help combat it.
Dave spotted that this was being triggered by FixMyStreet test runs, leading to pauses as the suite waited for the rate limiting to ease. Thankfully, all Gaze calls were being routed through one function (that had been created in order to cope gracefully with a Gaze failure) and so it was a simple matter for this function to be stubbed out if being run as part of a test.
There are many tests that still rely on the internet (e.g. for some MapIt lookups) and eventually it would be good to get to the point where they are all stubbed out and the test suite can run completely offline, probably even more quickly.
Multiple test running
When running the tests, the suite creates a test database (in PostgreSQL terms, it actually creates a temporary cluster) so that anything it does won’t affect your development database. Theoretically, this means you should be able to run the test suite multiple times simultaneously – perhaps it’s doing a full run, but you want to try and fix (and retest) the first error while it carries on. However, this was not working, and after some investigation it turned out that each run was creating (and overwriting) a test configuration
.ymlfile, which meant the existing runs got all confused. Adding a process ID to the test configuration file meant that each run is independent and can successfully coexist with each other.
Lastly, you used to have to run the full suite with
bin/run-tests t, but now if you run
bin/run-tests, it will assume you meant
t. A small thing, but it might save a few seconds over the years. ;-)
FixMyStreet has had a nice multiple image uploader since version 1.8. This uses multiple
input type=filefields, progressively enhanced to add drag’n’drop, image preview, and uploading in the background whilst you fill in the rest of the form.
For this new version, we had a different bug to fix. If the user had uploaded a picture, submitted the form, and was shown the form again due to a server side error of some sort (some validation not caught by client-side validation, for example, or because you were logging in during the reporting process), the image for the preview was then being loaded from the server (where it had already been uploaded), not the client, and not displaying. We patched the exif-js library., Now, if it is given a URL rather than a data: string, it will go off and fetch the image so that it can read out the orientation data.
Version 2.0 – Improved forms 13 December 2016
The new release of FixMyStreet includes a number of improvements to various forms on the site. In this post, we will take a brief look at the notable changes.
Public reporting form
This form has been rejigged, in order to more obviously split out details that will be public (e.g. photos, details) from those that will not be published on the site (e.g. the user’s email address and phone number). The category selector has also moved to the top, and if the category chosen requires the display of extra questions or information (e.g. through Open311 attributes or a custom built asset layer), they will be shown immediately.
If a report is made in an area that is covered by more than one body, the category the user selects will normally dictate which one the report is sent to. Now, when the category is selected, we update the list of bodies given at the top of the report page, if we know that the report will be sent there.
Talking about custom built asset layers, this is a good place to show how the FixMyStreet codebase can be put to other uses, with a bit of development.
Angus Council in Scotland provide a WFS layer (that is, vector format geographic information) containing the locations of all their streetlights, which we display if the street lighting category is selected within Angus on FixMyStreet.
Importantly, it can display which lights Angus already knows are broken. If the user is able to identify precisely which street light is affected, they can click on it. But picking a street light isn’t mandatory: we don’t want to put people off who aren’t certain, or who are unable to select an individual light. You can see an example of what this looks like on the Angus cobrand of FixMyStreet.com below.
Admin report editing
Previously, the form for administrators to edit a report was functional, but certainly nothing more than that! Due to the work we’ve done on the new user system, more admin users may well be accessing this form in future, and so we’ve taken the opportunity to make it much tidier.
It now looks much more like the front end of the site. We’ve added a map that lets you move the location of the report, tidied up the various functions an admin can perform, and so on. If a category change means the report should have been sent to a different body, it will be re-sent.
Admin category editing
This was a historical oddity, in that the Add category form and the Edit category form were completely separate creatures, though both contained the same fields, and were used for basically the same purpose:
You can see how they looked different in the screenshots above. They now share an HTML template, which also makes it easier for us to update should it need changing in future.