Blog

Version 5.0 10 May 2023

I am happy to announce the release of version 5.0 of FixMyStreet.

Mobile use and the Progressive Web App (PWA)

Mobile reporting now uses a crosshairs mechanism for locating the report rather than tapping, matching the old native app and providing an easier experience. We have added offline report drafting, where even if you do not have signal you should be able to start the basics of a report to carry on when back online.

The site is installable as a Progressive Web App (PWA) on Android and iOS, to be installed to your home screen via your browser, and both the UK and Sweden have successfully packaged the code as a native app using pwabuilder and submitted it to the app stores.

Accessibility

Early last year, we added keyboard navigation of the map for those who may be unable to use a pointing device. An accessibility audit was then performed on one installation of the site later in the year, and that has led to a number of other improvements, including some improved visual contrast, a couple of missing labels and fieldsets, and better focus states.

Admin improvements

You can now set a message in the admin per-body that will be shown on the reporting page, as well as one on the front page. We’ve paginated alerts on the user edit page for cases where a user has a lot of alerts, and now display photos in report moderation updates rather than just the image hashes.

If you are using response templates, you can now have a different template displayed as an update to the report to the text emailed to the user.

Development improvements

As our own installation of FixMyStreet spans multiple front end servers, and to help with stability of URLs for the PWA service worker, we have switched our static asset cache-busting from using last modified times to using a digest of the file contents. There is also an option to use the hash in the filename itself, rather than a query parameter, which again may be useful if you deploy on multiple servers.

Upgrading

A full list of changes can be seen in the v5.0 changelog as usual.
Version 4.0 3 December 2021

Well, 2021 has certainly been a year, hasn’t it. Apologies for the delay between releases, but we are now releasing version 4.0, with a number of new features, outlined below. For those who don’t want to update immediately, we are also releasing version 3.1.1 as an update to 3.1 including a number of bugfixes and the update necessary to deal with issues caused by the rollover of the Let’s Encrypt root certificate.

Multi-page reporting form

The main change in this release is the move to a multi-page reporting form, where you pick the category first, then the subcategory if relevant, any extra questions for that category, photos, problem details, and then user details last. Research shows people find a form easier to fill in this way, and it makes it clearer what is expected at each step of the process. We have also switched from a category drop-down to radio buttons, as this is easier to use, especially on mobile.

Photo redaction

We have added photo redaction support, letting you moderate parts of a photo that should not be public without removing the whole photo, and photos can also be moderated individually.

Development Docker environment

We now include a development Docker environment, which should hopefully make it easier for people to spin up a local copy of the code for development. If you have Docker and Docker Compose installed, a fresh clone and then docker/compose-dev up should set everything up for you.

Other improvements

You can now specify a radius when signing up for an email alert, the mobile site has an improved navigation menu, users can set global notification preferences, and the search box supports Maidenhead Locator references.

A few admin pages have had their layout improved, and you can now customise hints on a per-category basis. The CSV export includes device type, staff can find non-public reports when using the front page ID search, and user login attempts can be throttled per user.

The alert script has been split up, so you can e.g. choose to send local alerts at a different frequency to update alerts.

Bugfixes

As always, we have fixed a number of issues that have arisen – for example, if a category has more than 10 extra questions, they should now be sorted in the correct order, there was a bug in update text moderation, and categories with slashes in no longer break the CSV export.

Upgrading

A full list of changes can be seen in the v4.0 changelog or v3.1.1 changelog as usual.
Version 3.1 16 November 2020

I hope you are all okay with the ongoing situations around the world. We are happy to release version 3.1, with some new features and bugfixes.

Front end improvements

A random assortment of improvements to the front end, including aerial maps for Bing and OSM maps, lazy image loading on list items, the ability to check passwords against Have I Been Pwned, and Open Location Codes support in the search box.

Admin improvements

We’ve added a full text index so searching reports in the admin should be much quicker. You can disable updates or reopening on a per-category basis, as well as enable anonymous reporting. CSV generation is now done asynchronously, with progress shown, in case it takes a long time. Staff users can use HTML in updates, and response templates can also include HTML. The site now records whether a report was made on desktop or mobile, and includes any extra fields in email submissions to bodies. Staff status can be removed from users in bulk, and inspectors can update the asset on a report via the inspector form.

Bugfixes

As always, a wide variety of issues covered here. We added PostgreSQL 12 compatibility to the schema update script, fixed an issue where the CSV export returned multiple entries per row, added a fix for photo orientation in modern browsers, and improved the moderation diff display.

Upgrading

A full list of changes can be seen in the changelog as usual.
Version 3.0.1 6 May 2020

First of all, I hope you are all safe wherever you may be. We held a couple of FixMyStreet user groups recently for body users of FixMyStreet in the UK, and someone from a large body said that their FixMyStreet installation was proving really useful in these times, as they were making far fewer internal reports due to the UK lockdown.

Admist all this, we continue to develop the software, and today are releasing version 3.0.1, a bugfix release with a couple of new features.

Admin improvements

You can now provide an automatic initial update on reports made in a particular category to a particular body, perhaps to provide information on timings or similar. Make sure the body has a user to associate comments with, and then add a response template in the Open state with auto-response checked, restricted to a list of categories if you wish. After that, the template will be used to provide an initial update on new reports made.

We have also added “staff-only” categories, which are categories that staff users can see but normal users cannot. One council is using this to provide emergency categories that their contact centre staff can use to make reports after manual triage.

The dashboard export and report search should now be quicker, after some investigation that area. We’ve also moved the overall stats off the index page to the stats page, so the index page loads more quickly.

Bugfixes

Thanks to those of you letting us know about bugs or problems. Those we have fixed include an incorrect To header on emails about inactive accounts, a couple of issues with the front page recent reports list showing different results depending on whether the cache was used or not, and a double escape in the Google Maps URL.

Others we have fixed include maintaining the category group on pin move with same category in multiple groups, and fixing sorting by most commented on the /around map view.

Development improvements

As well as the cron scripts, this release now includes a dæmon that you can use to send reports and updates. Using the dæmon will mean reports and updates are sent almost immediately after they are confirmed, but will require a bit more setup, as you’ll need to set it up as a dæmon running all the time in your system. We’ve provided an example config file for systemd to hopefully help with that. If you do run the dæmon, be sure to remove the lines of your crontab that send reports and updates :)

Alongside that, send-reports no longer prints out failures in verbose mode, there is a separate send-reports-failure-summary script to do that. Also the fetch-comments and fetch-reports scripts have been consolidated into one fetch script, which can also now parallelize fetching.

Upgrading

A full list of changes can be seen in the changelog as usual.
Version 3.0 4 March 2020

It has been quite a while since the last release, apologies, but today we are happy to be releasing version 3.0 of the FixMyStreet Platform, which has a number of improvements.

Front end improvements
- FixMyStreet can now be installed as a progressive web app. This means we’ve added a web manifest (and an admin UI for managing this) and a basic service worker that shows a page if you’re offline, and continues the functionality of allowing staff users to store and view their shortlisted reports offline.
  
  If you serve your site over HTTPS, you will be able to add the website to your homescreen (browsers may prompt the user) and have it work like an app. This provides us with a solid base on which to continue improving this in future, including hopefully adding functionality such as offline report drafting through the web site.
- Various improvements have been made to the site on mobile – the “try again” process is clearer, duplicate suggestions show an inline map, the photo upload message is better, and map filters can now be accessed.
- Category groups are now used wherever a category list is shown – admin pages, map filters, and so on; and you can pass a filter_category or filter_group parameter to the front page or around page to pre-select that option, which makes it easier to deep link to FixMyStreet from a page or form on another site.
- If you use geolocation, your location will now be displayed on the map, as shown in the screenshot.
- As asked for a few times on our mailing list, we now use a report’s image as its OpenGraph image on an individual report page when shared.
- We’ve added XSL to our RSS feeds which means browsers no longer display them as raw XML but as a nice simple web page that explains its purpose. Before and after shots below:
Security

All template variables are now automatically escaped by default, to help protect against any future XSS vulnerabilities. We also rotate the user’s session ID after successful login, and scrub the admin description fields.

If any of your own templates outputs a variable that contains HTML that you wish to continue to allow to display as HTML, you will need to alter your template to escape the variable with the safe filter, e.g. [% some_html | safe %].

Admin improvements
- FixMyStreet now has a new roles system, allowing you to create groups of permissions and apply those roles to users.
- The category edit form has been drastically improved; category names can now be edited, categories can be listed under more than one group, and categories or particular extra questions can disable the reporting form (for e.g. emergency “please call” categories or questions).
- Two-factor authentication can be used by any staff member, and you can choose to optionally enforce it for all staff.
- The admin report edit page now stores moderation history, like the front end, and you can now view a user’s admin log history.
- We’ve added a heatmap dashboard for staff users, which can show hotspots. To enable this, you will need to add heatmap: { yourcobrand: 1 } to your COBRAND_FEATURES configuration.
- There’s a new “staff only” contact state, for categories that can only be used by staff.
- Staff users can report as other users even if they only have a name, and can sign other people up to alerts.
Bugfixes

Of course there have been a lot of bugfixes as well. One I remember is when going back to the initial state with popstate, a change event was being triggered on every single option of the filter selects. This led to a lot of change events running on the category/status multi-selects which then needlessly repeated the same activities over and over. This locked up the browser for seconds in locations with many categories. Below is a chart showing browser performance before and after:

Development improvements

We’ve upgraded the underlying framework and other packages, added a banner to the staging website/emails to make it obvious when you’re in development, added configuration for admin resending, a Content-Security-Policy header, and stopped hard coding the site name in the database fixture.

Open311 improvements
- It is now possible for an external Open311 service to POST updates on a report to FixMyStreet, rather than have FixMyStreet poll an external service for updates.
- Email templates can include a placeholder to include the description fetched from the Open311 server in the update.
- Private reports are supported, in that an Open311 server can mark a category as private which will then automatically mark all reports sent and received in that category as private.
- Meta questions added in the admin can be marked as protected so that they won’t be overridden by data fetched from an Open311 server. This is useful for e.g. an “emergency” question that the Open311 server does not care about.
Upgrading

As mentioned above, but it is worth repeating, if any of your own templates outputs a variable that contains HTML that you wish to continue to allow to display as HTML, you will need to alter your template to escape the variable with the safe filter, e.g. [% some_html | safe %].

A full list of changes can be seen in the changelog as usual.
Version 2.6 29 April 2019

Image © Ben Waddington

Today we have released version 2.6 of FixMyStreet.

This release fixes a cross-site scripting security issue where someone could create a report through the site with a specially constructed query parameter, and then viewing that report on the admin report edit page would allow the report creator to run their own JavaScript. We have also released version 2.5.1 which is identical to 2.5 including this fix.

There’s a new, optional, feature to auto-suggest similar nearby problems while reporting, to discourage duplicate reports; and the map state is now updated in the URL to make sharing links easier. A bit more work has been done on moderation, spotting conflicts and showing moderation history to staff on report pages, as well as in the admin.

Mostly this release is bugfixes, please see the changelog for full details.

Upgrading

The admin body and user sections have been refactored – if you have custom templates/code, you may need to update links to those.

If you wish the default for the showname checkbox to be checked, you can add sub default_show_name { 1 } to your cobrand file.
Version 2.5 21 December 2018

Today we have released version 2.5 of FixMyStreet; happy solstice!

This release contains a number of front end improvements, especially to the sign in flow. This was the result of a lot of work to simplify the whole process, and we’ve written more about it on the mySociety blog.

Other front end improvements range from a more prominent display of “state” on report page, making sure you don’t need two taps on a report list entry on a touchscreen, and clearer relocation options while you’re reporting a problem.

Moderation has had some work done – it can now potentially edit a report’s category, should you wish it to do so; it stores more of a report’s original data upon moderation, and also now stores all moderation history, making that visible in the report/update admin interface.

Bugfixes include one reported by FixaMinGata (back in early 2017!) about better map zoom behaviour when clicking on pins and going back; not escaping HTML entities in report titles pulled in by Ajax; and show missing reopening/fixed questionnaire responses when tey lacked their own separate update. We also fixed some issues with our Open311 contact group handling, and improved the validation of fetched reports timestamps.

We now strip Exif data on uploaded photos, and have added a new config variable, SYMLINK_FULL_SIZE, that can be used to symlink your full size photos out of the photo cache rather than copy them, if your static files are being served by your web server.

Full changelog

Upgrading

Due to the sign-in and banner changes, this release changes a number of the base templates. If you have overridden any related templates in your cobrand and not fed those changes upstream, you may need to make adjustments to match; please use our bin/cobrand-checks script to help make comparisons between your changes.
Version 2.4.2 6 November 2018

Today we have released version 2.4.2 of FixMyStreet.

This is mainly a bugfix release, but does contain a couple of new features - the admin lets you see and manage a user’s alerts, and the dashboard has update CSV export.

Bugfixes include an issue when setting up a new Vagrant-based installation, updating the areas column when a report is moved by a staff user, giving superusers access to update staff dropdown, and an issue with subcategories when visiting /report/new directly.

We have renamed a few SCSS variables for clarity; the old names still work, but if you wish to update your cobrand, then change $map_nav_bg to $nav_background_colour, $nav_fg to $nav_colour, and $nav_fg_hover to $nav_hover_background_colour.

Full changelog
Migrating to S3 photo storage 8 October 2018

The recent release of FixMyStreet v2.4.1 brought with it the ability to store photos for reports & updates in a bucket on Amazon’s S3 instead of locally on disk.

Getting a new site configured to store newly uploaded photos in S3 is explained in the configuration pages, and is a matter of tweaking a few keys in your general.yml file. But what about if you have an existing FixMyStreet installation that you would like to migrate to storing all photos in S3? This post outlines a process for achieving just that - without any downtime.

Prerequisites
- A running FixMyStreet site with at least one report or update with a photo
- An account on Amazon AWS, and access/secret keys for a role that can create/manage S3 buckets
- The AWS CLI tool installed on your FixMyStreet server, and configured with access credentials
Creating the bucket

First off, we need to create a bucket to hold all the photos. If you’ve already done this, skip ahead to the Migrating photos section.

We’ll show two ways to create a bucket: via the S3 web console and using the aws command line tool.

Using the S3 web console

Log in to S3 web console, and click the ‘Create Bucket’ button

Enter a name for the bucket, e.g., my-fixmystreet-photos, and choose the region closest to your FixMyStreet server (more about buckets and regions):

Your newly created bucket should then appear in the list of buckets. Make sure public access isn’t allowed (photos are served to users via your FixMyStreet server, not directly from S3):

If everything looks OK, skip to the Migrating photos section.

Using the aws CLI tool

You can also create buckets from the command line using the aws tool.

If you’ve not already done so, now is the time to install and configure the aws CLI tool.

To create a bucket named my-fixmystreet-photos in the eu-west-1 region, run the following:
```
$ aws s3 mb s3://my-fixmystreet-photos --region eu-west-1
```
The following success message should be shown:
```
make_bucket: my-fixmystreet-photos
```
The bucket’s region can be inspected:
```
$ aws s3api get-bucket-location --bucket my-fixmystreet-photos
{
    "LocationConstraint": "eu-west-1"
}
```
The bucket should be created as private by default, but you may wish to check its access policy on the S3 web console.

Migrating photos

Now that the bucket has been created, the next step is to fill it with all the photos from your existing FixMyStreet installation. For this we’ll need the aws CLI tool, so make sure you’ve installed and configured it.

Existing photos are stored on disk in the directory specified by the UPLOAD_DIR (or PHOTO_STORAGE_OPTIONS.UPLOAD_DIR) configuration key in conf/general.yml. For this example, let’s assume that’s /home/fixmystreet/photos.

Running the following (with the path and bucket name altered to suit your configuration) will make copy-pasting the subsequent commands a little easier:
```
$ export UPLOAD_DIR=/home/fixmystreet/photos
$ export S3_BUCKET=my-fixmystreet-photos
```
We’ll use aws s3 sync to copy all photos to the new S3 bucket. To get a preview of the operations it will perform without actually copying anything, run:
```
$ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET --dryrun
```
You should see output similar to the following, one line for each file:
```
(dryrun) upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
(dryrun) upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
(dryrun) upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
[...]
```
If the output looks correct, re-run without the --dryrun flag to actually copy everything to the S3 bucket:
```
$ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET
upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
[...]
```
All your photos are now in S3, so the next step is to tell FixMyStreet about the new bucket. (See the note in the final section for reports made between now and when the FixMyStreet config has been updated to use S3.)

Configuring FixMyStreet to use S3

We need to make a few changes to conf/general.yml to configure FixMyStreet to use the new S3 bucket for photo storage.

Your conf/general.yml probably has a section similar to the following:
```
PHOTO_STORAGE_BACKEND: 'FileSystem'
PHOTO_STORAGE_OPTIONS:
    UPLOAD_DIR: '../photos'
```
Or, if you set the site up before v2.4.1:
```
UPLOAD_DIR: '../photos'
```
Either way, you need to replace that section with the following:
```
PHOTO_STORAGE_BACKEND: 'S3'
PHOTO_STORAGE_OPTIONS:
    BUCKET: 'my-fixmystreet-photos'
    ACCESS_KEY: 'AKIA12345'
    SECRET_KEY: '1234/1234'
```
Make sure you set BUCKET, ACCESS_KEY and SECRET_KEY to the same values as when you configured the aws CLI tool and created the bucket.

These changes are all that’s needed to switch your FixMyStreet installation to using S3 for photos. Once you’re happy with the changes you’ve made to conf/general.yml, restart the FixMyStreet app server (example) and check the output for any errors.

Once the server has restarted, try making a new report and uploading a couple of photos. You should see new files appear in the S3 bucket (you can view them via the S3 web console) as soon as you’ve drag-dropped them into the new report (or update) form. Check that report photos appear on the site for newly-created reports too. NB photos for existing reports are likely already cached on disk by FixMyStreet, so do check a new report.

Final steps & tidying up

Because there was a small window of time between syncing existing photos to the S3 bucket and switching over the conf/general.yml config, there’s a chance that a report with photos was made whose photos weren’t synced to S3. Now that FixMyStreet is using S3, you may wish to re-run the aws s3 sync command from above in order to make sure any such photos have been transferred to S3.

The old directory containing photos on disk isn’t needed any more, so once you’re happy the S3 storage is working as expected, you can delete this.
Version 2.4.1 1 October 2018

Today we have released version 2.4.1 of FixMyStreet.

This release contains a wide variety of new features for use by reusers of our software, funded by NDI as part of their FixMyCommunity project; thanks very much to them. It also includes a number of smaller features and bugfixes.

FixMyStreet now comes with a Dockerfile and example Docker Compose setup for use by those tools, and we provide a hosted Vagrant box that can be used also. You can choose to store photos in Amazon S3 rather then the local filesystem, use an external memcache server,

We’ve added a script, cobrand-checks, that will list/diff template changes in core that might need applying to a cobrand, to help with upgrading, and added more checks to our Pull Requests on GitHub to spot included cobrands that need checking with each PR made.

We have sped up fetching full lists of bodies to help installs that have thousands of bodies, sped up the display of /_dev pages, fixed the display of pins on a body page when using Bing or TonerLite maps, and fixed an issue where the from_body field was lost on a user when edited by a non-superuser admin.

Full changelog