FixMyStreet Platform

Version 6.0

2024-11-14T00:00:00+00:00

I am happy to announce the release of version 6.0 of FixMyStreet.

Some features in this release include:

The ability to add a category filter to the reporting process, letting people search for a particular category. (This involves adding a category_filter.html template to your cobrand, similar to the one used on fixmystreet.com.)

Improved display of the filters on map pages:

A new geolocation button to centre the map at your location:

Adding the report ID reference to lists of reports in emails:

We redirect to a confirmation page when creating a report, reducing the risk of Safari creating a duplicate report
Cobrands can have different map display times for open/closed/fixed reports
All uploaded images are now converted to JPEGs
Users can opt out of questionnaires

Admin improvements

We have improved the navigation in the admin, moving the main navigation to a sidebar. This is accompanied by minor display and performance improvements in various admin pages.

Extra questions can now be multiple-choice using checkboxes, or a datetime picker, in addition to the existing options of text field, textarea, and single value. You can also make a phone number required for a particular category.

On the dashboard, you can now filter by multiple categories, and we’ve fixed the CSV export to not ignore reports that had updates that were only unconfirmed or hidden.

A long standing bug viewing a category when it has more than 1,000 history entries has been fixed.

Development improvement examples

FixMyStreet has been around for a long time, and implemented its own way of sorting extra data on a per-report or update basis. Since then, PostgreSQL has added JSON support, and we have now moved the database tables to store data in this standard way.
The site can now automatically spot a favicon.ico file present in the web/cobrands/<your-cobrand> directory, rather than having to add your own header template.
Cobrands can provide (optionally per-category) custom distances for duplicate lookups
The daemon will deal with waiting reports in a random order

Upgrading

A full list of changes can be seen in the v6.0 changelog as usual.

Version 5.0

2023-05-10T00:00:00+00:00

I am happy to announce the release of version 5.0 of FixMyStreet.

Mobile use and the Progressive Web App (PWA)

Mobile reporting now uses a crosshairs mechanism for locating the report rather than tapping, matching the old native app and providing an easier experience. We have added offline report drafting, where even if you do not have signal you should be able to start the basics of a report to carry on when back online.

The site is installable as a Progressive Web App (PWA) on Android and iOS, to be installed to your home screen via your browser, and both the UK and Sweden have successfully packaged the code as a native app using pwabuilder and submitted it to the app stores.

Accessibility

Early last year, we added keyboard navigation of the map for those who may be unable to use a pointing device. An accessibility audit was then performed on one installation of the site later in the year, and that has led to a number of other improvements, including some improved visual contrast, a couple of missing labels and fieldsets, and better focus states.

Admin improvements

You can now set a message in the admin per-body that will be shown on the reporting page, as well as one on the front page. We’ve paginated alerts on the user edit page for cases where a user has a lot of alerts, and now display photos in report moderation updates rather than just the image hashes.

If you are using response templates, you can now have a different template displayed as an update to the report to the text emailed to the user.

Development improvements

As our own installation of FixMyStreet spans multiple front end servers, and to help with stability of URLs for the PWA service worker, we have switched our static asset cache-busting from using last modified times to using a digest of the file contents. There is also an option to use the hash in the filename itself, rather than a query parameter, which again may be useful if you deploy on multiple servers.

Upgrading

A full list of changes can be seen in the v5.0 changelog as usual.

Version 4.0

2021-12-03T00:00:00+00:00

Well, 2021 has certainly been a year, hasn’t it. Apologies for the delay between releases, but we are now releasing version 4.0, with a number of new features, outlined below. For those who don’t want to update immediately, we are also releasing version 3.1.1 as an update to 3.1 including a number of bugfixes and the update necessary to deal with issues caused by the rollover of the Let’s Encrypt root certificate.

Multi-page reporting form

The main change in this release is the move to a multi-page reporting form, where you pick the category first, then the subcategory if relevant, any extra questions for that category, photos, problem details, and then user details last. Research shows people find a form easier to fill in this way, and it makes it clearer what is expected at each step of the process. We have also switched from a category drop-down to radio buttons, as this is easier to use, especially on mobile.

Photo redaction

We have added photo redaction support, letting you moderate parts of a photo that should not be public without removing the whole photo, and photos can also be moderated individually.

Development Docker environment

We now include a development Docker environment, which should hopefully make it easier for people to spin up a local copy of the code for development. If you have Docker and Docker Compose installed, a fresh clone and then docker/compose-dev up should set everything up for you.

Other improvements

You can now specify a radius when signing up for an email alert, the mobile site has an improved navigation menu, users can set global notification preferences, and the search box supports Maidenhead Locator references.

A few admin pages have had their layout improved, and you can now customise hints on a per-category basis. The CSV export includes device type, staff can find non-public reports when using the front page ID search, and user login attempts can be throttled per user.

The alert script has been split up, so you can e.g. choose to send local alerts at a different frequency to update alerts.

Bugfixes

As always, we have fixed a number of issues that have arisen – for example, if a category has more than 10 extra questions, they should now be sorted in the correct order, there was a bug in update text moderation, and categories with slashes in no longer break the CSV export.

Upgrading

A full list of changes can be seen in the v4.0 changelog or v3.1.1 changelog as usual.

Version 3.1

2020-11-16T00:00:00+00:00

I hope you are all okay with the ongoing situations around the world. We are happy to release version 3.1, with some new features and bugfixes.

Front end improvements

A random assortment of improvements to the front end, including aerial maps for Bing and OSM maps, lazy image loading on list items, the ability to check passwords against Have I Been Pwned, and Open Location Codes support in the search box.

Admin improvements

We’ve added a full text index so searching reports in the admin should be much quicker. You can disable updates or reopening on a per-category basis, as well as enable anonymous reporting. CSV generation is now done asynchronously, with progress shown, in case it takes a long time. Staff users can use HTML in updates, and response templates can also include HTML. The site now records whether a report was made on desktop or mobile, and includes any extra fields in email submissions to bodies. Staff status can be removed from users in bulk, and inspectors can update the asset on a report via the inspector form.

Bugfixes

As always, a wide variety of issues covered here. We added PostgreSQL 12 compatibility to the schema update script, fixed an issue where the CSV export returned multiple entries per row, added a fix for photo orientation in modern browsers, and improved the moderation diff display.

Upgrading

A full list of changes can be seen in the changelog as usual.

Version 3.0.1

2020-05-06T00:00:00+00:00

First of all, I hope you are all safe wherever you may be. We held a couple of FixMyStreet user groups recently for body users of FixMyStreet in the UK, and someone from a large body said that their FixMyStreet installation was proving really useful in these times, as they were making far fewer internal reports due to the UK lockdown.

Admist all this, we continue to develop the software, and today are releasing version 3.0.1, a bugfix release with a couple of new features.

Admin improvements

You can now provide an automatic initial update on reports made in a particular category to a particular body, perhaps to provide information on timings or similar. Make sure the body has a user to associate comments with, and then add a response template in the Open state with auto-response checked, restricted to a list of categories if you wish. After that, the template will be used to provide an initial update on new reports made.

We have also added “staff-only” categories, which are categories that staff users can see but normal users cannot. One council is using this to provide emergency categories that their contact centre staff can use to make reports after manual triage.

The dashboard export and report search should now be quicker, after some investigation that area. We’ve also moved the overall stats off the index page to the stats page, so the index page loads more quickly.

Bugfixes

Thanks to those of you letting us know about bugs or problems. Those we have fixed include an incorrect To header on emails about inactive accounts, a couple of issues with the front page recent reports list showing different results depending on whether the cache was used or not, and a double escape in the Google Maps URL.

Others we have fixed include maintaining the category group on pin move with same category in multiple groups, and fixing sorting by most commented on the /around map view.

Development improvements

As well as the cron scripts, this release now includes a dæmon that you can use to send reports and updates. Using the dæmon will mean reports and updates are sent almost immediately after they are confirmed, but will require a bit more setup, as you’ll need to set it up as a dæmon running all the time in your system. We’ve provided an example config file for systemd to hopefully help with that. If you do run the dæmon, be sure to remove the lines of your crontab that send reports and updates :)

Alongside that, send-reports no longer prints out failures in verbose mode, there is a separate send-reports-failure-summary script to do that. Also the fetch-comments and fetch-reports scripts have been consolidated into one fetch script, which can also now parallelize fetching.

Upgrading

A full list of changes can be seen in the changelog as usual.

Version 3.0

2020-03-04T00:00:00+00:00

It has been quite a while since the last release, apologies, but today we are happy to be releasing version 3.0 of the FixMyStreet Platform, which has a number of improvements.

Front end improvements

FixMyStreet can now be installed as a progressive web app. This means we’ve added a web manifest (and an admin UI for managing this) and a basic service worker that shows a page if you’re offline, and continues the functionality of allowing staff users to store and view their shortlisted reports offline.

If you serve your site over HTTPS, you will be able to add the website to your homescreen (browsers may prompt the user) and have it work like an app. This provides us with a solid base on which to continue improving this in future, including hopefully adding functionality such as offline report drafting through the web site.
Various improvements have been made to the site on mobile – the “try again” process is clearer, duplicate suggestions show an inline map, the photo upload message is better, and map filters can now be accessed.
Category groups are now used wherever a category list is shown – admin pages, map filters, and so on; and you can pass a filter_category or filter_group parameter to the front page or around page to pre-select that option, which makes it easier to deep link to FixMyStreet from a page or form on another site.
If you use geolocation, your location will now be displayed on the map, as shown in the screenshot.
As asked for a few times on our mailing list, we now use a report’s image as its OpenGraph image on an individual report page when shared.
We’ve added XSL to our RSS feeds which means browsers no longer display them as raw XML but as a nice simple web page that explains its purpose. Before and after shots below:

Security

All template variables are now automatically escaped by default, to help protect against any future XSS vulnerabilities. We also rotate the user’s session ID after successful login, and scrub the admin description fields.

If any of your own templates outputs a variable that contains HTML that you wish to continue to allow to display as HTML, you will need to alter your template to escape the variable with the safe filter, e.g. [% some_html | safe %].

Admin improvements

FixMyStreet now has a new roles system, allowing you to create groups of permissions and apply those roles to users.
The category edit form has been drastically improved; category names can now be edited, categories can be listed under more than one group, and categories or particular extra questions can disable the reporting form (for e.g. emergency “please call” categories or questions).
Two-factor authentication can be used by any staff member, and you can choose to optionally enforce it for all staff.
The admin report edit page now stores moderation history, like the front end, and you can now view a user’s admin log history.
We’ve added a heatmap dashboard for staff users, which can show hotspots. To enable this, you will need to add heatmap: { yourcobrand: 1 } to your COBRAND_FEATURES configuration.
There’s a new “staff only” contact state, for categories that can only be used by staff.
Staff users can report as other users even if they only have a name, and can sign other people up to alerts.

Bugfixes

Of course there have been a lot of bugfixes as well. One I remember is when going back to the initial state with popstate, a change event was being triggered on every single option of the filter selects. This led to a lot of change events running on the category/status multi-selects which then needlessly repeated the same activities over and over. This locked up the browser for seconds in locations with many categories. Below is a chart showing browser performance before and after:

Development improvements

We’ve upgraded the underlying framework and other packages, added a banner to the staging website/emails to make it obvious when you’re in development, added configuration for admin resending, a Content-Security-Policy header, and stopped hard coding the site name in the database fixture.

Open311 improvements

It is now possible for an external Open311 service to POST updates on a report to FixMyStreet, rather than have FixMyStreet poll an external service for updates.
Email templates can include a placeholder to include the description fetched from the Open311 server in the update.
Private reports are supported, in that an Open311 server can mark a category as private which will then automatically mark all reports sent and received in that category as private.
Meta questions added in the admin can be marked as protected so that they won’t be overridden by data fetched from an Open311 server. This is useful for e.g. an “emergency” question that the Open311 server does not care about.

Upgrading

As mentioned above, but it is worth repeating, if any of your own templates outputs a variable that contains HTML that you wish to continue to allow to display as HTML, you will need to alter your template to escape the variable with the safe filter, e.g. [% some_html | safe %].

A full list of changes can be seen in the changelog as usual.

Version 2.6

2019-04-29T00:00:00+00:00

Image © Ben Waddington

Today we have released version 2.6 of FixMyStreet.

This release fixes a cross-site scripting security issue where someone could create a report through the site with a specially constructed query parameter, and then viewing that report on the admin report edit page would allow the report creator to run their own JavaScript. We have also released version 2.5.1 which is identical to 2.5 including this fix.

There’s a new, optional, feature to auto-suggest similar nearby problems while reporting, to discourage duplicate reports; and the map state is now updated in the URL to make sharing links easier. A bit more work has been done on moderation, spotting conflicts and showing moderation history to staff on report pages, as well as in the admin.

Mostly this release is bugfixes, please see the changelog for full details.

Upgrading

The admin body and user sections have been refactored – if you have custom templates/code, you may need to update links to those.

If you wish the default for the showname checkbox to be checked, you can add sub default_show_name { 1 } to your cobrand file.

Version 2.5

2018-12-21T00:00:00+00:00

Today we have released version 2.5 of FixMyStreet; happy solstice!

This release contains a number of front end improvements, especially to the sign in flow. This was the result of a lot of work to simplify the whole process, and we’ve written more about it on the mySociety blog.

Other front end improvements range from a more prominent display of “state” on report page, making sure you don’t need two taps on a report list entry on a touchscreen, and clearer relocation options while you’re reporting a problem.

Moderation has had some work done – it can now potentially edit a report’s category, should you wish it to do so; it stores more of a report’s original data upon moderation, and also now stores all moderation history, making that visible in the report/update admin interface.

Bugfixes include one reported by FixaMinGata (back in early 2017!) about better map zoom behaviour when clicking on pins and going back; not escaping HTML entities in report titles pulled in by Ajax; and show missing reopening/fixed questionnaire responses when tey lacked their own separate update. We also fixed some issues with our Open311 contact group handling, and improved the validation of fetched reports timestamps.

We now strip Exif data on uploaded photos, and have added a new config variable, SYMLINK_FULL_SIZE, that can be used to symlink your full size photos out of the photo cache rather than copy them, if your static files are being served by your web server.

Full changelog

Upgrading

Due to the sign-in and banner changes, this release changes a number of the base templates. If you have overridden any related templates in your cobrand and not fed those changes upstream, you may need to make adjustments to match; please use our bin/cobrand-checks script to help make comparisons between your changes.

Version 2.4.2

2018-11-06T00:00:00+00:00

Today we have released version 2.4.2 of FixMyStreet.

This is mainly a bugfix release, but does contain a couple of new features - the admin lets you see and manage a user’s alerts, and the dashboard has update CSV export.

Bugfixes include an issue when setting up a new Vagrant-based installation, updating the areas column when a report is moved by a staff user, giving superusers access to update staff dropdown, and an issue with subcategories when visiting /report/new directly.

We have renamed a few SCSS variables for clarity; the old names still work, but if you wish to update your cobrand, then change $map_nav_bg to $nav_background_colour, $nav_fg to $nav_colour, and $nav_fg_hover to $nav_hover_background_colour.

Full changelog

Migrating to S3 photo storage

2018-10-08T00:00:00+00:00

The recent release of FixMyStreet v2.4.1 brought with it the ability to store photos for reports & updates in a bucket on Amazon’s S3 instead of locally on disk.

Getting a new site configured to store newly uploaded photos in S3 is explained in the configuration pages, and is a matter of tweaking a few keys in your general.yml file. But what about if you have an existing FixMyStreet installation that you would like to migrate to storing all photos in S3? This post outlines a process for achieving just that - without any downtime.

Prerequisites

A running FixMyStreet site with at least one report or update with a photo
An account on Amazon AWS, and access/secret keys for a role that can create/manage S3 buckets
The AWS CLI tool installed on your FixMyStreet server, and configured with access credentials

Creating the bucket

First off, we need to create a bucket to hold all the photos. If you’ve already done this, skip ahead to the Migrating photos section.

We’ll show two ways to create a bucket: via the S3 web console and using the aws command line tool.

Using the S3 web console

Enter a name for the bucket, e.g., my-fixmystreet-photos, and choose the region closest to your FixMyStreet server (more about buckets and regions):

Your newly created bucket should then appear in the list of buckets. Make sure public access isn’t allowed (photos are served to users via your FixMyStreet server, not directly from S3):

If everything looks OK, skip to the Migrating photos section.

Using the `aws` CLI tool

You can also create buckets from the command line using the aws tool.

If you’ve not already done so, now is the time to install and configure the aws CLI tool.

To create a bucket named my-fixmystreet-photos in the eu-west-1 region, run the following:

$ aws s3 mb s3://my-fixmystreet-photos --region eu-west-1

The following success message should be shown:

make_bucket: my-fixmystreet-photos

The bucket’s region can be inspected:

$ aws s3api get-bucket-location --bucket my-fixmystreet-photos
{
    "LocationConstraint": "eu-west-1"
}

The bucket should be created as private by default, but you may wish to check its access policy on the S3 web console.

Migrating photos

Now that the bucket has been created, the next step is to fill it with all the photos from your existing FixMyStreet installation. For this we’ll need the aws CLI tool, so make sure you’ve installed and configured it.

Existing photos are stored on disk in the directory specified by the UPLOAD_DIR (or PHOTO_STORAGE_OPTIONS.UPLOAD_DIR) configuration key in conf/general.yml. For this example, let’s assume that’s /home/fixmystreet/photos.

Running the following (with the path and bucket name altered to suit your configuration) will make copy-pasting the subsequent commands a little easier:

$ export UPLOAD_DIR=/home/fixmystreet/photos
$ export S3_BUCKET=my-fixmystreet-photos

We’ll use aws s3 sync to copy all photos to the new S3 bucket. To get a preview of the operations it will perform without actually copying anything, run:

$ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET --dryrun

You should see output similar to the following, one line for each file:

(dryrun) upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
(dryrun) upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
(dryrun) upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
[...]

If the output looks correct, re-run without the --dryrun flag to actually copy everything to the S3 bucket:

$ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET
upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
[...]

All your photos are now in S3, so the next step is to tell FixMyStreet about the new bucket. (See the note in the final section for reports made between now and when the FixMyStreet config has been updated to use S3.)

Configuring FixMyStreet to use S3

We need to make a few changes to conf/general.yml to configure FixMyStreet to use the new S3 bucket for photo storage.

Your conf/general.yml probably has a section similar to the following:

PHOTO_STORAGE_BACKEND: 'FileSystem'
PHOTO_STORAGE_OPTIONS:
    UPLOAD_DIR: '../photos'

Or, if you set the site up before v2.4.1:

UPLOAD_DIR: '../photos'

Either way, you need to replace that section with the following:

PHOTO_STORAGE_BACKEND: 'S3'
PHOTO_STORAGE_OPTIONS:
    BUCKET: 'my-fixmystreet-photos'
    ACCESS_KEY: 'AKIA12345'
    SECRET_KEY: '1234/1234'

Make sure you set BUCKET, ACCESS_KEY and SECRET_KEY to the same values as when you configured the aws CLI tool and created the bucket.

These changes are all that’s needed to switch your FixMyStreet installation to using S3 for photos. Once you’re happy with the changes you’ve made to conf/general.yml, restart the FixMyStreet app server (example) and check the output for any errors.

Once the server has restarted, try making a new report and uploading a couple of photos. You should see new files appear in the S3 bucket (you can view them via the S3 web console) as soon as you’ve drag-dropped them into the new report (or update) form. Check that report photos appear on the site for newly-created reports too. NB photos for existing reports are likely already cached on disk by FixMyStreet, so do check a new report.

Final steps & tidying up

Because there was a small window of time between syncing existing photos to the S3 bucket and switching over the conf/general.yml config, there’s a chance that a report with photos was made whose photos weren’t synced to S3. Now that FixMyStreet is using S3, you may wish to re-run the aws s3 sync command from above in order to make sure any such photos have been transferred to S3.

The old directory containing photos on disk isn’t needed any more, so once you’re happy the S3 storage is working as expected, you can delete this.

Version 2.4.1

2018-10-01T00:00:00+00:00

Today we have released version 2.4.1 of FixMyStreet.

This release contains a wide variety of new features for use by reusers of our software, funded by NDI as part of their FixMyCommunity project; thanks very much to them. It also includes a number of smaller features and bugfixes.

FixMyStreet now comes with a Dockerfile and example Docker Compose setup for use by those tools, and we provide a hosted Vagrant box that can be used also. You can choose to store photos in Amazon S3 rather then the local filesystem, use an external memcache server,

We’ve added a script, cobrand-checks, that will list/diff template changes in core that might need applying to a cobrand, to help with upgrading, and added more checks to our Pull Requests on GitHub to spot included cobrands that need checking with each PR made.

We have sped up fetching full lists of bodies to help installs that have thousands of bodies, sped up the display of /_dev pages, fixed the display of pins on a body page when using Bing or TonerLite maps, and fixed an issue where the from_body field was lost on a user when edited by a non-superuser admin.

Full changelog

Version 2.4

2018-09-06T00:00:00+00:00

Today we have released version 2.4 of FixMyStreet.

This release contains a security fix to prevent the potential leak of a user account’s phone number, if you knew their email address; please upgrade your installation as soon as you can, or if that is not possible, please apply commit abcb1f86 to your installation in the meantime. We have also released version 2.3.5 which is identical to 2.3.4 except with this fix applied.

The front end has removed its input placeholders, for better accessibility; improved the report button in the navigation bar to be more context-aware and allow for easier reporting in the same location; will only show 6 months of reports on the around page by default, to hopefully deal with the issues of too many or too few reports being shown; and removes the need for a separate per-category ajax call by returning all category data up front.

The admin now trims spaces from search input (at last!), lets you edit a category group, and has a ‘send login email’ button on a user’s edit page, so you don’t have to ask them to do it themselves.

Bugfixes include an issue with the category filter if a category contained a comma, fixing duplicate category display on Your Account page, and making sure the Home link on mobile is always clickable. Staff workflows had various fixes as well, such as inspectors being able to unset priority, seeing all fixed issues with the map page filter, and fixing the pin dragging.

For developers, there are new hooks for e.g. custom search results, extra login conditions, and changing the details placeholder; a new /_dev/ URL (accessible to superusers) for previewing confirmation/submission pages; and the client can set bodis a repot must not be sent to (e.g. if asset selection means it has that knowledge).

Full changelog

Version 2.3.4

2018-06-07T00:00:00+00:00

A very quick release, only a day after 2.3.3, but version 2.3.4 fixes a bug introduced in 2.3.3 as part of the fix for selecting pins on mobile; whilst that was fixed, the fix then prevented pins being selected on non-/around pages, whoops.

Full changelog

Version 2.3.3

2018-06-06T00:00:00+00:00

Today we have released version 2.3.3 of FixMyStreet.

This fixes a few bugs found in the past week of the last release, including one potential data leak. It was possible to obtain from a crafted URL the textual content of an update, and the name of the user who left that update (only if they had made the update non-anonymously), even if that update was unconfirmed or hidden. No other details of the update or user were made available.

Other fixes include incorrect behaviour when selecting report pins on mobile, bad admin navigation links in multi-language installs, and an issue with the map display when an inspector clicked back from a report page on mobile.

Full changelog

Version 2.3.2

2018-05-31T00:00:00+00:00

Today we have released version 2.3.2 of FixMyStreet.

This is a minor release, but one with quite a number of bug fixes and other smaller front end and admin improvements.

The questionnaire process has been improved, with “Don’t know” now an option in the email, and recording the answer as soon as the link is clicked. Smaller front end fixes include increasing the size of “sub map links” (hide pins, permalink, etc), clicking the “Click map” instruction banner now begins a new report, and improved cursor/display of the new report pin.

Front end bug fixes include improving chart display in old IE versions, a CSS padding bug in the sidebar “drawer”, a race condition when making a new report quickly, some small RTL text display issues, and making sure the loading spinner is always shown when it should be.

We now store a user’s creation and last active times, and have provided scripts so you can use tis new information to anonymize inactive users or reports, email inactive users, and/or close reports to new updates.

At long last, the contents of a report’s extra field is now fully displayed in the admin.

Open311 has had a number of improvements and bug fixes – we can now fetch problems over Open311, send multiple photos via an Open311 extension, and have the ability to have automated attributes that can be filled in but not shown to the user (e.g. asset IDs).

For development, we’ve added an HTML email previewer at /dev/_email/ which our designers love, added some Cypress browser-based testing so that some of our JavaScript is also tested, and upgraded our Vagrantfile to use Ubuntu Xenial.

Full changelog

Version 2.3.1

2018-02-12T00:00:00+00:00

Today we have released version 2.3.1 of FixMyStreet.

This is a minor release, with a number of bug fixes but also a number of smaller front end and admin improvements.

The site should be clearer when things are loading now, be that on initial page load or when an asset layer is incoming. Our phone number library has been updated, so national phone numbers should be displayed in a nicer format as long as you’ve set the PHONE_COUNTRY variable.

FixMyStreet now asks for the current password (or sends an email) on password change, sets a minimum password length and checks against a list of commons passwords. Superusers can have optional two-factor authentication to protect their accounts.

We fixed an error in sending requires_inspection reports, issues with multiple select-multiples on a page, a questionnaire CSS snafu, showing deleted bodies in the dashboard list, alongside other minor bugfixes.

For staff users, ‘report as another user’ now allows phone number without email, and the inspector form shows any extra fields again.

Admins can now anonymize/hide all a user’s reports, log a user out, or remove a user’s account details. We’ve provided a script to expire old sessions in the database (plus this script can be run with --init to set up the database for the new “log a user out” feature).

Full changelog

Version 2.3

2017-12-18T00:00:00+00:00

Today we have released version 2.3 of FixMyStreet.

The major new feature in this release is adding confirmation by phone text instead of email, via Twilio, to allow optional verification of reports and updates, and logging in.

Alongside that, there have been numerous front end improvements and bugfixes. Front end improvements include paginating reports everywhere, making sure all maps can be expanded on mobile, the pin loading indicator no longer covering the whole map whilst pins are loading, and improved location disambiguation on small screens.

Some effort has gone into improving performance of various pages, especially the front page, reducing the amount of JavaScript loaded, replacing our image sprite with more SVG assets, and switching to modern prefetch. fixmystreet.com goes a step further by inlining critical CSS on the front page so no external requests are needed to show the start of the site. You can read more about this process on the mySociety blog.

We now support Open311 category groups, and if we fetch updates via Open311, we can use auto-response templates to fill in updates without their own description.

Lastly, the dashboard and statistics have been streamlined and improved, and are all now available under /dashboard. This now includes lookup by date range, ward, category or state, and the CSV export uses machine-readable dates and is accessible via token-based authentication.

Bugfixes

The number of updates on fixmystreet.com hit a million! So we made sure large numbers don’t overflow on the homepage. Also, multiple ‘Expand map’ links should no longer trouble your maps on mobile; with JavaScript off, the pins should no longer be double the size they’re meant to be; and also some bad interaction between the list filters and the back button has been fixed.

Staff users got a number of bugfixes to their interfaces, including making sure reports could always be removed from your shortlist (even if they’ve switched body), only creating one update when changing category, and making sure the text-only contacts output was indeed text, not HTML.

Full changelog

Version 2.2

2017-09-14T00:00:00+00:00

Today we have released version 2.2 of FixMyStreet, with some new features, some bugfixes, and other improvements.

At long last, body and category names can be translated in the admin interface, letting you easily run sites in multiple languages without having to perform various amounts of custom work;
Report states can now be edited (and translated) in the admin interface, if you wish to have different options for the specific workflow of your site. If you want to have a closed state of “object removed”, now you can;
Also through the admin interface you can now add extra fields to be shown in the reporting form, letting you ask custom questions for your installation without needing to edit any code or templates.

All the above improvements were funded by NDI as part of their FixMyCommunity project; thanks very much to them.

Staff users can now create reports on behalf of people anonymously, and have more filters on report list pages; new configuration options allow you to limit your site to logged-in users only, or prevent new users from using the site.

The email alerts page has had some work to improve its clarity, and some work has gone into improving the performance of various pages. Some deep diving into SQL queries led to some notable cases where things could be sped up, and we also found one place (listing nearby duplicate reports for staff users) where the database was being queried at a radius of 1000km, rather than 1km :)

One notable bugfix is if you logged in during the reporting process, or had a server side error, your photo thumbnails weren’t being redisplayed correctly (though the photo itself was stored fine) due to change in the image library we use – everything should now be okay.

Lastly, if you’re developing on the code, we have added a debug toolbar (similar to the Django Debug Toolbar, if you’re familiar with that) which lets you see which templates have been used, what SQL queries have been made, what external requests, and so on. This is automatically enabled when you run the development server.

Full changelog

Version 2.1.1

2017-08-03T00:00:00+00:00

Today we have released version 2.1.1 of FixMyStreet, a bugfix release with a few small improvements.

The map in HTML emails is now a link through to the report page, and we’ve fixed a bug causing the wrong pin size to be shown there. We’ve also fixed a bug that could cause a different language from the requested one to be shown on about pages.

Admin improvements include resending reports if changing the category has changed the send_method and displaying the reporter’s phone number on the inspector form. Text deleted during moderation is no longer replaced with [...].

Full changelog

Version 2.1

2017-07-18T00:00:00+00:00

Today we have released version 2.1 of FixMyStreet, with some new features and a variety of improvements for users and reusers.

New features include allowing users to hide their own name on reports/updates, and a new graph-based /reports page, which you can see in operation on our main UK site. We also now resize photos client-side before uploading, given the increase in very large images these days.

Admin improvements include adding an ‘inactive’ state to categories, so you can prevent a category being used for new reports but still have it available in filters, various improvements to the inspect form and the new report process for inspectors, and an easier way for inspectors to shortlist all reports in their view.

Geolocation is back on the alert page, lost sometime last year, and some random blank spaces are no longer clickable, hooray!

Development-wise, FixMyStreet now supports Debian stretch and perl 5.24, badly configured SMTP options should now be spotted, and we have refactored and simplified the CSS used for header/content/navigation. This may mean you can simplify overrides in your cobrand. If you wish to have a static front page rather than the normal report page, that’s now possible by creating an about/homepage.html template in your cobrand, and cobrands can now easily change the new report pin colour.

The test suite now runs each file in a transaction which means it can be run in parallel and sped up dramatically; on the other hand, on Travis we added code coverage which slowed it down again.

Two backwards incompatible changes that may require changes to your own templates if you have customised them:

The nav-wrapper-2 class has been removed. If you have a custom footer template, replace that class with ‘container’.
The /reports page now uses different generated data. If you have a custom reports/index.html template and wish to keep it, you may need to call update-all-reports with a --table argument to generate the old style data.

Full changelog

Version 2.0.4

2017-04-13T00:00:00+00:00

Today we have released version 2.0.4 of FixMyStreet, a bugfix release along with a couple of other improvements.

The bugfixes are updating our Facebook library so that Facebook login works again, stopping an error if you had a devolved body and its contact both lacking a send method, and an issue in the multi-select front end if you had characters such as brackets in your category name.

Strangely, while the backend has always allowed multiple email addresses for a contact, separated by commas, the admin interface hasn’t allowed them to be entered - this has now been corrected.

Lastly, body pages under /reports will now limit the reports shown to those within the visible map, which should make moving and zooming around much more intuitive.

Full changelog

Version 2.0.3

2017-03-31T00:00:00+00:00

Today we have released version 2.0.3 of FixMyStreet, a bugfix release along with some other improvements.

The map on a mobile report page can now be made full screen, and if you are using Google Maps you can supply a custom map styling. There’s also now a loading indicator whilst data is being fetched for the map.

Various missing translations have been added, for moderation, social login, and offline usage.

We’ve upgraded our email sending to deal with issues sending with SSL, dealt with IE11 caching report Ajax calls too aggressively, and with Safari 5/ Android 4 not showing our questionnaire answer buttons.

Performance improvements include moving admin-related JavaScript to its own file (so normal users don’t need to download it), and reducing the amount of disk stats the code performs.

Lastly, all the test suite can now run offline, and amusingly I found a bug in a test that only happened if the test was run c. 55 hours before daylight savings time began :)

Full changelog

Version 2.0.2

2017-02-03T00:00:00+00:00

Today we have released version 2.0.2 of FixMyStreet, a bugfix release along with some new admin improvements.

The main bugfixes are to mark two missing strings for translation, to make sure email is lowercase when signing in via /auth, and to make sure a language subdomain can be included in calls to base_url_for_report. There are also some CSS fixes to the homepage and an improved print layout for report list pages. Fixes to the admin interface include fixing filtering on the shortlist page, and fixing the ‘save with public update’ toggle.

For international users, there have been a few string renames, but we have maintained translations as the concept remains unchanged, so please only feel free to update if you consider it necessary:

‘unable to fix’ has been renamed to ‘no further action’
‘marked as a duplicate report’ has been changed to ‘closed as a duplicate report’

An offline fallback page has been added using appcache, which inspectors can use for offline updating of reports in their shortlist.

Admin improvements include allowing response templates to be associated with a state (so that template is used by default when the state is changed), allowing shortlist addition/removal from report lists, and shortlist reordering. Users with a new permission can see the body user who left a contribute_as_body report or update.

Full changelog

Version 2.0.1

2016-12-16T00:00:00+00:00

Today we have released version 2.0.1 of FixMyStreet, a bugfix release. The issues fixed are:

Chrome 55 changes the way it handles mouse/touch events, unifying them as pointer events, and OpenLayers, the JavaScript mapping library we use, needed patching to deal with this. Thanks to the Chrome team for helping debug this.
strftime output (e.g. used to display dates) was sometimes being double decoded, giving incorrect output in some languages.
If a category was filtered, it was not being properly carried through to the new report form.
The body/area inputs in the admin were being fixed to a too-small height.

Some bugfixes and improvements have also been made to the admin inspector report view, along with a couple of development improvements – the tests coping better if run on a network that intercepts NXDOMAINs, and a better way of showing the git version in the admin config.

Version 2.0 – JavaScript performance

2016-12-16T00:00:00+00:00

The JavaScript on FixMyStreet has gradually evolved over many years (we launched in 2007, remember!), and while we were working on other features in this area (such as HTML5 History) it was a good opportunity to tidy up the JavaScript, making it clearer and simpler. Below I’m going to go through most of the steps I took, not necessarily in the order I took them, which hopefully might prove useful to your own websites. And there are exciting pictures at the end, I promise!

It also let us add the Content-Security-Policy header to FixMyStreet, which is a method in browsers to prevent cross-site scripting (XSS), clickjacking and other code injection attacks by specifying the valid sources for script execution.

Separate scripting and styling

When you report an issue on FixMyStreet on mobile, the map /around page is full-screen to make reporting easier. This was being done in JavaScript using some jQuery css() calls when setting up (or ending) the mobile view. It was straightforward to move this CSS to a mobile-reporting-map class and have the JavaScript do no more than add or remove this class, making both parts clearer. (Later during this process, I also added an only-map class to prevent the map being scrolled until it is clicked, when the rest of the form can then be shown.)

Similarly, the JavaScript was requisitioning the desktop-width green banner to provide a different mobile banner; updating this to use a separate mobile banner made the code clearer and shorter.

Tidy up the JavaScript set up

The main setting up of our JavaScript was taking place, for historical reasons, in two files, both confusingly called fixmystreet.js. Whilst in the future we may want to split this up into separate files more tailored to the particular pages where the code is used (though our main use of JavaScript is our map page, which already has its JavaScript separate), for now it made most sense to combine these in one file, and tidy up all the setup functions into a list of feature-based functions each called in turn on page load.

We were bundling a copy of Modernizr (for media query detection) that contained html5shiv and yepnope. But html5shiv is only needed in old versions of Internet Explorer, and yepnope is only used on FixMyStreet’s front page (to try and preload the map page JavaScript mentioned above), so I could move html5shiv into an IE conditional comment, and included yepnope.js only on the front page, reducing Modernizr to only Modernizr itself.

Move vital JavaScript as early as possible

Now that I had a mobile-reporting-map class, I wanted this class activated as soon as possible as the page is loading, not only when the document had been parsed. There were also a couple of site-wide variables, page (the type of page, e.g. around or new), and cobrand (the branding of the site you’re on). Lastly, I wanted to be able to set a class on the document if JavaScript was activated, so that CSS using that class would be instantly active, preventing a flash of style change or content.

To achieve all this, I created a header.js file that performed the above three tasks, setting a class on <html>, setting two variables on our global fixmystreet variable, and if we’re on a small width (using Modernizr) and perhaps a map page, setting the appropriate classes. I then minimized and inlined this script in the header of each page, so that we don’t have to wait for any external script to load.

Move JavaScript to the end of the HTML, remove inline JavaScript

As recommended by Jake Archibald here, I moved all the JavaScript to the end of the HTML. To make this easier, and also to make adding a Content-Security-Policy header easier, I removed all the inline JavaScript from FixMyStreet (I didn’t think FixMyStreet had that much inline JavaScript, and it didn’t, but it still had more than I had remembered!). This was most commonly being used to set up some JavaScript variables with server-side data, so the easiest way to replace this was to place this data as attributes on HTML elements (preferably semantically related elements), and set up the JavaScript variables in an external script.

Minify JavaScript

I didn’t want to make this mandatory, as we have done with SCSS/CSS, but I wanted the option available, so I added an option to our templating code that means it will prefer an .auto.min.js file in preference to a .js file of the same basename. This lets you compile your JavaScript in a deploy process, for example, should you wish to. We do this with the standard Closure Compiler from Google; I haven’t yet been brave enough to try and check/get the JavaScript working with the advanced option of the Closure Compiler :)

Activate Content-Security-Policy

The Content-Security-Policy header lets you specify domains from which JavaScript will run, plus lets you choose to run inline JavaScript either en masse or only if you provide a specific unique nonce ID in the <script> tag that matches the same ID in the CSP header. That latter option is how we kept our inline header JavaScript running without having to externalise it again.

“nonce” was only added in the second version of the CSP spec, so you may note our header also specifies unsafe-inline. Any browser that supports version 2 will ignore this when it sees the nonce header, but it is needed in order for the inline script to still run in any browser only supporting version 1.

Conclusion

In Google Page Speed Insights, with manual minification of the main JS files, this moves the front page from 68/84 to 85/92ish (filmstrip from webpagetest.org, top is live site, bottom is my dev site):

These are requests from the US: most of the initial delay is in that initial download. Now here’s a report page going from 58/77 to 85/86ish (“ish” because e.g. live site will have analytics that my dev site doesn’t):

Due to page elements displaying more quickly than might previously have been expected, there are still a few things to tidy up. For example, when an element displays before a bit of JavaScript kicks in and makes it look slightly different… but hopefully nothing major.

Version 2.0 – testing improvements

2016-12-15T00:00:00+00:00

FixMyStreet has a large and hopefully comprehensive test suite that runs through all aspects of the codebase, checking everything is working. This makes it easier to change code and add new features, safe in the knowledge that any breakages will be quickly highlighted.

Speeding up the tests

Every time someone commits code to our GitHub repository, or opens a pull request, the tests are automatically run for us by Travis CI. We’re alerted to success or failure with little green ticks or red crosses on GitHub, and by notice in IRC.

The tests seemed to have slowed down considerably in recent times, but we couldn’t identify any changes at the FixMyStreet side which might have caused this.

However, there had recently been some spam scraping of Gaze, our web service that provides population density information to FixMyStreet (so that e.g. the maps can try and guess an appropriate zoom level, and so alerts can try and guess an appropriate radius), and rate limiting had been added to try and help combat it.

Dave spotted that this was being triggered by FixMyStreet test runs, leading to pauses as the suite waited for the rate limiting to ease. Thankfully, all Gaze calls were being routed through one function (that had been created in order to cope gracefully with a Gaze failure) and so it was a simple matter for this function to be stubbed out if being run as part of a test.

Before:

After:

There are many tests that still rely on the internet (e.g. for some MapIt lookups) and eventually it would be good to get to the point where they are all stubbed out and the test suite can run completely offline, probably even more quickly.

Multiple test running

When running the tests, the suite creates a test database (in PostgreSQL terms, it actually creates a temporary cluster) so that anything it does won’t affect your development database. Theoretically, this means you should be able to run the test suite multiple times simultaneously – perhaps it’s doing a full run, but you want to try and fix (and retest) the first error while it carries on. However, this was not working, and after some investigation it turned out that each run was creating (and overwriting) a test configuration .yml file, which meant the existing runs got all confused. Adding a process ID to the test configuration file meant that each run is independent and can successfully coexist with each other.

Keystroke saving

Lastly, you used to have to run the full suite with bin/run-tests t, but now if you run bin/run-tests, it will assume you meant t. A small thing, but it might save a few seconds over the years. ;-)

Version 2.0 – EXIF rotation in JavaScript

2016-12-14T00:00:00+00:00

FixMyStreet has had a nice multiple image uploader since version 1.8. This uses multiple input type=file fields, progressively enhanced to add drag’n’drop, image preview, and uploading in the background whilst you fill in the rest of the form.

In version 1.8.4, we patched the third party library we use, dropzone, to correctly orient photos in the image preview. We did this by including a cut-down version of exif-js to read in the EXIF orientation data, and then make sure we rotated the image as instructed in JavaScript before drawing the thumbnail preview. The rotation was accomplished by moving the image so its centre was over (0,0), rotating the appropriate amount, and then re-translating it back.

For this new version, we had a different bug to fix. If the user had uploaded a picture, submitted the form, and was shown the form again due to a server side error of some sort (some validation not caught by client-side validation, for example, or because you were logging in during the reporting process), the image for the preview was then being loaded from the server (where it had already been uploaded), not the client, and not displaying. We patched the exif-js library., Now, if it is given a URL rather than a data: string, it will go off and fetch the image so that it can read out the orientation data.

Version 2.0 – Improved forms

2016-12-13T00:00:00+00:00

The new release of FixMyStreet includes a number of improvements to various forms on the site. In this post, we will take a brief look at the notable changes.

Public reporting form

This form has been rejigged, in order to more obviously split out details that will be public (e.g. photos, details) from those that will not be published on the site (e.g. the user’s email address and phone number). The category selector has also moved to the top, and if the category chosen requires the display of extra questions or information (e.g. through Open311 attributes or a custom built asset layer), they will be shown immediately.

If a report is made in an area that is covered by more than one body, the category the user selects will normally dictate which one the report is sent to. Now, when the category is selected, we update the list of bodies given at the top of the report page, if we know that the report will be sent there.

Talking about custom built asset layers, this is a good place to show how the FixMyStreet codebase can be put to other uses, with a bit of development.

Angus Council in Scotland provide a WFS layer (that is, vector format geographic information) containing the locations of all their streetlights, which we display if the street lighting category is selected within Angus on FixMyStreet.

Importantly, it can display which lights Angus already knows are broken. If the user is able to identify precisely which street light is affected, they can click on it. But picking a street light isn’t mandatory: we don’t want to put people off who aren’t certain, or who are unable to select an individual light. You can see an example of what this looks like on the Angus cobrand of FixMyStreet.com below.

Admin report editing

Previously, the form for administrators to edit a report was functional, but certainly nothing more than that! Due to the work we’ve done on the new user system, more admin users may well be accessing this form in future, and so we’ve taken the opportunity to make it much tidier.

It now looks much more like the front end of the site. We’ve added a map that lets you move the location of the report, tidied up the various functions an admin can perform, and so on. If a category change means the report should have been sent to a different body, it will be re-sent.

Admin category editing

This was a historical oddity, in that the Add category form and the Edit category form were completely separate creatures, though both contained the same fields, and were used for basically the same purpose:

You can see how they looked different in the screenshots above. They now share an HTML template, which also makes it easier for us to update should it need changing in future.

Version 2.0 – HTML5 History

2016-12-09T00:00:00+00:00

We’ve mentioned before that FixMyStreet is built on a progressively-enhanced base, a concept explained neatly in a nice blog post from the UK government.

This means that e.g. the slippy map is, underneath it all, an old-style server-side image map that works out where you click; URLs are all shareable and pages are functional even if JavaScript is not available; the front page loads quickly and doesn’t need to preload an entire application.

None of this means that we don’t use or like JavaScript, however. This post is about adding JavaScript to FixMyStreet to provide a quicker experience to users looking at reports on our map.

When viewing a list of reports on FixMyStreet, you might want to look at a few reports one after the other, much as you can on a Google Maps or OpenStreetMap results screen.

So now, when you click a report in the list or a pin on the map, the report page is pulled in via JavaScript. This updates the page, pin and URL in situ, rather than loading a new page. Other pins remain visible, plus the page feels (and probably is) a bit quicker as the page header doesn’t reload, and it is easy to switch back to the list view.

This feature uses HTML5’s History API, ie. pushState and popState, to update the URL as the page changes. That means that when you share it, the page that loads will always be the one that the user intended it to be.

This improvement did not come without problems, however. Overriding the browser’s own behaviour when it comes to history and navigation means you have to think carefully, and I’m sure we’ll need to make further refinements to ensure that everything works as the user would expect.

There were small issues: for example, pushState stores the document title at the point when it’s called, for the ‘Back’ button list, meaning we had to make sure any title change happened after that. Some browsers have a popstate on page load, which can cause an issue if you assume it’s only fired due to history events.

There was the complication of needing to tell the difference between someone clicking back to the ‘initial state’ of the page, and an internal hashchange or other less crucial event – as well as using a replaceState on page load, we store the original URL and title for use in such a situation.

Then we forgot that the code would be running on /reports lists as well as /around which led to some confusion until we realised what was happening! And of course, you have to make sure everything JavaScript-wise is set up appropriately for content brought in via JavaScript.

We also used pushState in the new report process, to update the URL as you select the report’s location, and on list pages when you select one of the filters or sort. This has worked well, and is certainly much more preferable to the ‘hash-bang’ technique used by some sites in previous years (and still now), which is reliant on JavaScript functioning.

Version 2.0 – area highlighting

2016-12-08T00:00:00+00:00

FixMyStreet’s report pages for a particular body have always highlighted the area of the body covered, by fetching the KML shape from the associated MapIt and plotting it on a map. In version two we have made it look much nicer:

Until now we have coloured in the shape in a light orange, as pictured above left. This is fine for an overall view of a body, but if you want to zoom in to a particular area, it makes it quite hard to see the underlying map.

It would be preferable if every part of the map except the body’s area could be shaded, highlighting the correct area without losing any clarity from the part you’re interested in, as in the new image above right.

Polygon make-up

Polygons in KML and other similar formats are made up of multiple linear rings – the first ring is the polygon’s outline, and any subsequent entries are holes within the polygon. You can therefore ‘invert’ a polygon by having the outer ring be something surrounding the polygon, then having the polygon be a hole within that.

So the way we have implemented this in our new version is to fetch the area from MapIt, but before plotting it preface it with a giant rectangle covering the whole world. (Thankfully, any holes within the original shape are inverted too, though that is implementation dependent and it would have been a lot harder if they weren’t!).

There were a couple of issues along the way. MapIt may return either a polygon or multiple polygons, so we needed to deal with each differently (we turn multiple polygons into one polygon, which also thankfully Just Works). And OpenLayers has a hardcoded maximum pixel co-ordinate for SVG rendering, dating from 2007 and an issue in Firefox 2. We hope that any such issues have been fixed by now. We haven’t had any reports of crashes yet, anyway…

Version 2.0 – New user system

2016-12-07T00:00:00+00:00

Version 2 launches a new user system for admins, including more granular permissions and a variety of new features.

Logging in

Admins can now use the same login system as the main front end. We highly recommend running your site over HTTPS in order to keep credentials secure; LetsEncrypt can supply free 3-month certificates.

The main admin user type is the ‘superuser’; a user with this flag set (which can be done in the database, or by running the createsuperuser command) has full access to all areas of the admin, just like admins under the old system. By default, these are the only types of user allowed to log in to /admin. (It is possible to change this, e.g. in the UK, non-superuser admin users associated with a body can log in to /admin on their own cobrand.)

User permissions

Users associated with a body (called ‘body users’) can be given a variety of different feature-based permissions; the screenshot here shows the list of different options.

Categories: You can associate a user with a list of categories, which e.g. pre-selects those categories when the user visits the All Reports page.
Response priorities: This allows you to set a list of different priorities for a body, or again for a particular category in a body, letting you note different priorities for different reports.
Response templates: You can create and edit templates associated with your body, or with a particular category in that body, and then when leaving an update you can select one of these templates to allow easy updating of reports.
You can give a user access to a front-end report ‘inspect’ view, which lets a user edit a report’s category, state, or location. If the category change moves the report to a different body, it will be re-sent. Alternatively, a user can be given only category edit or priority edit permission. Here is a screenshot of the top of an inspect form view:

Create as another/body

This permission gives a user the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user. Below is a short animation showing this in action on the Oxfordshire cobrand of FixMyStreet.com:

Shortlists

A user with the shortlist permission gains a shortlist button on each report; clicking this adds the report to your own personal shortlist of reports, which you can view in a section of Your Account. This may be useful for an ‘inspector’ type of admin user, who wishes to compile the day’s list of reports before going out and investigating them. You can also see if a report is on someone else’s shortlist, and take it off them if you need to.

Version 2.0 – Multi-select dropdown jQuery plugin

2016-12-06T00:00:00+00:00

There are several types of ‘list page’ on FixMyStreet: for example, when you view all reports from a specific body, or when you log into your account to see reports you have made.

For quite a while now, users have been able to filter these by state (eg ‘fixed’ or ‘in progress’) or category (eg ‘pothole’, ‘streetlight’, etc), but a recent suggestion from Oxfordshire County Council prompted us to look again at this functionality and improve it.

So now it’s simple to filter by multiple states or categories: want to see all unfixed potholes? All streetlights or flytipping reports that are fixed? No problem.

HTML’s <select multiple> is not the nicest tool in the box, especially when it comes to actually selecting multiple options. It usually involves holding down a key, but that key is different depending upon your operating system.

So I looked at existing JavaScript plugins and finding nothing suitable created a very simple jQuery plugin which is available at https://github.com/mysociety/jquery-multi-select.

By default, it converts a multiple select into a dropdown with checkboxes:

The options can be wrapped within a container to stop the dropdown overlapping other content:

With options in the constructor, you can specify text for when all/no options are selected, and specify groups of options as presets that will be listed at the top of the dropdown:

On FixMyStreet, we have used this for both the state and category filters on list pages:

Let us know if you find the plugin useful!

Version 2.0 – email validation

2016-12-05T00:00:00+00:00

When someone enters an email address on FixMyStreet, we try and do some simple validation before sending a confirmation email, to catch typos and the like. We do the following:

Check the address is correctly formed (e.g. that it contains an @ sign and a domain, and doesn’t include irregular characters, etc);
Check that the email address’s domain is a valid top-level domain (TLD) (it’s not going to be delivered anywhere if it’s not); and
Check that the email address’s domain has a valid DNS entry (so we’ll be able to try and deliver an email there).

The middle point means using a Perl module, Net::Domain::TLD, that contains a list of all valid TLDs. This list has grown since we last updated it, and we found ourselves incorrectly rejecting an email address that was perfectly valid (a @domain.cymru address). It was at least easy to update our version of Net::Domain::TLD to ensure that our list of top-level domains is current.

Changing your email address

The confirmation email is itself another form of validation, and the only one that can actually confirm that the user entering the email address has access to that email inbox.

Version 2 of FixMyStreet now allows users to change their own email address, which threw up a complication – we can’t update the user’s table to their new email address until it’s been confirmed.

So we have to store their old email address with the token that is sent to the new one. When the new address has been confirmed, we deal with it differently depending on whether or not that address already exists in the database. If it does, it’s a matter of merging the two accounts.

Version 2.0 – HTML emails

2016-11-29T00:00:00+00:00

FixMyStreet sends a number of automated emails, both to users (confirmation emails, follow-up questionnaires) and to bodies (the reports themselves). Previously these were plain text, but we have now introduced HTML emails, with all the design possibilities that this implies.

One reason for this is to make the site’s communications look more professional; another is an attempt to minimise one of the most time-consuming admin tasks - dealing with users who reply to our automated emails.

Designing better emails

In Version 2.0 we wanted to afford FixMyStreet emails the same design and usability attention that we normally spend on the FixMyStreet website.

Incorporating feedback from our users, our support team, and our council partners, we quickly identified a few key places that HTML (graphical) emails could improve the FixMyStreet experience for everyone:

Attaching a map and photo of each problem to our “Has your problem been fixed?” questionnaire email, to help people remember the report we’re asking about.
Styling calls to action as attractive, clickable buttons, to make the emails easier to quickly scan and comprehend, and to reduce user support queries.
Using photos and a clearer typographical hierarchy to make the area alert emails easier to scan, especially when there are multiple new problems in your chosen area.
Letting our council partners and international reusers maintain their brand image across the website and emails, by easily customising the logo and colour scheme of emails sent to their users.

The best way to see how we’ve improved FixMyStreet’s emails is to give FixMyStreet a try, and receive the emails yourself! Try reporting a new problem near you, or subscribing to email alerts about new reports in your area.

If you’d like to know how we implemented some of our more technical changes, read on…

Attaching the static map image

One thing that we wanted to include in the email was a map showing the location of the report (be that the one you’ve just made, or the one you’re receiving an alert or questionnaire for). The map that you see on the website is made up of many 256x256px tiles stitched together in HTML/CSS to appear as one smooth map, with separate pin images superimposed in the correct location; for the email, we needed just one image covering whatever portion of the map was necessary, plus the pin.

This is the point at which FixMyStreet’s progressively-enhanced base came in very useful. If JavaScript does not work for whatever reason, the site has always displayed an alternative: a small map made up of four tiles in a square, with the pin located appropriately on top. It also makes sure that the pin location (the point at the bottom middle of the pin) is contained within the central half of the tiled map (the red dashed area in our image here), so that there’s no chance the pin overlaps the edge of the map.

This was perfect for the image to be used in an HTML email. The new static map function fetches the same data used by the front end, requests the tile data for those four tiles, stitches them together in one 512x512px image, composites the pin on top in the correct location, and then cuts off the bottom 128 and top 64 pixels – as the pin’s central location means those will always be pin-less. The image is then shrunk to 310px in width, providing output that looks like this, appearing in the top right hand corner of the email:

Inline images

The new HTML email - whether they’re reports, alerts, confirmations or questionnaires - normally have upwards of three images: the static map image, an image of the report (or repots for alert emails), and a site logo. We wanted to include all these images within the email itself, rather than use remote images, as due to spam many people have remote images switched off.

We also wanted to keep things as simple as possible when including images in the email templates. Inline images in HTML emails use an image source of cid:UNIQUE_ID within the HTML (or CSS), and then give a particular attachment of the email the same UNIQUE_ID in its Content-ID header. Lastly, we may potentially also have normal attachments.

We settled on an inline_image function in the template, which can be provided with either a path to an image file (for the static logo), or a function that returns image data and a content type (for the static map or report image). As the email template is processed, each call to this function generates a unique ID for the image and stores the information in a list to be added after the plain text/HTML email parts.

There are a variety of ways to attach images to an email. Content types are used to identify what is contained by each part of the email. As well as the various types of image, and text/plain and text/html for the text and HTML parts, there are various containers: multipart/alternative as a container for parts containing the same information in different formats, multipart/related as a container for parts that are linked in some way, and multipart/mixed as a general container.

The setups we decided on were as follows:

If the HTML email has no inline images or other attachments, then we create a multipart/alternative email, containing the two textual parts.
If the HTML email has inline images but no other attachments (the most common case), we create a multipart/related email, its first part being the multipart/alternative as above, the subsequent parts the inline images.

An alternative here would be to create a multipart/alternative email, with its first part being text, and its second part being multipart/related containing the HTML part and the inline images. This would mean that an email client that didn’t support HTML email might only see the textual part and not any of the images. If your inline images were not ‘important’ to the email (e.g. an email footer signature) then this might be a way to go; we thought that the map and image should be visible to all users if possible.
If the HTML email has attachments but no inline images, we create a multipart/mixed email, its first part being the multipart/alternative as above, the subsequent parts the attachments.
If the HTML email has both inline images and other attachments, then we create a multipart/mixed email. Its first part is a multipart/related email (that again contains multipart/alternative and the inline images), and then its other attachments form the later parts of the mixed email. If you imagine the parts as envelopes with brackets, it would look like this:
```
multipart/mixed (
  multipart/related (
    multipart/alternative (
      text/plain
      text/html
    )
    image/jpeg
    image/png
  )
  application/pdf
)
```

As part of this work, I discovered that the oldest open GitHub issue of the Rails framework was related to this topic – if you used Rails to create an email containing both inline images and normal attachments, the normal attachments were not accessible to most email clients (that support HTML email) as they had embedded all the normal attachments inside a multipart/related part. I have submitted a pull request to fix this structure, which I hope will be accepted in some way.

Templating

This work was also a good opportunity to move some text generation out of some code into the templates (necessary because the text being generated now needed some HTML around each entry), for all the alert emails.

Testing

Finally, this post wouldn’t be complete without a few words about email testing.

Any of you who have built HTML emails in the past will agree that they are like taking a time machine back to web development in the mid 1990s. Email clients like Outlook, Gmail, and iOS Mail have dramatically different capabilities and ways of rendering the same email code.

One way of avoiding cross-client complications is to keep your HTML layout as simple as possible; maybe add an <img> tag in for your site logo, some <strong> or <em> tags for emphasis, and call it a day.

But our plans for FixMyStreet required much more complex email layouts than this. It was a hard requirement that the details of the report (the map, the name, the photo) in our questionnaire emails, were given equal priority to the introductory text and the call to action buttons. The requirements led us to a two-column layout, which, inevitably, required two or three layers of nested table cells. (Remember, we’re in our 1990s time machine!)

To help us test these layout changes in all the required email clients, we used Litmus, which is like Browserstack but for emails. You send a single email to Litmus, and it renders that email in dozens of different email clients, grabbing screenshots of each one, and presenting them to you in a handy dashboard. Building HTML emails this way still requires a good deal of trial and error, and obscure CSS knowledge, but at least with Litmus, the process of iterating on your design is made as short and fast as possible. It’s an expensive service, but well worth the cost for the peace of mind that your new layout works in even the most uncooperative email clients.

With the help of our Litmus checklists, we made a bunch of unintuitive discoveries, including:

Using <th> rather than <td> elements, so that the Android 4.x mail client can give them block styling in the small screen media query.
Defining our font settings on every table cell, rather than simply inheriting font-family from the body, so that sans-serif fonts are used in Outlook, rather than Times New Roman.
Using a three-column wrapper table to create a 620px centred content area that also shrinks down on narrow screens. (Outlook doesn’t like max-width, so this is the simplest alternative.)
Enforcing a sensible (500px) min-width for the main content area, on clients that don’t support media queries.
Using giant borders on <a> elements, to make them into Outlook-friendly buttons without resorting to less accessible alternatives like images.
Aligning images with the deprecated align attribute, rather than CSS floats.
Applying the email background colour to a wrapper element inside the body, and thus leaving the body to keep its default white background, so that replies sent from Outlook (which inserts the reply message inside the body of the original message) will have a white background.

Version 2.0

2016-11-15T00:00:00+00:00

We’re proud to announce the release of version 2.0 of FixMyStreet.

This version contains a wide array of new features, including HTML email and multiple state/category filtering, a new admin user system with graduated permissions, and various bugfixes and development improvements.

Over the next few days and weeks, we will be writing a series of blog posts, going into details about a number of the changes, which I hope will be of interest. But before then, do set up the code or update your installation, and ask us questions :)

New front end features

HTML email: There is now the option for all emails sent by FixMyStreet to be HTML formatted where previously they were plain text only. This includes confirmation and questionnaire emails to the user, and report emails to the public body. These emails include any image added to the report, plus a small static map of the problem’s location.
State/category filtering and sorting of list pages: When viewing a list of reports, you can now filter and sort them in pretty much any way you choose, including sorting by most- or least-recently updated, newest or oldest, or most commented. You can also select multiple categories or states (e.g. “fixed”).
Pretty area highlighting on body pages: The highlighting of areas on a body page has been inverted, so that the unimportant parts of the map are shaded and you can interact more easily with reports on the page.
Users can now update their own email address This was a frequent request from users and we’re glad to report that they can now do it themselves on their account page.
Performance improvements: When looking at reports from a list page, the other report pins stay visible so that it is easier to switch between them. The report itself is being pulled in behind the scenes, meaning the whole page does not need to reload. The map no longer extends underneath the sidebar and header, which makes things easier, and a scroll wheel can now zoom the map in and out.
Making privacy options clearer: The reporting form has been separated into public and private sections, to make it clearer which parts of what you provide will be made visible on the site.

Showing the relevant recipient: If you live in an area where there’s more than one body, the category you pick normally dictates which body we send your report to. Now, when you select the category we update the name of the body given at the top of the report page, if we know that the report will be sent there.

New admin user system

Admin users can now use the same log-in right across the site - whether they’re making a report like a standard user, or logging in to make edits and moderate the site.

In the past, the distinction between admin and other users was black and white. As an admin user, you had access to every part of the site, but users can now be given individual permissions for various layers of access. These include:

Proxy users This layer grants the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user;
Report editors Giving the power to edit a report’s category, state, or location. If the admin user changes the category, and that change means that a different body is now responsible for the report, it will be re-sent;
List makers, who can compile their own shortlist of reports they wish to go and inspect. This may be useful for a contractor or team who wishes to compile the day’s tasks;
Quick responders These users have access to response templates, allowing them to edit and publish templated updates;
Prioritisers These users may set different priorities on reports;
Trusted users A simple reputation system, which e.g. potentially lets reports from trusted users be actioned more quickly.

The admin report edit form has also been greatly improved, including a map to update a report’s location (and re-sending the report if the body changes), and much tidier layout.

Bugfixes and development changes

Bugfixes include updating the top-level domain (TLD) list for email validation, hiding authorities which don’t exist any more on the /reports page, and fixing the previously-broken photo preview display after form submission. We have dropped support for Internet Explorer 6.

If you’re a re-user of the codebase, there are a number of changes that will hopefully help you out. Geocoder results won’t be cached in development, the UPLOAD_DIR and GEO_CACHE variables are now relative to the project root, we’ve added a server-side MapIt proxy, and you can add your own fields to the contact form. Open311 support has been tidied up and improved. If you run the tests, you can now run the test suite multiple times simultaneously, and the tests have been sped up quite a bit.

Plus quite a few other things; as always, see the full list of changes over on GitHub.

Version 1.8.4

2016-07-06T00:00:00+00:00

We’ve released version 1.8.4 of FixMyStreet, along with versions 1.7.2 and 1.6.3.

These releases include a security fix, closing two cross-site scripting (XSS) vulnerabilities. Please update your installation as soon as possible.

Version 1.8.4 also contains other minor bug fixes, such as correctly orientating preview images, wrapping emails better for differing screen sizes, a bug with filter redirect when JavaScript wasn’t available, a race condition when starting a new report, and a couple of display bugs in IE7.

See the full list of changes over on GitHub.

Version 1.8.3

2016-06-03T00:00:00+00:00

We’ve released version 1.8.3 of FixMyStreet.

For developers, the main improvement in this version is the great speed up to CSS compilation by moving from sass to libsass.

We’ve fixed some map issues, getting the Google Maps layer working again and dealing with tap sensitivity on some devices.

The admin interface gets a bit of love, adding common search boxes to the index page, allowing change of email to an existing address, and speeding up photo removal.

Upgrading notes

If your templates aren’t kept in the main repository, there are a couple of things to be aware of with this release:

If you’ve used the cf class, you’ll need to rename it to clearfix.
If you have customised any of the email templates that use <?=...?> style variables (e.g. alert emails, questionnaire, submit), you’ll need to rewrite them to use the standard [%...%] variables.

See the full list of changes over on GitHub.

Version 1.8.2

2016-05-03T00:00:00+00:00

We’ve released version 1.8.2 of FixMyStreet, along with versions 1.7.1 and 1.6.2.

These releases include an important security fix, whereby a malicious user could craft an image upload to the server that allowed them to run external commands as the user running the site. Please update your installation as soon as possible.

Version 1.8.2 also contains other improvements and additions to existing features:

Twitter social login, alongside the existing Facebook login;
PNG and GIF image upload support;
Some development improvements, including the final merging of base and fixmystreet templates, storing any Open311 error in the database, and tidying up some unused cobrands;
A few bug fixes, such as showing the right body user form value for fixed reports (thanks Jon Kristensen).

See the full list of changes over on GitHub.

Version 1.8.1

2016-03-23T00:00:00+00:00

We’ve released version 1.8.1 of FixMyStreet, which fixes some bugs and makes some improvements to the 1.8 release.

Now there is multiple photo support, the display of those photos on a report page is now a bit nicer, and if there’s an error message on photo upload it should now always be visible. Auto-scrolling of the sidebar when you hover over a pin has been removed as it was confusing, and the site now remembers a user’s last anonymous state. Fixes have been made for running on more recent versions of Perl, one of which was causing the geocoder to break.

For developers, an easier way of adding cobrand-specific custom reporting fields has been added.

There are a couple of other minor changes; see the full list of changes over on GitHub.

Version 1.8

2016-03-02T00:00:00+00:00

We’ve released version 1.8 of FixMyStreet.

The two main new features in this release are Facebook login – provide a Facebook app ID and secret in your configuration and it will smoothly fit into the creation and login flow – and multiple photo support, along with a modern interface for previewing and uploading photos whilst you create your report.

Smaller improvements include highlighting the pin when you hover over an item in the sidebar, and vice-versa; fixing some small display bugs such as how updates were displayed in Your Reports and preventing a chevron being stretched in Firefox; improving the look of the 404 page, and making sure you can see an update if you got to it via an in-page link.

Memory performance has been improved, meaning cron jobs can take up to half as much memory, and this release also fixes a number of small bugs, including an embarrassing swap of latitude and longitude in the Google geocoder, making sure you’re signed up for updates if you used the app and were logged in, and better internationalisation and display of numbers.

For developers, we’ve added a generic static route handler, so that adding new static HTML pages to your installation involves only creating a new file in your template directory and nothing more; improved bounce handling; and fixed the cobrand restriction handling on Your Reports and list pages.

Plus quite a few other things; as always, see the full list of changes over on GitHub.

Version 1.7

2015-10-23T00:00:00+00:00

We’ve released version 1.7 of FixMyStreet.

This version adds some new features. First off is that the FixMyStreet design is now bi-directional, providing an easy switch to flip the design to either left-to-right or right-to-left. This was done with the kind support of the National Democratic Institute.

We have added state and category filters to the list pages, letting users view only e.g. open reports in the potholes category, or all reports in the graffiti category. Various design improvements have been made, including the showing of the report on a questionnaire page and the email confirmation pages, and we’ve added a nicer default OpenGraph image.

Database performance has been improved in a number of areas, and the accessibility of the map pages has been improved.

This release also fixes a number of small bugs, including translating report states in the admin index, dealing with DMARC email issues, and fixes for Google Maps API users.

For developers, we’ve made it easier to run gettext-extract if you’re performing your own translations, removed some confusing warnings, finally removed the final few hardcoded “FixMyStreet” strings so it’s easy to rename your site, streamlined the navigation menu and list item CSS using a BEM style naming scheme so it is easy to change and override, and lastly fixed a long standing issue where errors were not always logged correctly.

Plus quite a few other things; as always, see the full list of changes over on GitHub.

Version 1.6.1

2015-07-31T00:00:00+00:00

We’ve released version 1.6.1 of FixMyStreet.

This release fixes a bug introduced in the previous release when setting multiple areas for a body in the administration interface.

It also includes improvements to the All Reports page, adding a fixed header and tooltips, and stops the sidebar running over the footer on alerts pages. The admin gets a variety of minor improvements, with better internal linking and a mark as sent button. Plus a Danish translation :)

As ever, see the full list of changes over on GitHub.

Version 1.6

2015-07-10T00:00:00+00:00

We’ve released version 1.6 of FixMyStreet (previously numbered 1.5.5).

This release includes important security fixes:

A vulnerability in login email sending that could allow an account to be hijacked by a third party;
Alterations to token logging in and timeout behaviour;
A dependency update to fix an issue with Unicode characters in passwords.

More details on those items below. Other items in this release include a Chinese translation, a bug fix with shrunken update photos, and some front end improvements, such as a ‘hamburger’ menu icon and an easier Report button on mobile, and resized map pins based on zoom level.

See the full list of changes over on GitHub.

Security fixes

Login email account hijacking: Due to the way parameters were passed into the token table in the database, it was possible for someone to request a login email for one email address, but have the login email sent to different address. This would allow a third party to log in as someone else, letting them make reports or updates as that person.

The code has been rewritten so all user parameter passing goes through central functions that return only one parameter even if the user has passed multiple parameters. More details of this class of vulnerability.

Email authentication tokens: Problem confirmation tokens had to be used within a month; this now applies to all confirmation tokens, and email sign in tokens are valid for a day. Using those tokens after confirmation will redirect correctly, but no longer log you in; links in alert emails will no longer log you in.

Unicode characters in passwords: The package our code uses to encode database columns, DBIx::Class::EncodedColumn, could have issues with Unicode characters provided to it. This was fixed by upgrading the version we use.

Version 1.5.4

2015-03-18T00:00:00+00:00

Female Yellow Warbler by Alan Vernon on Flickr

We’ve released version 1.5.4 of FixMyStreet.

This includes a couple of new map layers, Bing Maps and Stamen’s toner-lite, and nicer confirmation pages for after you’ve made a report or update, along with other smaller improvements and bug fixes. See the full list of changes over on GitHub.

For developers, it includes a few small improvements, to do with Mac installation, making some things optional, and including a new configuration variable for if you’re running behind an SSL proxy. We’ve also added some test URLs so that you can view confirmation pages without having to leave a new report or update, e.g. see it in action on fixmystreet.com: https://www.fixmystreet.com/P/_test_.

As always, do ask on the mailing list if you’d like more information on any of the above, or submit an issue or pull request on GitHub.

Version 1.5

2014-11-20T00:00:00+00:00

Resting Tahr by Jason Armstrong on Flickr

We’ve released version 1.5 of FixMyStreet. This version fully supports the new Long Term Support (LTS) version of Ubuntu, Trusty Tahr 14.04 (the code did already run fine on Ubuntu Trusty if you set it up manually, but now the install script will work and a few other bits have been tidied).

This release comes with a few improvements to the admin interface, including pagination of search results, validation of new categories, and some display enhancements.

We’ve moved the map sidebar to be flush with the window edge, which we think is simpler and easier on the eye, and we’ve continued making the template structure easier to change and override.

We’ve also fixed some bugs, such as map submission not working with JavaScript disabled or unavailable. As another example, we had a report of the Android browser crashing when showing a map page, which we tracked down to the slightly transparent map navigation controls – crashing wasn’t worth this, so now on mobile they’re fully opaque.

From Transifex we’ve added four new languages (as well as updating the existing ones): Albanian, Bulgarian, Hebrew, and Ukranian.

See the full changes over on GitHub.

As always, do ask on the mailing list if you’d like more information on any of the above, or submit an issue or pull request on GitHub.

Version 1.4.2

2014-07-15T00:00:00+00:00

Bicycle Rack by Theen Moy on Flickr

We’ve released version 1.4.2, a maintenance release, but also with a couple of new features. The main reason for this release was to fix a couple of issues with the installation script, which are hopefully now resolved. It also upgrades our bundled copy of cpanm (to cope better when an external module website is down), tidies up the template structure, copes with browser autofill on the /auth sign in page, and adds links from the All Reports page to a body’s open or fixed reports.

The main new feature is the addition of SMTP username/password and SSL/TLS options for your setup. You can read about the new options on the configuration settings page

See the full changes over on GitHub at https://github.com/mysociety/fixmystreet/releases.

As always, do ask on the mailing list if you’d like more information on any of the above, or submit an issue or pull request on GitHub.

Version 1.4

2014-05-23T00:00:00+00:00

Yellow by RozSheffield on Flickr

Say hello to version 1.4 of FixMyStreet. As usual, the install script and AMI have both been updated to this version.

Hopefully this release will run more smoothly on EC2 micro instances, with some cron rejigging to alleviate memory problems.

There have been a few minor user-facing improvements, such as automatically selecting the reporting category if there’s only one choice, and removing the indenting of emails; some bug fixes, including changes for the new version of Debian, a problem with language setting in email alerts, and removal of cached photos; and a number of improvements for people reusing the code, including a no-op send method, having reports on staging sites be sent to the reporter, adding an external URL field to bodies, and making it easier to change the pin icons.

See the full changes over on GitHub at https://github.com/mysociety/fixmystreet/releases.

Thanks to Andy Lulham, Chris Mytton, Dave Arter, Dave Whiteland, Gerald, Hakim Cassimally, Ian Chard, Jon Kristensen, Jonas Oberg, Kindrat, Matthew Somerville, Rikard, Steven Day, and Struan Donald for contributing to this release.

As always, do ask on the mailing list if you’d like more information on any of the above, or submit an issue or pull request on GitHub.

Version 1.3

2013-11-25T00:00:00+00:00

Version 1.3 of FixMyStreet is now out :-) The install script and AMI are both updated to this version.

I realise I haven’t posted here with each point release during version 1.2, sorry. You can see the changes in each release here on GitHub at https://github.com/mysociety/fixmystreet/releases and below is a list of all the main things that have changed since version 1.2 (* means new since the last version, 1.2.6, if you were keeping track):

OpenLayers upgraded to 2.13.1, giving e.g. animated zooming
A fully functional Google Maps layer via OpenLayers
* If you only specify one cobrand in the configuration file, the site will always use it, rather than only if your hostname also matches. This is probably what you would expect to happen!
* A contact can be given multiple email addresses
* A body can be marked as deleted, and then it will not be used by the front end at all
The admin interface has had a lot of inline documentation, hints and notices added, along with a page showing the site’s current configuration
* The admin interface has some feature additions from coding volunteers, such as a date picker on the stats page thanks to Andrew Black, and searching by external ID thanks to Andy Lulham
We’ve added an example Vagrantfile and improved the install scripts
* The test suite should now run regardless of the contents of your configuration file
Translation improvements - some better wording of strings, some missed or UK-only URLs/translations (thanks Jonas and Rikard), and a fix for the long-standing issue where multiline strings were not being translated (hooray)
* Bug fixes, most notably sometimes when changing report state in the admin interface, and an issue with the bottom navbar in Chrome (which we’ve reported to the Chromium project)

As always, do ask on the mailing list if you’d like more information on any of the above, or submit an issue or pull request on GitHub.

Version 1.2

2013-05-03T00:00:00+00:00

Today we’re releasing version 1.2 of the FixMyStreet platform. The AMI has been updated and the install script will automatically now install this version.

The main items in this release are things prompted by requests on our mailing list :-)

Postfix is now installed as part of the install script, or in the AMI. This means email should work out of the box. For anyone already installed, you can run the commands in the GitHub ticket.
A new configuration option MAPIT_ID_WHITELIST has been added, to restrict usage to the IDs specified, if given. This means Claudio, who emailed last week, could have [ 239540 ] as his MAPIT_ID_WHITELIST, and then reports could only be made within the Marche region of Italy. We already use this new option ourselves on https://www.zueriwieneu.ch/ where before it was hard-coded in the code.
Other things include being able to zoom in further on OSM maps, and HTML pages are now being gzipped.

Lastly, as you can see this site has had a redesign to make it more friendly, and we’ve added some more documentation about e.g. updating an AMI instance to a newer version. If there’s anything unclear, please do ask on the mailing list or submit an issue or pull request on GitHub.

Version 1.1 - Bodies

2013-02-22T00:00:00+00:00

Today we’re releasing version 1.1 of the FixMyStreet platform. The AMI has been updated and the install script will automatically now install this version.

The main change since version 1.0 is the addition of bodies. Historically, FixMyStreet has assumed that the administrative areas that are returned from MapIt are the same thing as the bodies to which the reports will be sent. This has led over time to a number of workarounds when this hasn’t been the case, either in manual code changes in FixMyStreet or by adding new types to a MapIt install, and dealing with it in that way.

We have updated the code so that FixMyStreet holds its own records of bodies to which reports can be sent, and the MapIt area – or areas – that they cover. This is conceptually much clearer, as well as making it much easier to have a body covering multiple areas, an area with multiple bodies, or a combination.

Smaller functional changes in this release include admin improvements (it now looks like the front end site, and has add a number of other additions), and a couple of new configuration variables, DO_NOT_REPLY_EMAIL and SEND_REPORTS_ON_STAGING, to make debugging a little easier, along with a --debug option to send-reports. Also, we found on the mailing list a couple of times that people ran into trouble because their MapIt had debug turned on, and FixMyStreet didn’t cope well with the debug output MapIt included in its responses. This has now been fixed.

Many others of the commits in the past few months have been for various installations of the codebase, from the forthcoming FixMyBarangay in the Philippines to local UK council installs such as Oxfordshire or Bromley. These have in many cases led to small improvements and bugfixes to the central codebase, which can then be used by any reusers of the code.

Lastly, all the strings in the JavaScript are now translatable, along with a few other strings that had previously been missed; do let us know if you find any other strings that can’t be translated and we’ll look into it.

Easy Installation

2012-10-02T00:00:00+00:00

Four months ago, someone raised a ticket on FixMyStreet’s GitHub account, asking for alternative ways of setting up an installation. We certainly agreed this was a good idea, as we’re well aware that there are various different parts to FixMyStreet that might require quite a bit of knowledge in setting up.

We’re now pleased to announce that we have created an AMI (Amazon Machine Image) containing an already set-up default installation of FixMyStreet. You can use this to create a running server on an Amazon EC2 instance. If you haven’t used Amazon Web Services before, then you can get a Micro instance free for a year.

If you have your own server, then we have separately released the install script that is used to create the AMI, which can be run on any clean Debian or Ubuntu server to set everything up for you, from the PostgreSQL database to nginx.

If you prefer to do things manually, and already know how to set up your database and web server, our manual documentation is still available.

An AMI and install script is also available for MapIt – see our MapIt documentation for more details. This should make it very straightforward to get something set up for testing and development.

Do let us know how you get on.

Improving Configuration

2012-08-17T00:00:00+00:00

Now that a default install is a bit more straightforward to set up, our thoughts turn to improving the customistation of that default install. Currently, apart from the options already present in the main configuration file, that involves knowing a bit of Perl, in order to create a Cobrand .pm file containing the various customistations. So to reduce that dependency, we’ve moved a number of these options into the main configuration file, so that hopefully a standard customisation might not need a Cobrand .pm file at all.

These changes range from simple text strings that are now in templates, through to specifying what areas from MapIt you are interested in, or what languages the site is available in. The general.yml-example file contains information on each option, and we’ve updated our customisation documentation as well.

Also, thanks to some testing of a current installation by Anders for FiksGataMi, we’ve made more incremental improvements to the installation, including fixing a couple of tests that shouldn’t run unless your configuration is set up in a particular way, making sure inherited cobrands use the best templates, and including the Catalyst::Devel module so running the development server is easier.

Default Workings

2012-07-27T00:00:00+00:00

In the past few weeks, a number of improvements have been made to the FixMyStreet default set up, so that installation should provide you with a working setup more easily and quickly, before you get on and make all the necessary customisations you will want to for the service you are setting up.

Firstly, we’ve tidied and consolidated the documentation on to this site, putting everything you need in one place. We are using GitHub pages, which means that the documentation is bundled along with the repository when checking out, which might be useful. The installation guide now includes help for installing on Mac OS X, and various other tweaks and improvements.

Next, the codebase now automatically defaults to OpenStreetMap maps and geocoding – these are available, with more or less data, everywhere in the world, so you should be able to test your installation and see working maps.

Whilst an installation of MapIt may be necessary for your FixMyStreet to work as you want – mapping locations picked to the right authority might need some private boundary data, for example – the code will now default to work as if everywhere is one administrative area.

The code for sending reports has been refactored and modularised, enabling proprietary options to be more easily added alongside the standard email, Open311, and so on.

We have removed any UK specific code from the default cobrand, moving it to a UK cobrand (which is then in turn inherited by the various council cobrands we have made in the UK). This should mean that you find you have less to override, and more things should work by default.

Lastly, the default cobrand now uses the new style that you can see on https://www.fixmystreet.com. By default, we have picked a pretty yet garish colour scheme, in order to remind you that you almost certainly want to change the colours being used for your own installation :)

An Introduction

2012-02-06T00:00:00+00:00

mySociety wrote FixMyStreet in 2007 in order to make it easy for people in the UK to report problems in their area. Since then over 200,000 problems have been reported to UK councils using the site. As with all our sites we want people everywhere to be able to benefit so we were delighted to work with the Norwegian Unix User’s Group (NUUG) last year to set up a FixMyStreet in Norway.

We want to build on that and help more people around the world install their own copy of FixMyStreet. In the months since FiksGataMi was launched we’ve made a number of changes to FixMyStreet to make it easier to install and customise for other countries. We’ve rewritten much of the code to use the Catalyst web framework which has helped make it easier to customise and we’re now improving the documentation and ironing out the installation procedure.

However, we know that easy to install code is only part of the process. Building a community around the software is far more valuable in encouraging people to use and improve it. This site and the new mailing list are our first steps along that road.

So, if you’ve ever wanted to have a FixMyStreet for your town, state or country then join the mailing list and we’ll help you along the way.

FixMyStreet Platform

Version 6.0

Admin improvements

Development improvement examples

Upgrading

Version 5.0

Mobile use and the Progressive Web App (PWA)

Accessibility

Admin improvements

Development improvements

Upgrading

Version 4.0

Multi-page reporting form

Photo redaction

Development Docker environment

Other improvements

Bugfixes

Upgrading

Version 3.1

Front end improvements

Admin improvements

Bugfixes

Upgrading

Version 3.0.1

Admin improvements

Bugfixes

Development improvements

Upgrading

Version 3.0

Front end improvements

Security

Admin improvements

Bugfixes

Development improvements

Open311 improvements

Upgrading

Version 2.6

Upgrading

Version 2.5

Upgrading

Version 2.4.2

Migrating to S3 photo storage

Prerequisites

Creating the bucket

Using the S3 web console

Using the aws CLI tool

Migrating photos

Configuring FixMyStreet to use S3

Final steps & tidying up

Version 2.4.1

Version 2.4

Version 2.3.4

Version 2.3.3

Version 2.3.2

Version 2.3.1

Version 2.3

Bugfixes

Version 2.2

Version 2.1.1

Version 2.1

Version 2.0.4

Version 2.0.3

Version 2.0.2

Version 2.0.1

Version 2.0 – JavaScript performance

Separate scripting and styling

Tidy up the JavaScript set up

Move vital JavaScript as early as possible

Move JavaScript to the end of the HTML, remove inline JavaScript

Minify JavaScript

Activate Content-Security-Policy

Conclusion

Version 2.0 – testing improvements

Speeding up the tests

Multiple test running

Keystroke saving

Version 2.0 – EXIF rotation in JavaScript

Version 2.0 – Improved forms

Public reporting form

Admin report editing

Using the `aws` CLI tool