Blog

  • Version 3.1 16 November 2020

    Palouse Wheat ready for harvest

    I hope you are all okay with the ongoing situations around the world. We are happy to release version 3.1, with some new features and bugfixes.

    Front end improvements

    A random assortment of improvements to the front end, including aerial maps for Bing and OSM maps, lazy image loading on list items, the ability to check passwords against Have I Been Pwned, and Open Location Codes support in the search box.

    Admin improvements

    We’ve added a full text index so searching reports in the admin should be much quicker. You can disable updates or reopening on a per-category basis, as well as enable anonymous reporting. CSV generation is now done asynchronously, with progress shown, in case it takes a long time. Staff users can use HTML in updates, and response templates can also include HTML. The site now records whether a report was made on desktop or mobile, and includes any extra fields in email submissions to bodies. Staff status can be removed from users in bulk, and inspectors can update the asset on a report via the inspector form.

    Bugfixes

    As always, a wide variety of issues covered here. We added PostgreSQL 12 compatibility to the schema update script, fixed an issue where the CSV export returned multiple entries per row, added a fix for photo orientation in modern browsers, and improved the moderation diff display.

    Upgrading

    A full list of changes can be seen in the changelog as usual.

  • Version 3.0.1 6 May 2020

    First of all, I hope you are all safe wherever you may be. We held a couple of FixMyStreet user groups recently for body users of FixMyStreet in the UK, and someone from a large body said that their FixMyStreet installation was proving really useful in these times, as they were making far fewer internal reports due to the UK lockdown.

    Admist all this, we continue to develop the software, and today are releasing version 3.0.1, a bugfix release with a couple of new features.

    Admin improvements

    You can now provide an automatic initial update on reports made in a particular category to a particular body, perhaps to provide information on timings or similar. Make sure the body has a user to associate comments with, and then add a response template in the Open state with auto-response checked, restricted to a list of categories if you wish. After that, the template will be used to provide an initial update on new reports made.

    We have also added “staff-only” categories, which are categories that staff users can see but normal users cannot. One council is using this to provide emergency categories that their contact centre staff can use to make reports after manual triage.

    The dashboard export and report search should now be quicker, after some investigation that area. We’ve also moved the overall stats off the index page to the stats page, so the index page loads more quickly.

    Bugfixes

    Thanks to those of you letting us know about bugs or problems. Those we have fixed include an incorrect To header on emails about inactive accounts, a couple of issues with the front page recent reports list showing different results depending on whether the cache was used or not, and a double escape in the Google Maps URL.

    Others we have fixed include maintaining the category group on pin move with same category in multiple groups, and fixing sorting by most commented on the /around map view.

    Development improvements

    As well as the cron scripts, this release now includes a dæmon that you can use to send reports and updates. Using the dæmon will mean reports and updates are sent almost immediately after they are confirmed, but will require a bit more setup, as you’ll need to set it up as a dæmon running all the time in your system. We’ve provided an example config file for systemd to hopefully help with that. If you do run the dæmon, be sure to remove the lines of your crontab that send reports and updates :)

    Alongside that, send-reports no longer prints out failures in verbose mode, there is a separate send-reports-failure-summary script to do that. Also the fetch-comments and fetch-reports scripts have been consolidated into one fetch script, which can also now parallelize fetching.

    Upgrading

    A full list of changes can be seen in the changelog as usual.

  • Version 3.0 4 March 2020

    Follow the Yellow Line

    It has been quite a while since the last release, apologies, but today we are happy to be releasing version 3.0 of the FixMyStreet Platform, which has a number of improvements.

    Front end improvements

    • FixMyStreet can now be installed as a progressive web app. This means we’ve added a web manifest (and an admin UI for managing this) and a basic service worker that shows a page if you’re offline, and continues the functionality of allowing staff users to store and view their shortlisted reports offline.

      If you serve your site over HTTPS, you will be able to add the website to your homescreen (browsers may prompt the user) and have it work like an app. This provides us with a solid base on which to continue improving this in future, including hopefully adding functionality such as offline report drafting through the web site.

    • Various improvements have been made to the site on mobile – the “try again” process is clearer, duplicate suggestions show an inline map, the photo upload message is better, and map filters can now be accessed.

    • Category groups are now used wherever a category list is shown – admin pages, map filters, and so on; and you can pass a filter_category or filter_group parameter to the front page or around page to pre-select that option, which makes it easier to deep link to FixMyStreet from a page or form on another site.

    • Screenshot of map geolocation blue dot

      If you use geolocation, your location will now be displayed on the map, as shown in the screenshot.

    • As asked for a few times on our mailing list, we now use a report’s image as its OpenGraph image on an individual report page when shared.

    • We’ve added XSL to our RSS feeds which means browsers no longer display them as raw XML but as a nice simple web page that explains its purpose. Before and after shots below:

    RSS feed before changes, raw XML RSS feed after changes, looks much nicer

    Security

    All template variables are now automatically escaped by default, to help protect against any future XSS vulnerabilities. We also rotate the user’s session ID after successful login, and scrub the admin description fields.

    If any of your own templates outputs a variable that contains HTML that you wish to continue to allow to display as HTML, you will need to alter your template to escape the variable with the safe filter, e.g. [% some_html | safe %].

    Admin improvements

    • FixMyStreet now has a new roles system, allowing you to create groups of permissions and apply those roles to users.

      Category edit form screenshot
    • The category edit form has been drastically improved; category names can now be edited, categories can be listed under more than one group, and categories or particular extra questions can disable the reporting form (for e.g. emergency “please call” categories or questions).

    • Two-factor authentication can be used by any staff member, and you can choose to optionally enforce it for all staff.

    • The admin report edit page now stores moderation history, like the front end, and you can now view a user’s admin log history.

    • Heatmap web page

      We’ve added a heatmap dashboard for staff users, which can show hotspots. To enable this, you will need to add heatmap: { yourcobrand: 1 } to your COBRAND_FEATURES configuration.

    • There’s a new “staff only” contact state, for categories that can only be used by staff.

    • Staff users can report as other users even if they only have a name, and can sign other people up to alerts.

    Bugfixes

    Of course there have been a lot of bugfixes as well. One I remember is when going back to the initial state with popstate, a change event was being triggered on every single option of the filter selects. This led to a lot of change events running on the category/status multi-selects which then needlessly repeated the same activities over and over. This locked up the browser for seconds in locations with many categories. Below is a chart showing browser performance before and after:

    Performance chart before bugfix, 12 seconds locked browser Performance chart after bugfix, 0.2 seconds

    Development improvements

    We’ve upgraded the underlying framework and other packages, added a banner to the staging website/emails to make it obvious when you’re in development, added configuration for admin resending, a Content-Security-Policy header, and stopped hard coding the site name in the database fixture.

    Open311 improvements

    • It is now possible for an external Open311 service to POST updates on a report to FixMyStreet, rather than have FixMyStreet poll an external service for updates.

    • Email templates can include a placeholder to include the description fetched from the Open311 server in the update.

    • Private reports are supported, in that an Open311 server can mark a category as private which will then automatically mark all reports sent and received in that category as private.

    • Meta questions added in the admin can be marked as protected so that they won’t be overridden by data fetched from an Open311 server. This is useful for e.g. an “emergency” question that the Open311 server does not care about.

    Upgrading

    As mentioned above, but it is worth repeating, if any of your own templates outputs a variable that contains HTML that you wish to continue to allow to display as HTML, you will need to alter your template to escape the variable with the safe filter, e.g. [% some_html | safe %].

    A full list of changes can be seen in the changelog as usual.

  • Version 2.6 29 April 2019

    Image © Ben Waddington

    Today we have released version 2.6 of FixMyStreet.

    This release fixes a cross-site scripting security issue where someone could create a report through the site with a specially constructed query parameter, and then viewing that report on the admin report edit page would allow the report creator to run their own JavaScript. We have also released version 2.5.1 which is identical to 2.5 including this fix.

    There’s a new, optional, feature to auto-suggest similar nearby problems while reporting, to discourage duplicate reports; and the map state is now updated in the URL to make sharing links easier. A bit more work has been done on moderation, spotting conflicts and showing moderation history to staff on report pages, as well as in the admin.

    Mostly this release is bugfixes, please see the changelog for full details.

    Upgrading

    The admin body and user sections have been refactored – if you have custom templates/code, you may need to update links to those.

    If you wish the default for the showname checkbox to be checked, you can add sub default_show_name { 1 } to your cobrand file.

  • Version 2.5 21 December 2018

    White christmas!!! =D

    Today we have released version 2.5 of FixMyStreet; happy solstice!

    This release contains a number of front end improvements, especially to the sign in flow. This was the result of a lot of work to simplify the whole process, and we’ve written more about it on the mySociety blog.

    Other front end improvements range from a more prominent display of “state” on report page, making sure you don’t need two taps on a report list entry on a touchscreen, and clearer relocation options while you’re reporting a problem.

    Moderation has had some work done – it can now potentially edit a report’s category, should you wish it to do so; it stores more of a report’s original data upon moderation, and also now stores all moderation history, making that visible in the report/update admin interface.

    Bugfixes include one reported by FixaMinGata (back in early 2017!) about better map zoom behaviour when clicking on pins and going back; not escaping HTML entities in report titles pulled in by Ajax; and show missing reopening/fixed questionnaire responses when tey lacked their own separate update. We also fixed some issues with our Open311 contact group handling, and improved the validation of fetched reports timestamps.

    We now strip Exif data on uploaded photos, and have added a new config variable, SYMLINK_FULL_SIZE, that can be used to symlink your full size photos out of the photo cache rather than copy them, if your static files are being served by your web server.

    Full changelog

    Upgrading

    Due to the sign-in and banner changes, this release changes a number of the base templates. If you have overridden any related templates in your cobrand and not fed those changes upstream, you may need to make adjustments to match; please use our bin/cobrand-checks script to help make comparisons between your changes.

  • Version 2.4.2 6 November 2018

    Water & Oil

    Today we have released version 2.4.2 of FixMyStreet.

    This is mainly a bugfix release, but does contain a couple of new features - the admin lets you see and manage a user’s alerts, and the dashboard has update CSV export.

    Bugfixes include an issue when setting up a new Vagrant-based installation, updating the areas column when a report is moved by a staff user, giving superusers access to update staff dropdown, and an issue with subcategories when visiting /report/new directly.

    We have renamed a few SCSS variables for clarity; the old names still work, but if you wish to update your cobrand, then change $map_nav_bg to $nav_background_colour, $nav_fg to $nav_colour, and $nav_fg_hover to $nav_hover_background_colour.

    Full changelog

  • Migrating to S3 photo storage 8 October 2018

    The recent release of FixMyStreet v2.4.1 brought with it the ability to store photos for reports & updates in a bucket on Amazon’s S3 instead of locally on disk.

    Getting a new site configured to store newly uploaded photos in S3 is explained in the configuration pages, and is a matter of tweaking a few keys in your general.yml file. But what about if you have an existing FixMyStreet installation that you would like to migrate to storing all photos in S3? This post outlines a process for achieving just that - without any downtime.

    Prerequisites

    • A running FixMyStreet site with at least one report or update with a photo
    • An account on Amazon AWS, and access/secret keys for a role that can create/manage S3 buckets
    • The AWS CLI tool installed on your FixMyStreet server, and configured with access credentials

    Creating the bucket

    First off, we need to create a bucket to hold all the photos. If you’ve already done this, skip ahead to the Migrating photos section.

    We’ll show two ways to create a bucket: via the S3 web console and using the aws command line tool.

    Using the S3 web console

    Log in to S3 web console, and click the ‘Create Bucket’ button

    Enter a name for the bucket, e.g., my-fixmystreet-photos, and choose the region closest to your FixMyStreet server (more about buckets and regions):

    Your newly created bucket should then appear in the list of buckets. Make sure public access isn’t allowed (photos are served to users via your FixMyStreet server, not directly from S3):

    If everything looks OK, skip to the Migrating photos section.

    Using the aws CLI tool

    You can also create buckets from the command line using the aws tool.

    If you’ve not already done so, now is the time to install and configure the aws CLI tool.

    To create a bucket named my-fixmystreet-photos in the eu-west-1 region, run the following:

    $ aws s3 mb s3://my-fixmystreet-photos --region eu-west-1
    

    The following success message should be shown:

    make_bucket: my-fixmystreet-photos
    

    The bucket’s region can be inspected:

    $ aws s3api get-bucket-location --bucket my-fixmystreet-photos
    {
        "LocationConstraint": "eu-west-1"
    }
    

    The bucket should be created as private by default, but you may wish to check its access policy on the S3 web console.

    Migrating photos

    Now that the bucket has been created, the next step is to fill it with all the photos from your existing FixMyStreet installation. For this we’ll need the aws CLI tool, so make sure you’ve installed and configured it.

    Existing photos are stored on disk in the directory specified by the UPLOAD_DIR (or PHOTO_STORAGE_OPTIONS.UPLOAD_DIR) configuration key in conf/general.yml. For this example, let’s assume that’s /home/fixmystreet/photos.

    Running the following (with the path and bucket name altered to suit your configuration) will make copy-pasting the subsequent commands a little easier:

    $ export UPLOAD_DIR=/home/fixmystreet/photos
    $ export S3_BUCKET=my-fixmystreet-photos
    

    We’ll use aws s3 sync to copy all photos to the new S3 bucket. To get a preview of the operations it will perform without actually copying anything, run:

    $ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET --dryrun
    

    You should see output similar to the following, one line for each file:

    (dryrun) upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
    (dryrun) upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
    (dryrun) upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
    [...]
    

    If the output looks correct, re-run without the --dryrun flag to actually copy everything to the S3 bucket:

    $ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET
    upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
    upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
    upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
    [...]
    

    All your photos are now in S3, so the next step is to tell FixMyStreet about the new bucket. (See the note in the final section for reports made between now and when the FixMyStreet config has been updated to use S3.)

    Configuring FixMyStreet to use S3

    We need to make a few changes to conf/general.yml to configure FixMyStreet to use the new S3 bucket for photo storage.

    Your conf/general.yml probably has a section similar to the following:

    PHOTO_STORAGE_BACKEND: 'FileSystem'
    PHOTO_STORAGE_OPTIONS:
        UPLOAD_DIR: '../photos'
    

    Or, if you set the site up before v2.4.1:

    UPLOAD_DIR: '../photos'
    

    Either way, you need to replace that section with the following:

    PHOTO_STORAGE_BACKEND: 'S3'
    PHOTO_STORAGE_OPTIONS:
        BUCKET: 'my-fixmystreet-photos'
        ACCESS_KEY: 'AKIA12345'
        SECRET_KEY: '1234/1234'
    

    Make sure you set BUCKET, ACCESS_KEY and SECRET_KEY to the same values as when you configured the aws CLI tool and created the bucket.

    These changes are all that’s needed to switch your FixMyStreet installation to using S3 for photos. Once you’re happy with the changes you’ve made to conf/general.yml, restart the FixMyStreet app server (example) and check the output for any errors.

    Once the server has restarted, try making a new report and uploading a couple of photos. You should see new files appear in the S3 bucket (you can view them via the S3 web console) as soon as you’ve drag-dropped them into the new report (or update) form. Check that report photos appear on the site for newly-created reports too. NB photos for existing reports are likely already cached on disk by FixMyStreet, so do check a new report.

    Final steps & tidying up

    Because there was a small window of time between syncing existing photos to the S3 bucket and switching over the conf/general.yml config, there’s a chance that a report with photos was made whose photos weren’t synced to S3. Now that FixMyStreet is using S3, you may wish to re-run the aws s3 sync command from above in order to make sure any such photos have been transferred to S3.

    The old directory containing photos on disk isn’t needed any more, so once you’re happy the S3 storage is working as expected, you can delete this.

  • Version 2.4.1 1 October 2018

    Docks

    Today we have released version 2.4.1 of FixMyStreet.

    This release contains a wide variety of new features for use by reusers of our software, funded by NDI as part of their FixMyCommunity project; thanks very much to them. It also includes a number of smaller features and bugfixes.

    FixMyStreet now comes with a Dockerfile and example Docker Compose setup for use by those tools, and we provide a hosted Vagrant box that can be used also. You can choose to store photos in Amazon S3 rather then the local filesystem, use an external memcache server,

    We’ve added a script, cobrand-checks, that will list/diff template changes in core that might need applying to a cobrand, to help with upgrading, and added more checks to our Pull Requests on GitHub to spot included cobrands that need checking with each PR made.

    We have sped up fetching full lists of bodies to help installs that have thousands of bodies, sped up the display of /_dev pages, fixed the display of pins on a body page when using Bing or TonerLite maps, and fixed an issue where the from_body field was lost on a user when edited by a non-superuser admin.

    Full changelog

  • Version 2.4 6 September 2018

    Fattoria Palazetta, Field of Sunflowers, Cecina, Tuscany, Italy

    Today we have released version 2.4 of FixMyStreet.

    This release contains a security fix to prevent the potential leak of a user account’s phone number, if you knew their email address; please upgrade your installation as soon as you can, or if that is not possible, please apply commit abcb1f86 to your installation in the meantime. We have also released version 2.3.5 which is identical to 2.3.4 except with this fix applied.

    The front end has removed its input placeholders, for better accessibility; improved the report button in the navigation bar to be more context-aware and allow for easier reporting in the same location; will only show 6 months of reports on the around page by default, to hopefully deal with the issues of too many or too few reports being shown; and removes the need for a separate per-category ajax call by returning all category data up front.

    The admin now trims spaces from search input (at last!), lets you edit a category group, and has a ‘send login email’ button on a user’s edit page, so you don’t have to ask them to do it themselves.

    Bugfixes include an issue with the category filter if a category contained a comma, fixing duplicate category display on Your Account page, and making sure the Home link on mobile is always clickable. Staff workflows had various fixes as well, such as inspectors being able to unset priority, seeing all fixed issues with the map page filter, and fixing the pin dragging.

    For developers, there are new hooks for e.g. custom search results, extra login conditions, and changing the details placeholder; a new /_dev/ URL (accessible to superusers) for previewing confirmation/submission pages; and the client can set bodis a repot must not be sent to (e.g. if asset selection means it has that knowledge).

    Full changelog

  • Version 2.3.4 7 June 2018

    A very quick release, only a day after 2.3.3, but version 2.3.4 fixes a bug introduced in 2.3.3 as part of the fix for selecting pins on mobile; whilst that was fixed, the fix then prevented pins being selected on non-/around pages, whoops.

    Full changelog