Blog

  • Version 2.4.2 6 November 2018

    Water & Oil

    Today we have released version 2.4.2 of FixMyStreet.

    This is mainly a bugfix release, but does contain a couple of new features - the admin lets you see and manage a user’s alerts, and the dashboard has update CSV export.

    Bugfixes include an issue when setting up a new Vagrant-based installation, updating the areas column when a report is moved by a staff user, giving superusers access to update staff dropdown, and an issue with subcategories when visiting /report/new directly.

    We have renamed a few SCSS variables for clarity; the old names still work, but if you wish to update your cobrand, then change $map_nav_bg to $nav_background_colour, $nav_fg to $nav_colour, and $nav_fg_hover to $nav_hover_background_colour.

    Full changelog

  • Migrating to S3 photo storage 8 October 2018

    The recent release of FixMyStreet v2.4.1 brought with it the ability to store photos for reports & updates in a bucket on Amazon’s S3 instead of locally on disk.

    Getting a new site configured to store newly uploaded photos in S3 is explained in the configuration pages, and is a matter of tweaking a few keys in your general.yml file. But what about if you have an existing FixMyStreet installation that you would like to migrate to storing all photos in S3? This post outlines a process for achieving just that - without any downtime.

    Prerequisites

    • A running FixMyStreet site with at least one report or update with a photo
    • An account on Amazon AWS, and access/secret keys for a role that can create/manage S3 buckets
    • The AWS CLI tool installed on your FixMyStreet server, and configured with access credentials

    Creating the bucket

    First off, we need to create a bucket to hold all the photos. If you’ve already done this, skip ahead to the Migrating photos section.

    We’ll show two ways to create a bucket: via the S3 web console and using the aws command line tool.

    Using the S3 web console

    Log in to S3 web console, and click the ‘Create Bucket’ button

    Enter a name for the bucket, e.g., my-fixmystreet-photos, and choose the region closest to your FixMyStreet server (more about buckets and regions):

    Your newly created bucket should then appear in the list of buckets. Make sure public access isn’t allowed (photos are served to users via your FixMyStreet server, not directly from S3):

    If everything looks OK, skip to the Migrating photos section.

    Using the aws CLI tool

    You can also create buckets from the command line using the aws tool.

    If you’ve not already done so, now is the time to install and configure the aws CLI tool.

    To create a bucket named my-fixmystreet-photos in the eu-west-1 region, run the following:

    $ aws s3 mb s3://my-fixmystreet-photos --region eu-west-1
    

    The following success message should be shown:

    make_bucket: my-fixmystreet-photos
    

    The bucket’s region can be inspected:

    $ aws s3api get-bucket-location --bucket my-fixmystreet-photos
    {
        "LocationConstraint": "eu-west-1"
    }
    

    The bucket should be created as private by default, but you may wish to check its access policy on the S3 web console.

    Migrating photos

    Now that the bucket has been created, the next step is to fill it with all the photos from your existing FixMyStreet installation. For this we’ll need the aws CLI tool, so make sure you’ve installed and configured it.

    Existing photos are stored on disk in the directory specified by the UPLOAD_DIR (or PHOTO_STORAGE_OPTIONS.UPLOAD_DIR) configuration key in conf/general.yml. For this example, let’s assume that’s /home/fixmystreet/photos.

    Running the following (with the path and bucket name altered to suit your configuration) will make copy-pasting the subsequent commands a little easier:

    $ export UPLOAD_DIR=/home/fixmystreet/photos
    $ export S3_BUCKET=my-fixmystreet-photos
    

    We’ll use aws s3 sync to copy all photos to the new S3 bucket. To get a preview of the operations it will perform without actually copying anything, run:

    $ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET --dryrun
    

    You should see output similar to the following, one line for each file:

    (dryrun) upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
    (dryrun) upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
    (dryrun) upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
    [...]
    

    If the output looks correct, re-run without the --dryrun flag to actually copy everything to the S3 bucket:

    $ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET
    upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
    upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
    upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
    [...]
    

    All your photos are now in S3, so the next step is to tell FixMyStreet about the new bucket. (See the note in the final section for reports made between now and when the FixMyStreet config has been updated to use S3.)

    Configuring FixMyStreet to use S3

    We need to make a few changes to conf/general.yml to configure FixMyStreet to use the new S3 bucket for photo storage.

    Your conf/general.yml probably has a section similar to the following:

    PHOTO_STORAGE_BACKEND: 'FileSystem'
    PHOTO_STORAGE_OPTIONS:
        UPLOAD_DIR: '../photos'
    

    Or, if you set the site up before v2.4.1:

    UPLOAD_DIR: '../photos'
    

    Either way, you need to replace that section with the following:

    PHOTO_STORAGE_BACKEND: 'S3'
    PHOTO_STORAGE_OPTIONS:
        BUCKET: 'my-fixmystreet-photos'
        ACCESS_KEY: 'AKIA12345'
        SECRET_KEY: '1234/1234'
    

    Make sure you set BUCKET, ACCESS_KEY and SECRET_KEY to the same values as when you configured the aws CLI tool and created the bucket.

    These changes are all that’s needed to switch your FixMyStreet installation to using S3 for photos. Once you’re happy with the changes you’ve made to conf/general.yml, restart the FixMyStreet app server (example) and check the output for any errors.

    Once the server has restarted, try making a new report and uploading a couple of photos. You should see new files appear in the S3 bucket (you can view them via the S3 web console) as soon as you’ve drag-dropped them into the new report (or update) form. Check that report photos appear on the site for newly-created reports too. NB photos for existing reports are likely already cached on disk by FixMyStreet, so do check a new report.

    Final steps & tidying up

    Because there was a small window of time between syncing existing photos to the S3 bucket and switching over the conf/general.yml config, there’s a chance that a report with photos was made whose photos weren’t synced to S3. Now that FixMyStreet is using S3, you may wish to re-run the aws s3 sync command from above in order to make sure any such photos have been transferred to S3.

    The old directory containing photos on disk isn’t needed any more, so once you’re happy the S3 storage is working as expected, you can delete this.

  • Version 2.4.1 1 October 2018

    Docks

    Today we have released version 2.4.1 of FixMyStreet.

    This release contains a wide variety of new features for use by reusers of our software, funded by NDI as part of their FixMyCommunity project; thanks very much to them. It also includes a number of smaller features and bugfixes.

    FixMyStreet now comes with a Dockerfile and example Docker Compose setup for use by those tools, and we provide a hosted Vagrant box that can be used also. You can choose to store photos in Amazon S3 rather then the local filesystem, use an external memcache server,

    We’ve added a script, cobrand-checks, that will list/diff template changes in core that might need applying to a cobrand, to help with upgrading, and added more checks to our Pull Requests on GitHub to spot included cobrands that need checking with each PR made.

    We have sped up fetching full lists of bodies to help installs that have thousands of bodies, sped up the display of /_dev pages, fixed the display of pins on a body page when using Bing or TonerLite maps, and fixed an issue where the from_body field was lost on a user when edited by a non-superuser admin.

    Full changelog

  • Version 2.4 6 September 2018

    Fattoria Palazetta, Field of Sunflowers, Cecina, Tuscany, Italy

    Today we have released version 2.4 of FixMyStreet.

    This release contains a security fix to prevent the potential leak of a user account’s phone number, if you knew their email address; please upgrade your installation as soon as you can, or if that is not possible, please apply commit abcb1f86 to your installation in the meantime. We have also released version 2.3.5 which is identical to 2.3.4 except with this fix applied.

    The front end has removed its input placeholders, for better accessibility; improved the report button in the navigation bar to be more context-aware and allow for easier reporting in the same location; will only show 6 months of reports on the around page by default, to hopefully deal with the issues of too many or too few reports being shown; and removes the need for a separate per-category ajax call by returning all category data up front.

    The admin now trims spaces from search input (at last!), lets you edit a category group, and has a ‘send login email’ button on a user’s edit page, so you don’t have to ask them to do it themselves.

    Bugfixes include an issue with the category filter if a category contained a comma, fixing duplicate category display on Your Account page, and making sure the Home link on mobile is always clickable. Staff workflows had various fixes as well, such as inspectors being able to unset priority, seeing all fixed issues with the map page filter, and fixing the pin dragging.

    For developers, there are new hooks for e.g. custom search results, extra login conditions, and changing the details placeholder; a new /_dev/ URL (accessible to superusers) for previewing confirmation/submission pages; and the client can set bodis a repot must not be sent to (e.g. if asset selection means it has that knowledge).

    Full changelog

  • Version 2.3.4 7 June 2018

    A very quick release, only a day after 2.3.3, but version 2.3.4 fixes a bug introduced in 2.3.3 as part of the fix for selecting pins on mobile; whilst that was fixed, the fix then prevented pins being selected on non-/around pages, whoops.

    Full changelog

  • Version 2.3.3 6 June 2018

    Today we have released version 2.3.3 of FixMyStreet.

    This fixes a few bugs found in the past week of the last release, including one potential data leak. It was possible to obtain from a crafted URL the textual content of an update, and the name of the user who left that update (only if they had made the update non-anonymously), even if that update was unconfirmed or hidden. No other details of the update or user were made available.

    Other fixes include incorrect behaviour when selecting report pins on mobile, bad admin navigation links in multi-language installs, and an issue with the map display when an inspector clicked back from a report page on mobile.

    Full changelog

  • Version 2.3.2 31 May 2018

    London Loop Section 3

    Today we have released version 2.3.2 of FixMyStreet.

    This is a minor release, but one with quite a number of bug fixes and other smaller front end and admin improvements.

    The questionnaire process has been improved, with “Don’t know” now an option in the email, and recording the answer as soon as the link is clicked. Smaller front end fixes include increasing the size of “sub map links” (hide pins, permalink, etc), clicking the “Click map” instruction banner now begins a new report, and improved cursor/display of the new report pin.

    Front end bug fixes include improving chart display in old IE versions, a CSS padding bug in the sidebar “drawer”, a race condition when making a new report quickly, some small RTL text display issues, and making sure the loading spinner is always shown when it should be.

    We now store a user’s creation and last active times, and have provided scripts so you can use tis new information to anonymize inactive users or reports, email inactive users, and/or close reports to new updates.

    At long last, the contents of a report’s extra field is now fully displayed in the admin.

    Open311 has had a number of improvements and bug fixes – we can now fetch problems over Open311, send multiple photos via an Open311 extension, and have the ability to have automated attributes that can be filled in but not shown to the user (e.g. asset IDs).

    For development, we’ve added an HTML email previewer at /dev/_email/ which our designers love, added some Cypress browser-based testing so that some of our JavaScript is also tested, and upgraded our Vagrantfile to use Ubuntu Xenial.

    Full changelog

  • Version 2.3.1 12 February 2018

    Keep Behind This Line

    Today we have released version 2.3.1 of FixMyStreet.

    This is a minor release, with a number of bug fixes but also a number of smaller front end and admin improvements.

    The site should be clearer when things are loading now, be that on initial page load or when an asset layer is incoming. Our phone number library has been updated, so national phone numbers should be displayed in a nicer format as long as you’ve set the PHONE_COUNTRY variable.

    FixMyStreet now asks for the current password (or sends an email) on password change, sets a minimum password length and checks against a list of commons passwords. Superusers can have optional two-factor authentication to protect their accounts.

    We fixed an error in sending requires_inspection reports, issues with multiple select-multiples on a page, a questionnaire CSS snafu, showing deleted bodies in the dashboard list, alongside other minor bugfixes.

    For staff users, ‘report as another user’ now allows phone number without email, and the inspector form shows any extra fields again.

    Admins can now anonymize/hide all a user’s reports, log a user out, or remove a user’s account details. We’ve provided a script to expire old sessions in the database (plus this script can be run with --init to set up the database for the new “log a user out” feature).

    Full changelog

  • Version 2.3 18 December 2017

    IMG_20130617_072808.jpg

    Today we have released version 2.3 of FixMyStreet.

    The major new feature in this release is adding confirmation by phone text instead of email, via Twilio, to allow optional verification of reports and updates, and logging in.

    Alongside that, there have been numerous front end improvements and bugfixes. Front end improvements include paginating reports everywhere, making sure all maps can be expanded on mobile, the pin loading indicator no longer covering the whole map whilst pins are loading, and improved location disambiguation on small screens.

    Some effort has gone into improving performance of various pages, especially the front page, reducing the amount of JavaScript loaded, replacing our image sprite with more SVG assets, and switching to modern prefetch. fixmystreet.com goes a step further by inlining critical CSS on the front page so no external requests are needed to show the start of the site. You can read more about this process on the mySociety blog.

    We now support Open311 category groups, and if we fetch updates via Open311, we can use auto-response templates to fill in updates without their own description.

    Lastly, the dashboard and statistics have been streamlined and improved, and are all now available under /dashboard. This now includes lookup by date range, ward, category or state, and the CSV export uses machine-readable dates and is accessible via token-based authentication.

    Bugfixes

    The number of updates on fixmystreet.com hit a million! So we made sure large numbers don’t overflow on the homepage. Also, multiple ‘Expand map’ links should no longer trouble your maps on mobile; with JavaScript off, the pins should no longer be double the size they’re meant to be; and also some bad interaction between the list filters and the back button has been fixed.

    Staff users got a number of bugfixes to their interfaces, including making sure reports could always be removed from your shortlist (even if they’ve switched body), only creating one update when changing category, and making sure the text-only contacts output was indeed text, not HTML.

    Full changelog

  • Version 2.2 14 September 2017

    Pothole

    Today we have released version 2.2 of FixMyStreet, with some new features, some bugfixes, and other improvements.

    • At long last, body and category names can be translated in the admin interface, letting you easily run sites in multiple languages without having to perform various amounts of custom work;

    • Report states can now be edited (and translated) in the admin interface, if you wish to have different options for the specific workflow of your site. If you want to have a closed state of “object removed”, now you can;

    • Also through the admin interface you can now add extra fields to be shown in the reporting form, letting you ask custom questions for your installation without needing to edit any code or templates.

    All the above improvements were funded by NDI as part of their FixMyCommunity project; thanks very much to them.


    Staff users can now create reports on behalf of people anonymously, and have more filters on report list pages; new configuration options allow you to limit your site to logged-in users only, or prevent new users from using the site.

    The email alerts page has had some work to improve its clarity, and some work has gone into improving the performance of various pages. Some deep diving into SQL queries led to some notable cases where things could be sped up, and we also found one place (listing nearby duplicate reports for staff users) where the database was being queried at a radius of 1000km, rather than 1km :)

    One notable bugfix is if you logged in during the reporting process, or had a server side error, your photo thumbnails weren’t being redisplayed correctly (though the photo itself was stored fine) due to change in the image library we use – everything should now be okay.

    Lastly, if you’re developing on the code, we have added a debug toolbar (similar to the Django Debug Toolbar, if you’re familiar with that) which lets you see which templates have been used, what SQL queries have been made, what external requests, and so on. This is automatically enabled when you run the development server.

    Full changelog