Blog

  • Version 2.3.1 12 February 2018

    Keep Behind This Line

    Today we have released version 2.3.1 of FixMyStreet.

    This is a minor release, with a number of bug fixes but also a number of smaller front end and admin improvements.

    The site should be clearer when things are loading now, be that on initial page load or when an asset layer is incoming. Our phone number library has been updated, so national phone numbers should be displayed in a nicer format as long as you’ve set the PHONE_COUNTRY variable.

    FixMyStreet now asks for the current password (or sends an email) on password change, sets a minimum password length and checks against a list of commons passwords. Superusers can have optional two-factor authentication to protect their accounts.

    We fixed an error in sending requires_inspection reports, issues with multiple select-multiples on a page, a questionnaire CSS snafu, showing deleted bodies in the dashboard list, alongside other minor bugfixes.

    For staff users, ‘report as another user’ now allows phone number without email, and the inspector form shows any extra fields again.

    Admins can now anonymize/hide all a user’s reports, log a user out, or remove a user’s account details. We’ve provided a script to expire old sessions in the database (plus this script can be run with --init to set up the database for the new “log a user out” feature).

    Full changelog

  • Version 2.3 18 December 2017

    IMG_20130617_072808.jpg

    Today we have released version 2.3 of FixMyStreet.

    The major new feature in this release is adding confirmation by phone text instead of email, via Twilio, to allow optional verification of reports and updates, and logging in.

    Alongside that, there have been numerous front end improvements and bugfixes. Front end improvements include paginating reports everywhere, making sure all maps can be expanded on mobile, the pin loading indicator no longer covering the whole map whilst pins are loading, and improved location disambiguation on small screens.

    Some effort has gone into improving performance of various pages, especially the front page, reducing the amount of JavaScript loaded, replacing our image sprite with more SVG assets, and switching to modern prefetch. fixmystreet.com goes a step further by inlining critical CSS on the front page so no external requests are needed to show the start of the site. You can read more about this process on the mySociety blog.

    We now support Open311 category groups, and if we fetch updates via Open311, we can use auto-response templates to fill in updates without their own description.

    Lastly, the dashboard and statistics have been streamlined and improved, and are all now available under /dashboard. This now includes lookup by date range, ward, category or state, and the CSV export uses machine-readable dates and is accessible via token-based authentication.

    Bugfixes

    The number of updates on fixmystreet.com hit a million! So we made sure large numbers don’t overflow on the homepage. Also, multiple ‘Expand map’ links should no longer trouble your maps on mobile; with JavaScript off, the pins should no longer be double the size they’re meant to be; and also some bad interaction between the list filters and the back button has been fixed.

    Staff users got a number of bugfixes to their interfaces, including making sure reports could always be removed from your shortlist (even if they’ve switched body), only creating one update when changing category, and making sure the text-only contacts output was indeed text, not HTML.

    Full changelog

  • Version 2.2 14 September 2017

    Pothole

    Today we have released version 2.2 of FixMyStreet, with some new features, some bugfixes, and other improvements.

    • At long last, body and category names can be translated in the admin interface, letting you easily run sites in multiple languages without having to perform various amounts of custom work;

    • Report states can now be edited (and translated) in the admin interface, if you wish to have different options for the specific workflow of your site. If you want to have a closed state of “object removed”, now you can;

    • Also through the admin interface you can now add extra fields to be shown in the reporting form, letting you ask custom questions for your installation without needing to edit any code or templates.

    All the above improvements were funded by NDI as part of their FixMyCommunity project; thanks very much to them.


    Staff users can now create reports on behalf of people anonymously, and have more filters on report list pages; new configuration options allow you to limit your site to logged-in users only, or prevent new users from using the site.

    The email alerts page has had some work to improve its clarity, and some work has gone into improving the performance of various pages. Some deep diving into SQL queries led to some notable cases where things could be sped up, and we also found one place (listing nearby duplicate reports for staff users) where the database was being queried at a radius of 1000km, rather than 1km :)

    One notable bugfix is if you logged in during the reporting process, or had a server side error, your photo thumbnails weren’t being redisplayed correctly (though the photo itself was stored fine) due to change in the image library we use – everything should now be okay.

    Lastly, if you’re developing on the code, we have added a debug toolbar (similar to the Django Debug Toolbar, if you’re familiar with that) which lets you see which templates have been used, what SQL queries have been made, what external requests, and so on. This is automatically enabled when you run the development server.

    Full changelog

  • Version 2.1.1 3 August 2017

    Untitled

    Today we have released version 2.1.1 of FixMyStreet, a bugfix release with a few small improvements.

    The map in HTML emails is now a link through to the report page, and we’ve fixed a bug causing the wrong pin size to be shown there. We’ve also fixed a bug that could cause a different language from the requested one to be shown on about pages.

    Admin improvements include resending reports if changing the category has changed the send_method and displaying the reporter’s phone number on the inspector form. Text deleted during moderation is no longer replaced with [...].

    Full changelog

  • Version 2.1 18 July 2017

    Untitled

    Today we have released version 2.1 of FixMyStreet, with some new features and a variety of improvements for users and reusers.

    New features include allowing users to hide their own name on reports/updates, and a new graph-based /reports page, which you can see in operation on our main UK site. We also now resize photos client-side before uploading, given the increase in very large images these days.

    Admin improvements include adding an ‘inactive’ state to categories, so you can prevent a category being used for new reports but still have it available in filters, various improvements to the inspect form and the new report process for inspectors, and an easier way for inspectors to shortlist all reports in their view.

    Geolocation is back on the alert page, lost sometime last year, and some random blank spaces are no longer clickable, hooray!

    Development-wise, FixMyStreet now supports Debian stretch and perl 5.24, badly configured SMTP options should now be spotted, and we have refactored and simplified the CSS used for header/content/navigation. This may mean you can simplify overrides in your cobrand. If you wish to have a static front page rather than the normal report page, that’s now possible by creating an about/homepage.html template in your cobrand, and cobrands can now easily change the new report pin colour.

    The test suite now runs each file in a transaction which means it can be run in parallel and sped up dramatically; on the other hand, on Travis we added code coverage which slowed it down again.

    Two backwards incompatible changes that may require changes to your own templates if you have customised them:

    • The nav-wrapper-2 class has been removed. If you have a custom footer template, replace that class with ‘container’.

    • The /reports page now uses different generated data. If you have a custom reports/index.html template and wish to keep it, you may need to call update-all-reports with a --table argument to generate the old style data.

    Full changelog

  • Version 2.0.4 13 April 2017

    11

    Today we have released version 2.0.4 of FixMyStreet, a bugfix release along with a couple of other improvements.

    The bugfixes are updating our Facebook library so that Facebook login works again, stopping an error if you had a devolved body and its contact both lacking a send method, and an issue in the multi-select front end if you had characters such as brackets in your category name.

    Strangely, while the backend has always allowed multiple email addresses for a contact, separated by commas, the admin interface hasn’t allowed them to be entered - this has now been corrected.

    Lastly, body pages under /reports will now limit the reports shown to those within the visible map, which should make moving and zooming around much more intuitive.

    Full changelog

  • Version 2.0.3 31 March 2017

    Untitled

    Today we have released version 2.0.3 of FixMyStreet, a bugfix release along with some other improvements.

    The map on a mobile report page can now be made full screen, and if you are using Google Maps you can supply a custom map styling. There’s also now a loading indicator whilst data is being fetched for the map.

    Various missing translations have been added, for moderation, social login, and offline usage.

    We’ve upgraded our email sending to deal with issues sending with SSL, dealt with IE11 caching report Ajax calls too aggressively, and with Safari 5/ Android 4 not showing our questionnaire answer buttons.

    Performance improvements include moving admin-related JavaScript to its own file (so normal users don’t need to download it), and reducing the amount of disk stats the code performs.

    Lastly, all the test suite can now run offline, and amusingly I found a bug in a test that only happened if the test was run c. 55 hours before daylight savings time began :)

    Full changelog

  • Version 2.0.2 3 February 2017

    Lake Annecy

    Today we have released version 2.0.2 of FixMyStreet, a bugfix release along with some new admin improvements.

    The main bugfixes are to mark two missing strings for translation, to make sure email is lowercase when signing in via /auth, and to make sure a language subdomain can be included in calls to base_url_for_report. There are also some CSS fixes to the homepage and an improved print layout for report list pages. Fixes to the admin interface include fixing filtering on the shortlist page, and fixing the ‘save with public update’ toggle.

    For international users, there have been a few string renames, but we have maintained translations as the concept remains unchanged, so please only feel free to update if you consider it necessary:

    • ‘unable to fix’ has been renamed to ‘no further action’
    • ‘marked as a duplicate report’ has been changed to ‘closed as a duplicate report’

    An offline fallback page has been added using appcache, which inspectors can use for offline updating of reports in their shortlist.

    Admin improvements include allowing response templates to be associated with a state (so that template is used by default when the state is changed), allowing shortlist addition/removal from report lists, and shortlist reordering. Users with a new permission can see the body user who left a contribute_as_body report or update.

    Full changelog

  • Version 2.0.1 16 December 2016

    drowning

    Today we have released version 2.0.1 of FixMyStreet, a bugfix release. The issues fixed are:

    • Chrome 55 changes the way it handles mouse/touch events, unifying them as pointer events, and OpenLayers, the JavaScript mapping library we use, needed patching to deal with this. Thanks to the Chrome team for helping debug this.
    • strftime output (e.g. used to display dates) was sometimes being double decoded, giving incorrect output in some languages.
    • If a category was filtered, it was not being properly carried through to the new report form.
    • The body/area inputs in the admin were being fixed to a too-small height.

    Some bugfixes and improvements have also been made to the admin inspector report view, along with a couple of development improvements – the tests coping better if run on a network that intercepts NXDOMAINs, and a better way of showing the git version in the admin config.

  • Version 2.0 – JavaScript performance 16 December 2016

    The JavaScript on FixMyStreet has gradually evolved over many years (we launched in 2007, remember!), and while we were working on other features in this area (such as HTML5 History) it was a good opportunity to tidy up the JavaScript, making it clearer and simpler. Below I’m going to go through most of the steps I took, not necessarily in the order I took them, which hopefully might prove useful to your own websites. And there are exciting pictures at the end, I promise!

    It also let us add the Content-Security-Policy header to FixMyStreet, which is a method in browsers to prevent cross-site scripting (XSS), clickjacking and other code injection attacks by specifying the valid sources for script execution.

    Separate scripting and styling

    When you report an issue on FixMyStreet on mobile, the map /around page is full-screen to make reporting easier. This was being done in JavaScript using some jQuery css() calls when setting up (or ending) the mobile view. It was straightforward to move this CSS to a mobile-reporting-map class and have the JavaScript do no more than add or remove this class, making both parts clearer. (Later during this process, I also added an only-map class to prevent the map being scrolled until it is clicked, when the rest of the form can then be shown.)

    Similarly, the JavaScript was requisitioning the desktop-width green banner to provide a different mobile banner; updating this to use a separate mobile banner made the code clearer and shorter.

    Tidy up the JavaScript set up

    The main setting up of our JavaScript was taking place, for historical reasons, in two files, both confusingly called fixmystreet.js. Whilst in the future we may want to split this up into separate files more tailored to the particular pages where the code is used (though our main use of JavaScript is our map page, which already has its JavaScript separate), for now it made most sense to combine these in one file, and tidy up all the setup functions into a list of feature-based functions each called in turn on page load.

    We were bundling a copy of Modernizr (for media query detection) that contained html5shiv and yepnope. But html5shiv is only needed in old versions of Internet Explorer, and yepnope is only used on FixMyStreet’s front page (to try and preload the map page JavaScript mentioned above), so I could move html5shiv into an IE conditional comment, and included yepnope.js only on the front page, reducing Modernizr to only Modernizr itself.

    Move vital JavaScript as early as possible

    Now that I had a mobile-reporting-map class, I wanted this class activated as soon as possible as the page is loading, not only when the document had been parsed. There were also a couple of site-wide variables, page (the type of page, e.g. around or new), and cobrand (the branding of the site you’re on). Lastly, I wanted to be able to set a class on the document if JavaScript was activated, so that CSS using that class would be instantly active, preventing a flash of style change or content.

    To achieve all this, I created a header.js file that performed the above three tasks, setting a class on <html>, setting two variables on our global fixmystreet variable, and if we’re on a small width (using Modernizr) and perhaps a map page, setting the appropriate classes. I then minimized and inlined this script in the header of each page, so that we don’t have to wait for any external script to load.

    Move JavaScript to the end of the HTML, remove inline JavaScript

    As recommended by Jake Archibald here, I moved all the JavaScript to the end of the HTML. To make this easier, and also to make adding a Content-Security-Policy header easier, I removed all the inline JavaScript from FixMyStreet (I didn’t think FixMyStreet had that much inline JavaScript, and it didn’t, but it still had more than I had remembered!). This was most commonly being used to set up some JavaScript variables with server-side data, so the easiest way to replace this was to place this data as attributes on HTML elements (preferably semantically related elements), and set up the JavaScript variables in an external script.

    Minify JavaScript

    I didn’t want to make this mandatory, as we have done with SCSS/CSS, but I wanted the option available, so I added an option to our templating code that means it will prefer an .auto.min.js file in preference to a .js file of the same basename. This lets you compile your JavaScript in a deploy process, for example, should you wish to. We do this with the standard Closure Compiler from Google; I haven’t yet been brave enough to try and check/get the JavaScript working with the advanced option of the Closure Compiler :)

    Activate Content-Security-Policy

    The Content-Security-Policy header lets you specify domains from which JavaScript will run, plus lets you choose to run inline JavaScript either en masse or only if you provide a specific unique nonce ID in the <script> tag that matches the same ID in the CSP header. That latter option is how we kept our inline header JavaScript running without having to externalise it again.

    “nonce” was only added in the second version of the CSP spec, so you may note our header also specifies unsafe-inline. Any browser that supports version 2 will ignore this when it sees the nonce header, but it is needed in order for the inline script to still run in any browser only supporting version 1.

    Conclusion

    In Google Page Speed Insights, with manual minification of the main JS files, this moves the front page from 68/84 to 85/92ish (filmstrip from webpagetest.org, top is live site, bottom is my dev site):

    filmstrip

    These are requests from the US: most of the initial delay is in that initial download. Now here’s a report page going from 58/77 to 85/86ish (“ish” because e.g. live site will have analytics that my dev site doesn’t):

    Site visible a second quicker

    Due to page elements displaying more quickly than might previously have been expected, there are still a few things to tidy up. For example, when an element displays before a bit of JavaScript kicks in and makes it look slightly different… but hopefully nothing major.