Version 2.0 – testing improvements
FixMyStreet has a large and hopefully comprehensive test suite that runs through all aspects of the codebase, checking everything is working. This makes it easier to change code and add new features, safe in the knowledge that any breakages will be quickly highlighted.
Speeding up the tests
Every time someone commits code to our GitHub repository, or opens a pull request, the tests are automatically run for us by Travis CI. We’re alerted to success or failure with little green ticks or red crosses on GitHub, and by notice in IRC.
The tests seemed to have slowed down considerably in recent times, but we couldn’t identify any changes at the FixMyStreet side which might have caused this.
However, there had recently been some spam scraping of Gaze, our web service that provides population density information to FixMyStreet (so that e.g. the maps can try and guess an appropriate zoom level, and so alerts can try and guess an appropriate radius), and rate limiting had been added to try and help combat it.
Dave spotted that this was being triggered by FixMyStreet test runs, leading to pauses as the suite waited for the rate limiting to ease. Thankfully, all Gaze calls were being routed through one function (that had been created in order to cope gracefully with a Gaze failure) and so it was a simple matter for this function to be stubbed out if being run as part of a test.
There are many tests that still rely on the internet (e.g. for some MapIt lookups) and eventually it would be good to get to the point where they are all stubbed out and the test suite can run completely offline, probably even more quickly.
Multiple test running
When running the tests, the suite creates a test database (in PostgreSQL terms,
it actually creates a temporary cluster) so that anything it does won’t affect
your development database. Theoretically, this means you should be able to run
the test suite multiple times simultaneously – perhaps it’s doing a full run,
but you want to try and fix (and retest) the first error while it carries on.
However, this was not working, and after some investigation it turned out that
each run was creating (and overwriting) a test configuration
.yml file, which
meant the existing runs got all confused. Adding a process ID to the test
configuration file meant that each run is independent and can successfully
coexist with each other.
Lastly, you used to have to run the full suite with
bin/run-tests t, but now
if you run
bin/run-tests, it will assume you meant
t. A small thing, but it
might save a few seconds over the years. ;-)