Migrating to S3 photo storage
The recent release of FixMyStreet v2.4.1 brought with it the ability to store photos for reports & updates in a bucket on Amazon’s S3 instead of locally on disk.
Getting a new site configured to store newly uploaded photos in S3 is explained
in the configuration pages, and
is a matter of tweaking a few keys in your
general.yml file. But what about
if you have an existing FixMyStreet installation that you would like to
migrate to storing all photos in S3? This post outlines a process for
achieving just that - without any downtime.
- A running FixMyStreet site with at least one report or update with a photo
- An account on Amazon AWS, and access/secret keys for a role that can create/manage S3 buckets
- The AWS CLI tool installed on your FixMyStreet server, and configured with access credentials
Creating the bucket
First off, we need to create a bucket to hold all the photos. If you’ve already done this, skip ahead to the Migrating photos section.
We’ll show two ways to create a bucket: via the S3 web console and using the
aws command line tool.
Using the S3 web console
Log in to S3 web console, and click the ‘Create Bucket’ button
Enter a name for the bucket, e.g.,
my-fixmystreet-photos, and choose the region closest to your FixMyStreet server (more about buckets and regions):
Your newly created bucket should then appear in the list of buckets. Make sure public access isn’t allowed (photos are served to users via your FixMyStreet server, not directly from S3):
If everything looks OK, skip to the Migrating photos section.
aws CLI tool
You can also create buckets from the command line using the
To create a bucket named
my-fixmystreet-photos in the
eu-west-1 region, run the following:
$ aws s3 mb s3://my-fixmystreet-photos --region eu-west-1
The following success message should be shown:
The bucket’s region can be inspected:
$ aws s3api get-bucket-location --bucket my-fixmystreet-photos
The bucket should be created as private by default, but you may wish to check its access policy on the S3 web console.
Now that the bucket has been created, the next step is to fill it with all the photos
from your existing FixMyStreet installation. For this we’ll need the
aws CLI tool,
so make sure you’ve installed and configured it.
Existing photos are stored on disk in the directory specified by the
PHOTO_STORAGE_OPTIONS.UPLOAD_DIR) configuration key in
For this example, let’s assume that’s
Running the following (with the path and bucket name altered to suit your configuration) will make copy-pasting the subsequent commands a little easier:
$ export UPLOAD_DIR=/home/fixmystreet/photos
$ export S3_BUCKET=my-fixmystreet-photos
aws s3 sync to copy all photos to the new S3 bucket. To get a preview of
the operations it will perform without actually copying anything, run:
$ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET --dryrun
You should see output similar to the following, one line for each file:
(dryrun) upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
(dryrun) upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
(dryrun) upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
If the output looks correct, re-run without the
--dryrun flag to actually copy
everything to the S3 bucket:
$ aws s3 sync $UPLOAD_DIR s3://$S3_BUCKET
upload: /home/fixmystreet/photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg to s3://my-fixmystreet-photos/069a5d216321061757fe30a6d7f862669eb46d7d.jpeg
upload: /home/fixmystreet/photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg to s3://my-fixmystreet-photos/0d7db3b2cd615e27c50345c8144b6b9782d7ff4a.jpeg
upload: /home/fixmystreet/photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg to s3://my-fixmystreet-photos/137dfb5fe88c0207e2a339b02fcbc5e812b4e68b.jpeg
All your photos are now in S3, so the next step is to tell FixMyStreet about the new bucket. (See the note in the final section for reports made between now and when the FixMyStreet config has been updated to use S3.)
Configuring FixMyStreet to use S3
We need to make a few changes to
conf/general.yml to configure FixMyStreet to
use the new S3 bucket for photo storage.
conf/general.yml probably has a section similar to the following:
Or, if you set the site up before v2.4.1:
Either way, you need to replace that section with the following:
Make sure you set
SECRET_KEY to the same values
as when you configured the
aws CLI tool and created the bucket.
These changes are all that’s needed to switch your FixMyStreet installation to using
S3 for photos. Once you’re happy with the changes you’ve made to
restart the FixMyStreet app server (example) and
check the output for any errors.
Once the server has restarted, try making a new report and uploading a couple of photos. You should see new files appear in the S3 bucket (you can view them via the S3 web console) as soon as you’ve drag-dropped them into the new report (or update) form. Check that report photos appear on the site for newly-created reports too. NB photos for existing reports are likely already cached on disk by FixMyStreet, so do check a new report.
Final steps & tidying up
Because there was a small window of time between syncing existing photos to the S3 bucket
and switching over the
conf/general.yml config, there’s a chance that a report with
photos was made whose photos weren’t synced to S3.
Now that FixMyStreet is using S3, you may wish to re-run the
aws s3 sync command from above
in order to make sure any such photos have been transferred to S3.
The old directory containing photos on disk isn’t needed any more, so once you’re happy the S3 storage is working as expected, you can delete this.