Auto-Backup to Amazon S3

Lately I've become very critical about the safety of my backups. Especially since I experiment quite a lot with my vServer (which, among other things, hosts this website) and occasionally mess up a configuration. On top of that I love the good feeling of working on a fresh, clean system and am accordingly very "reinstall-happy". It can be very annoying to manually back up existing configurations, files and everything else you've set up each time. So what could be more obvious than automatically pulling backups of your files? For this I recently wrote a small script, which I have now published on my Git.

You can find it here: github.com/devdavede/s3backup

Installation / execution is quite simple.

Step 1: Create an S3 bucket

I'll assume you've already set up an Amazon Web Services account. If not, you can catch up here: aws.amazon.com

Then go to the S3 area by typing S3 into the search at the top, click Create bucket and enter a name and a region for the bucket.

You should disable ACLs and also disable all public access (you can read about what happens if you overlook this step here: rudrasarkar.medium.com - how-i-hacked-92k-users).

I disabled versioning (it's up to you whether you want file versioning. I decided against it because I only ever want to retrieve the latest version of my files) and also encryption (here too it's up to you whether you want to work with or without encryption).

In the next step you have to choose a username, and as the access type you choose access key - programmatic access.

In the permissions step you choose "Attach existing policies directly" and there "AmazonS3FullAccess". Click Next: Tags, you don't need to do anything here.

Go to the next step and copy the values under Access key ID and Secret access key. These keys are secret and you should definitely not share them - anyone who has this key combination can access your files.

Voilà, step one is done. You now have an Amazon S3 bucket you can access via API.

Step 2: Set up the credentials

There are now two ways to authenticate yourself against S3.

The first option is to create a credentials file. For this you create a file at ~/.aws/credentials with the following content:

[default]
aws_access_key_id = <access-key-id>
aws_secret_access_key = <secret-access-key>

and a file at ~/.aws/config with the following content:

[default]
region = <region>

(if you didn't note down which region you created the bucket in, you can look it up in the web management console)

That's it; the Python script can now authenticate against S3 with this data.

The second option is to enter the credentials directly in the script (see the variables aws_access_key_id, aws_secret_access_key and region_name). This approach is not recommended, though, since in this case the credentials are unprotected and visible to anyone who has access to the script.

Step 3: Create the my.cnf

If you want to use the option to back up your database as well, you need to store the database credentials. The best way is to do that (if not already done) via my.cnf.

Type the following commands into the shell:

m=`which mysqld`
$m --verbose --help | grep -A 1 "Default options"

and you'll get the path (usually /etc/my.cnf) to your my.cnf

Now open it and add the following part to the end of the file

[client]
user=<your-mysql-root-username>
password=<your-mysql-root-password>

That's it. After that your user can log in to the MySQL shell without entering a username/password.

Alternatively, here too: you can enter the credentials directly in the script. I've already mentioned the downsides of that above :-)

Step 4: Configure the script

Now upload the script to your server and edit it a bit. The following variables need to be edited:

• entities = an array of folders or files to be backed up (including subfolders!)
• bucketname = the name of your S3 bucket
• db_path = the path to the database in which the checksums of the files are stored
• aws_access_key_id = your AWS access key. Leave as None if you use the config file from step 2
• aws_secret_access_key = your AWS secret key. Leave as None if you use the config file from step 2
• region_name = the name of your bucket region. Leave as None if you use the config file from step 2
• backup_databases = True if backups of all databases should be created, otherwise False
• mysql_username = your MySQL username. Leave as None if you use the config file from step 3
• mysql_password = your MySQL password. Leave as None if you use the config file from step 3
• mysql_tmp_dump_file = a temporary database dump is created here. The folder must be writable.
• s3_mysql_dump_path = the path of your database dump in your S3 bucket

Step 5: Install the dependencies

For the upload to S3 the script uses Boto

TinyDB is also used to store the checksums of the files in a small database (the path in the variable db_path)

To install all dependencies, simply run the following command in the shell:

pip3 install -r requirements.txt

which installs all required packages.

Step 6: Run it!

Start a test run by entering the command

python3 s3backup.py

and when the script has finished, look into your S3 bucket. In the same hierarchy as on your server you should now see a 1:1 copy of all files/folders/subfolders specified in the variable entities.

Step 7 (Extra): Automate via cron job

This can also be automated easily via cron job

First, let's find out where our Python 3 installation is located:

which python3

that spits out something like /usr/bin/python3.

Then we enter

crontab -e

and put the following in

0 0 */5 * * <path-to-python3> <path-to-backup-script.py>

Confirm with Ctrl+X (close) and Y (yes, save!). This runs the backup automatically every 5 days at 24:00.