Transferring backups from to S3

I think this would make a good how to guide.

Asking here first. I understand I can install s3cmd or aws-cli. Is there any examples kicking around?

Do you mean backups taken by the “Backup” button in the UI, or something you would do manually with mysqldump and rsync?

I’m not sure of the format of the backup. But I assume (like mine) some clients have a need to store backups within their own infra or own specified location. So in my case I mean the database and files dumped and rsynced respectively over to S3 or wherever.

See: How to sync database backups to Amazon S3

I was able to transfer my PostgreSQL dumps to S3 doing the following:

  1. Create sensitive variables for my AWS credentials
platform variable:create --level=project --name=AWS_ACCESS_KEY_ID --value=<KEY_ID> --json=false --sensitive=true --prefix=env --visible-build=true --visible-runtime=true
platform variable:create --level=project --name=AWS_SECRET_ACCESS_KEY --value=<ACCESS_KEY> --json=false --sensitive=true --prefix=env --visible-build=true --visible-runtime=true
  1. Since PostgreSQL comes with a default password main, I created another sensitive environment variable PGPASSWORD to set up the dump cron job
platform variable:create --level=project --name=PGPASSWORD --value=main --json=false --sensitive=true --prefix=env --visible-build=true --visible-runtime=true
  1. I created a mount to save the dumps to in my
        source: local
        source_path: backups
  1. Installed the awscli in my build hook
        if [ ! -z "$AWS_ACCESS_KEY_ID" ] && [ ! -z "$AWS_SECRET_ACCESS_KEY" ]; then
            pip install futures
            pip install awscli --upgrade --user 2>/dev/null
  1. Created a cron job that called my forwarding script restricted to master
        spec: '*/15 * * * *'
        cmd: |
            if [ "$PLATFORM_BRANCH" = master ]; then
  1. Then included a script to perform the dumps and AWS syncs every 15 minutes
#!/usr/bin/env bash

TODAY_DATE=$(date +%Y-%m-%d-%H:%M:%S)


AWS_BUCKET=<your bucket name>
AWS_FOLDER=<bucket folder name>

pg_dump --host=$DB_HOST --port=$DB_PORT --username=$DB_USER --file=$DUMP_LOC/$DUMP_FILE

aws s3 sync ~/$DUMP_LOC s3://$AWS_BUCKET/$AWS_FOLDER/