Brontesaurus Blog

Previously, I was backing up my websites password-less ssh and rsync, but I have decided to change that for a couple of reasons such as passwordless ssh can have its security problems, and I dont trust hardware. Using S3 I can securely keep as many full backups as I’d like, not have to worry about hardware failing, all for pennies a month. Currently I backup my desktop with JungleDisk and thought I may as well consolidate all my backups to S3. 

To accomplish this goal of backing up your website you will need 2 things, SSH access and s3sync which is a handy little S3 client written in ruby.

1. Download and setup s3sync from http://s3sync.net/wiki. Once you untar the file open up s3config.yml (may be named s3config.sample.yml, just rename it) and this is where you set the path to your certficates, youAWS access, secret access keys, these can be found in “Access Identifiers” section of your AWS account.
2. Now lets test this out. S3sync needs to know where its configuration file is so we just export the variable S3CONF with the path to your s3config.yml file.

   [user@machine s3sync] export S3CONF=/home/bront1/s3sync

   To make sure everything is working do a simple command to list your buckets.

   [user@machine s3sync]$ ./s3cmd.rb listbuckets
   bucket1
   bucket2
   bucket3

   Yay we have s3 connectivity!

3. Next we dump the databases, tar and gzip everything up.

   [user@machine] mysqldump -u user -pPassword --all-databases > /home/user/backup.sql
   [user@machine] tar -cf /home/user/backup.tar /home/user/public_html /home/user/backup.sql | gzip > /home/usr/backup.tar

4. Copy the backup to your S3 bucket using s3sync’s s3cmd.rb

   [user@machine s3sync] ./s3cmd.rb put bucket-name:folder/target_name.tar.gz /home/user/backup.tar.gz

5. Success! You have now backed up your website and database to your S3 account.

S3sync has its own sort of built in rsync to make incremental backups, but I prefer to make keep my own daily backups. Heres the script I use to create daily backups of my websites and files. It creates a backup with day, month, and year in the filename and keeps backups for the last 10 days or so. (Note: The sed and awk commands are really messy due to my lack of sed/awk knowledge. This script is run everyday at 3am using a cron job with the output written to a log.

Crontab:

0 3 * * * export S3CONF=/home/user/s3sync; /home/user/backup.sh >> /home/user/backup.log

backup.sh

#!/bin/bash
 
export S3CONF=/home/user/s3sync
 
cd /home/user
 
TIMESTAMP=`date +%m%d%Y`
 
echo "$TIMESTAMP :: Backuping up the databases"
mysqldump -u user -pPassword --all-databases > /home/user/backup.sql
 
echo "$TIMESTAMP :: Bundling all the files up"
 
tar -cf /home/user/backup_`date +%m%d%Y`.tar public_html backup.sql
gzip -f /home/user/backup_`date +%m%d%Y`.tar
 
echo "$TIMESTAMP :: Copying backup to S3"
 
#we use full path because this script is running in a cron job
/usr/local/bin/ruby /home/user/s3sync/s3cmd.rb put bucket:folder/backup_`date +%m%d%Y`.tar.gz /home/user/backup_`date +%m%d%Y`.tar.gz
 
echo "$TIMESTAMP :: Cleaning up"
rm -f /home/user/backup_`date +%m%d%Y`.tar.gz
rm -f /home/user/backup.sql
 
echo "$TIMESTAMP :: Checking for old backups"
 
#check how many backups are saved
num=`/usr/local/bin/ruby /home/user/s3sync/s3cmd.rb list bucket:folder | wc -l`
 
#we save at least 10 days of backups
#13 is checked for due to other crap s3cmd prints out
if [ "$num" == "13" ]; then
  echo "$TIMESTAMP :: Deleting old backup"
 
  #i know there is a better way to check this, i just dont know how
  last=`/usr/local/bin/ruby /home/user/s3sync/s3cmd.rb list bucket:folder | sed -e 's/-//g' | awk '{printf("%s", $0 (NR==1 ? "" : " "))}' | awk '{print $2}'`
 
  /usr/local/bin/ruby /home/user/s3sync/s3cmd.rb delete bucket:$last
else
  echo "$TIMESTAMP :: No old backup to delete"
fi
 
echo "$TIMESTAMP :: Done"

S3 is Amazon web service’s cloud storage solution offering very cheap price-per-gb storage and bandwidth which makes this a very cost effective backup solution. Currently I use JungleDisk for mounting S3 buckets as drives and to schedule backups, and s3sync to backup my files in various unix enviroments but I dont have anything in between to where I can easily manage my S3 files. That is where Cloudberry Explorer for Amazon S3 comes in. I have been using this for the past couple of hours or so and to say the least I am impressed. 

Here is what I’ve noticed while using Cloudberry.

• First off, this software is completely free. That is a huge plus because what I was using before, CrossFTP, you had to pay 25$ for the pro version to get S3 support and I had a whole slew of gripes about it.
• The UI is very clean and responsive. 
• Tabbed! Everything is better with tabs, especially if you are dealing with multiple uploads and/or multiple S3 accounts
• Very easy to upload and manage files with the intent of serving them on the web
• Cloudfront integration
• The ability to bill the requestor for a file for the S3 usage. I messed around with this for a bit and I think its just a flag that is set on the file? Not sure how to implement it.
• Plugins to Microsoft Powershell

Though through using this there are a couple things that could be added that would be nice.

• In the “MyComputer” pane, shortcuts to Desktop, My Documents, etc would be nice.
• Also labels for each of the drives I have. I have so many drive labels I often forget which one is a HDD, network drive, or DVD drive.
• Refresh the S3 pane after I copy a file to a bucket so I can see that file there after it is finished copying.

Overall, this is a pretty solid S3 client and I plan to keep on using it. 

Cloudberry Explorer for Amazon S3

I have created a AWS EC2 AMI with a couple source dedicated servers installed on it. Lately I’ve been using EC2 to start on demand game servers for whenever my friends and I want a private server to play on with the settings we all like. 

This image is based on Amazon’s Fedora Core 8 32bit Official image. The latest HLDS update tool and Day of Defeat Source, Counter-Strike Source, and Team Fortress 2 games are installed with example server.cfg files for each game.

AMI Id: ami-7906e110

Manifest: fc8-srcds-01/image.manifest.xml

readme.txt:

This is the source dedicated server EC2 image.

Brought to you by Adam Bronte
Find updates and information at http://brontesaurus.com

All source ds servers are in /home/steam

Installed Games:
=================
* Team Fortress 2
* Counter-Strike Source
* Day of Defeat Source

Running Day of Defeat Source Server
========================
Dods Config Files: /home/steam/hlds/orangebox/dod/cfg

To start the server:
* cd /home/steam/hlds/orangebox
* ./srcds_run -game dod

Ex: ./srcds_run -game dod -secure +maxplayers 24 +exec server.cfg

Running Team Fortress 2 Server
========================
TF Config Files: /home/steam/hlds/orangebox/tf/cfg

To start the server:
* cd /home/steam/hlds/orangebox
* ./srcds_run -game tf

Ex: ./srcds_run -game tf -secure +maxplayers 24 +map cp_dustbowl +exec server.cfg

Running Counter-Strike Source Server
========================
CS:S Config Files: /home/steam/hlds/cstrike/cfg

To start the server:
* cd /home/steam/hlds
* ./srcds_run -game cstrike

Ex: ./srcds_run -game cstrike +maxplayers 24 +map de_dust +exec server.cfg

Amazon EC2 just rolled out a new sudo pre-payment plans for all its instance sizes. With this new plan you pay a flat up front fee for a one or three year term. In exchange for this pre-payment you receive discounted per-hour price. A lot of people are using EC2 as a permanent web host and dont need to scale up like other services with possibly hundreds of instances being created or terminated at any given time. So what Amazon did here was give their consistent users a break with this new pricing structure.

This is a table with the pre-payment fees and what discount you get with them.

Actual costs. Pre-payment 1/3 year vs none.

Currently the reserved instance pricing is only available for Linux/Unix instances.

[Via Techcrunch]