Pfthb

I'm making daily backups of my mail servers emails.

Each night a tar file is updated to include the new emails from that day, and downloaded via ftp to a backup server.

As time goes on, this tar file is getting too large to handle.

To get around this, I decided it would be best to create a new tar file, that has all mails in it, minus the ones that already exist in last night's backup. So it ends up with just the new files from that day.

I can then transfer this significantly smaller file to my backup server.

Finally I can merge the new tar file in to the master on both servers, ready for the next day.

Any ideas how I can do this?

Using tar with find
Here is an example script to create full and incremental backups using find and tar

#!/bin/bash
bkdir="/home/backup"
bklog="/var/log/backup.log"
dbkdir="/home/backup/files/daily"
wbkdir="/home/backup/files/weekly"


curdate=`date +%Y-%M-%d-%H:%M:%S`
ardate=`echo $curdate | sed -e 's/:/_/g' -e 's/-/_/g'`
wday=`date +%a`

files_full_backup () 
 echo -e "Archiving files...n"
 tar cjpf "$1/full_files_$ardate.tar.bz2" "$bkdir"


files_inc_backup () 
 echo -e "Archiving files...n"
 find $bkdir -mtime -1 -exec tar cvjpf "$1/inc_files_$ardate.tar.bz2" ;


### add some choice what kind of backup to do - full or incremental
if [ $wday != Sun ]
 then
 echo -e "As today is not Sunday - I'll start incremental backup.n"
 files_inc_backup $dbkdir
 else
 echo -e "As today is Sunday - I'll start full backup.n"
 files_full_backup $wbkdir
fi

Using tar only (incremetnal backup )

man tar shows that is has "incremental feature":

-g, --listed-incremental=FILE
 Handle new GNU-format incremental backups. FILE is the name of a snapshot file, where tar stores addi‐
 tional information which is used to decide which files changed since the previous incremental dump and,
 consequently, must be dumped again. If FILE does not exist when creating an archive, it will be cre‐
 ated and all files will be added to the resulting archive (the level 0 dump). To create incremental
 archives of non-zero level N, create a copy of the snapshot file created during the level N-1, and use
 it as FILE.

 When listing or extracting, the actual contents of FILE is not inspected, it is needed only due to syn‐
 tactical requirements. It is therefore common practice to use /dev/null in its place.

To create incremental backup use:

tar --create --file=`date +%s`.tbz2 --bzip --listed-incremental=example.snar --verbose example/

or in short form:

 tar -cvjg example.snar -f `date +%s`.tbz2 example/

To restore backup it is needed to unpack all part of a bcakup from oldest to newest:

 tar --extract --incremental --file level0.tar
 tar --extract --incremental --file level1.tar
 tar --extract --incremental --file level2.tar

Or, in short form:

 for i in *.tbz2; do tar -xjGf "$i"; done;

And here is a script to create a zero level archive will once a week (or once a month, depends on a commented row):

#!/bin/sh
 SOURCE="$1"
 test -d "$SOURCE" || exit 1

 DEST_DIR=`date +%G-%V`; #weekly
 #DEST_DIR=`date +%Y-%m`; #monthly

 mkdir -p $DEST_DIR;
 shift;
 tar --create "$@" --preserve-permissions --totals --bzip 
 --file="$DEST_DIR"/`date +%F-%s`.tbz2 
 --listed-incremental="$DEST_DIR"/backup.snar 
 --no-check-device --exclude-vcs 
 --exclude-tag-under=access.log --exclude='*.log' 
 --exclude-caches --exclude-tag-under=IGNORE.TAG "$SOURCE"

And execute it :

 ./backup.sh example/ -v