Overview: --------- CVS: $Id: systembackup-manual.txt,v 1.24 2003/11/09 20:49:23 pj Exp $ This is the manual for 'systembackup.sh', a shell script used as a wrapper around tar archiver 'star'. The purpose of this script is to simplify systembackups. It is not intended for ad hoc backups, tar or star can be used for that without a script. 'star' lives at this address: http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/star.html 'systembackup.sh' lives at this address: http://www.net-es.dk/net-es/Systembackup.do License: -------- THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. Systembackup.sh is licensed under GPL2 The file COPYING should accompany this script detailing the license terms. It not, please visit this URL 'http://www.gnu.org/licenses/gpl.txt' to read the terms. Requirements: ------------- - A Python interpreter >= than version 1.5.2 - 'star' installed - awk for extracting the cvs version Installation: ------------- 1. Get the most recent copy of 'star' from it's website and compile/install it. Don't be afraid, it's simple and well explained. 2. Get the most recent version of systembackup.sh from http://www.net-es.dk/systembackup/systembackup.tar and untar it to a directory. I personally use /opt/systembackup: 'star xv -C /opt -file systembackup.tar' 3. Edit /opt/systembackup/systembackup.sh and set variables correct. Variables in script: 'date': a date is appended to the archivenames to make them unique, default is of this form: '2002-12-08' 'hostname': name identifying the machine on which this script is executing. 'star': path to 'star' program, 'chunksize': archives are now split in chunks because I have a pro- blem getting mkisofs to accepts tar files with size > 2G. Chunksize is configurable and defaults to 1024M = 1 Gigabyte. 'dir': directory where tar archives are stored. 'installdir': directory where supportscripts, logfiles and incremental level files are stored. 'archivedir': directory where tarfiles, filelist files and partition info is stored. 'archiveincdir': directory where incremental backups are stored 'logfile': defaults to ${installdir}/systembackup.log 'email': who to send this run's logfile. 'level': used to control if systembackup is an incremental one. If the script is invoked with the '-sysinc' option, all files of name 'level.a_number' will be examined and the largest number determined. Files with 'atime' later than determined level-file's timestamp will be backed up. Each time 'sysinc' options are used a new 'level.a_number' will be touched to mark a new timestamp. Doing a -system backup clears all level files and the processed is repeated. 'backup-options': the set of options 'star' given when backup is performed. It defaults to: -c: create tar file -M: don't cross partitions -atime: reset access time, this changes ctime (change time) -dirmode: store directories after files -fifo: optimize data flow -no-statistics: don't want that -silent: suppresses informational messages not critical to operation -dump: record more inode data 'restore-options': the set of options used when restoring: -x: extract files -p: restore filemodes -nowarn: don't display non critical information -xdir: force extract directories Description: ------------ The purpose of this script is to take a complete snapshot of a server. The backup is divided into several files, one for each partition backed up. The backup of the partitions must be set up in the script and tested before going into unattended operation. When an archive has been written, a complete filelist of all files in each archive is written. This is for easy grep'ing later. After the filelist generation, the archive is verified against the file system. The star program's 'diff' verify is done on filesize. If for any reason a file is not found on the filesystem when the verification takes place, 'star' issues a warning and exits later with a return code different from 0. This causes the wrapper script to report a verification failure. This will happen somewhat often especially on the /var partition and the person responsible for the backup must take a look at the logfile to see what caused the verification failure. The tar archives are not compressed. I my opinion it is too risky to compress the whole file. If a single bit goes wrong the entire archive is lost. This is not the case if the archive is uncompressed. In that case 'star' will be able to continue and probably restore the greater part of the archive. The archives are split into separate chunks. This is transparent for users of the script. One reason is that the script can be used on older linux systems which support filesizes up to 2 GB to make archives of arbitrary large filesystems. Another reason is that tar files bigger than 2GB make (apparently) 'mkisofs' choke. As I use 'mkisofs' and 'cdrecord' to burn the backups to DVD, I need this functionality. Because split is used, the tarfiles are split into multiple chunks according to this principle: 'home-fs.tar' becomes 'home-fs.taraa' 'home-fs.tarab' 'home'fs.tarac' and so on. The locale settings are reset to 'C' in the script's session. This is to ensure that 'sort' works correctly when a tarfile is assembled from it's chunks. Systembackup.sh expects archive names without the 'aa'...'ax' extensions. The script handles that part automatically when backing up, verifying and restoring. If you want compression, change this line: star_options_system=" -c -M -atime -dirmode -fifo -no-statistics \ -silent -dump" to: star_options_system=" -c -M -z -atime -dirmode -fifo -no-statistics \ -silent -dump" A log file is kept in ${installdir}, it's called systembackup.log. All operations and output from 'star' is appended to the file for later perusal. Available script commandline options: ------------------------------------- option: -system: Start backup of entire system as defined in script function 'systembackup()'. Example: systembackup.sh -system ------------------------------------- option: -sysinc: Start incremental backup of entire system as defined in script function 'systembackup()'. Example: systembackup.sh -sysinc ---------------------------------- Option: -list archive '[patterns]': List files in 'archive' according to patterns. See example of patterns below. This example lists the /etc directory and it's contents from the archive named 'root-fs.tar' located in currect directory Example: systembackup.sh -list root-fs.tar 'pattern=etc*' ----------------------------------------------- Option: -restore directory archive ['patterns']: Restore files to 'directory' from 'archive' using optional patterns. The recommended procedure is first to list desired files using the intended pattern for the restore, and after a satisfactory list result do a restore. See example of operation below. The archive to restore from is expected to have a full path from /. This example extracts the /etc directory and it's contents from the archive named '/mnt/backup/root-fs.tar' Example: systembackup.sh -restore / /mnt/backup/root-fs.tar 'pattern=etc*' -------------------------------------------- Option: -diff archive topleveldirInArchive: Compute difference between files in archive and files in file sytem. The diff is based on filesize to make it fast. It is simple what to diff on. At this point one must know the toplevel directory in the archive. It is usually easy to guess from the archive name. This example computes a diff between the files in the archive 'home-fs.tar' and the files in /home. Example: systembackup.sh -diff home-fs.tar /home -------------------------------------------- Option: -version This option makes systembackup.sh print it's cvs version number. Configuration of backup: ------------------------ The function systembackup() is used to define the backup. Each entry is done in three lines like shown below: # / file system rootfile=root_fs_backup_${date}.tar backup ${rootfile} / "-not pattern=proc/* -not pattern=tmp/*" generate_filelist ${rootfile} root_fs.txt verify ${rootfile} / This particular setup takes a backup of my root partition, it does not decent into other partitions because of the '-M' option set in the 'backup-options' and does also not backup '/proc' and '/tmp' directories because of the given patterns. Patterns are surrounded by ", this makes it easier to write a simple pattern because the '*' doesn't need a backslash to escape it. The tarfile and the filelist file are stored in ${archivedir}. The backup is done by making a 'cd' to / and from there making the backup. Directories which shall not be backed up are named without any leading path, as shown in example. See the example definitions in the systembackup() function in the script for further details. The definitions show how to stop a Cyrus mail server, take the backup and restart the mail server. Example patterns: ----------------- List all files in tar archivefile home-pj.tar "./systembackup.sh -list home-pj.tar" List only files from Per's 'src' directory and the 'Documents' directory from the archive file home-pj.tar "./systembackup -list home-pj.tar 'pattern=src* pattern=Documents*'" Restore only files to Per's 'src' and 'Documents' directories "./systembackup -restore /home/pj home-pj.tar 'pattern=src* pattern=Documents*'" List all files in the tar archive home-pj.tar except 'tmp' directory "./systembackup -list home-pj.tar '-not pattern=tmp*'" Builtin tests: ---------------------- The script has builtin functions which tests both full backup and incremental backups. The variable 'testdir' names a directory into which the 'testdirs()' function put a suitable large hierarchy of test data. Customize the 'testdirs()' function to let it set up the test data. How to run the tests 1. cd to systembackup.sh installdir 2. Take a full backup of testhierarchy and restore it to './kurttest' './systembackup.sh -test' 3. Take an incremental of the same hierarchy wich all files touched after the 'level.0' timestamp. This backup should match the above exactly. './systembackup.sh -testinc' 4. Now touch some file in the testhierarchy to show that they will be backed up on next test. For example like this 'find /tmp/systembackuptest/some/test/dir -exec touch {} \;' Now backup the touched files './systembackup.sh -testincpartial' 5. A second partial backup should back up nothing as no files are touched after the last backup. './systembackup.sh -testincpartial' Cron usage: ----------- I have the systembackup.sh script in cron taking both full and incremental backups. The script which drives the automated backup is shown in 'systembackup-cron.sh'. Example cron entry for full backup: 6 4 * * 1 root /usr/sbin/systembackup-cron.sh -system Example cron entry for incremental backups: 6 4 * * 3,5 root /usr/sbin/systembackup-cron.sh -sysinc Contact: -------- Comments and improvements are welcome. Send them to Per Jensen, per@net-es.dk