Pour réaliser le backup d'un dossier vers un dossier d'une autre machine il faut :
Ce script n'est rien d'autre qu'un wrapper de rsync/ssh. Il peut être appelé directement, mais il est plutôt conseillé de l'appeler depuis un script spécifique qui fixe tous les paramètres (cela facilite la mise au point). Voir exemple plus bas.
#!/bin/bash # sshDirSynchro.sh # # Backup directory(ies) from/to local directory to/from remote directory using rsync/ssh # It is just an Rsync wrapper. # # V0.1 - 2014-08-17 # V0.2 - 2014-08-18 # Suppression du rapport de syncho intégré # Suppression de messages # V0.3 - 2014-08-18 # l'option "--delete" de rsync ne doit pas être positionnée par défaut (trop dangereux). # l'option "--delete" est ajoutée en parametre # l'option "--chmod" est ajoutée en parametre : elle emble nécessaire pour avoir des droits corrects # lors d'une copie depuis Windows (on fixe les droits chmod=u+rwx). TIME_START_1970=$(date +%s) #------------------------------------------------------------- # Define usage #------------------------------------------------------------- usage() { cat <<-'_usage_statement_' sshDirSynchro.sh: synchronizes a local directory and a distant directory using rsync/ssh. Usage: sshDirSynchro.sh --backup|--restore -ld <localDirectory> -rh <remotHost> -ru <remoteUser> -rd <remoteDirectory> [ -h -p <sshPort> -bwl <bandWidthLimit> -ksd <killSleepDelay> -lf <logFile> --delete --chmod --inconv --check] ] Mandatory parameters: --backup copy from local to remote --restore copy from remote to local -ld, --local-dir Local directory from where the data are copied -rh, --remote-host Remote host to where the data are copied -ru, --remote-user Authorized user for the remote host -rd, --remote-dir Remote directory to where the data are copied Optionnal parameters: -h, --help Display usage information. -p, --ssh-port Default port is 22. -bwl, --bandwidth-limit Embeded "--bwlimit" for rsync -ksd, --kill-sleep-delay Permit to kill the script after a user defined delay. Exemple: -ksd "500s" for 500 seconds or -ksd "5h" for 5 hour. See man sleep for delay format. -lf, --log-file If no log file is set, the report is writen on stdout. If -lf is set a log file is created for the report. --delete rsync delete extraneous files from dest dirs --chmod ensure chmod=u+rwx on remote host --inconv convert chrset from Windows to Linux --check compare local and remote dir content *** Caution: This script use several ssh commands in batch mode. To avoid password asking ssh keys must be installed. See man ssh-keygen _usage_statement_ } #------------------------------------------------------------- # Analyse les arguments de la ligne de commande. #------------------------------------------------------------- SSH_PORT=22 # echo -e "SYNC_OPTION: ${SYNC_OPTION} - $#" while [[ $# > 0 ]] do # echo -e "$# - OPTION: $1" case $1 in --backup) if [ "${SYNC_OPTION}" ] then echo -e "Error: You must choose between --backup OR --restore!" exit 1 fi SYNC_OPTION="backup" shift ;; --restore) if [ "${SYNC_OPTION}" ] then echo -e "Error: You must choose between --backup OR --restore!" exit 1 fi SYNC_OPTION="restore" shift ;; -ld|--local-dir) LOCAL_WORKING_DIR=$2; shift 2 ;; -rh|--remote-host) REMOTE_HOST=$2 shift 2 ;; -ru|--remote-user) REMOTE_USER=$2 shift 2 ;; -rd|--remote-dir) REMOTE_WORKING_DIR=$2 shift 2 ;; -p|--ssh-port) SSH_PORT=$2 shift 2 ;; -bwl|--bandwidth-limit) BAND_WIDTH_LIMIT=$2 shift 2 ;; -ksd|--kill-sleep-delay) KILL_SLEEP_DELAY=$2 shift 2 ;; -lf|--log-file) LOGFILE="`basename $0`.`date +%Y\-%m\-%d\-%Hh%Mmn%S`.log" shift ;; --delete) DELETE_OPTION="--delete-before" shift ;; --chmod) CHMOD_OPTION="--chmod=u=rwX" shift ;; --iconv) ICONV_OPTION="--iconv=ISO-8859-1,utf-8 --protect-args" shift ;; --check) CHECK=$1 shift ;; -h|--help) usage exit 0 ;; *) # Unknown option echo -e "Error : Unknown option $1\n" usage exit -1 ;; esac done [ "${SYNC_OPTION}" ] || echo -e "\nError: --backup or --restore must be specified!" [ "${LOCAL_WORKING_DIR}" ] || echo -e "\nError --local-dir must be defined!!" [ ${REMOTE_HOST} ] || echo -e "\nError REMOTE_HOST must be defined!!" [ ${REMOTE_USER} ] || echo -e "\nError REMOTE_USER must be defined!!" [ ${REMOTE_WORKING_DIR} ] || echo -e "\nError REMOTE_WORKING_DIR must be defined!!" if [ ! "${LOCAL_WORKING_DIR}" ] || [ ! "${REMOTE_HOST}" ] || [ ! "${REMOTE_USER}" ] || [ ! "${REMOTE_WORKING_DIR}" ] then usage exit -1 fi #------------------------------------------------------------- # Create a custom logger #------------------------------------------------------------ if [ ${LOGFILE} ] then logprint() { #echo -e "$(date +%T): $*" >> $LOGFILE echo -e "$*" >> $LOGFILE } else logprint() { #echo -e "$(date +%T): $*" echo -e "$*" } fi #------------------------------------------------------------- # Test local diretory is ok #------------------------------------------------------------- # if [ ! -d ${LOCAL_WORKING_DIR} ] # then # logprint "ERROR: ${LOCAL_WORKING_DIR} does not exist!"; # exit -1 # fi #------------------------------------------------------------- # Test remote host is ok #------------------------------------------------------------- if [ $LOGFILE ] then echo -e "Resulats dans `pwd`/$LOGFILE" fi # if ( ! ssh $REMOTE_USER@$REMOTE_HOST -p $SSH_PORT "pwd" > /dev/null 2>&1 ) # then # logprint "ERROR: $REMOTE_HOST unreachable!" # exit -1 # fi # if ( ! ssh $REMOTE_USER@$REMOTE_HOST -p $SSH_PORT "cd $REMOTE_WORKING_DIR" > /dev/null 2>&1 ) # then # logprint "ERROR: $REMOTE_HOST/$REMOTE_WORKING_DIR unreachable!" # exit -1 # fi #------------------------------------------------------------- # Lancement du kill de ce processus en tâche de fond # pour limiter la durée d'execution (au cas où cela déborderait) #------------------------------------------------------------- # Attention: # la commande rsync s'execute un dans un process fils # l'utilisation du ssh crée un nouveau processus fils du rsync. # il faut donc faire un "pkill -P $$" pour tuer l'arborescence des processus. if [ ${KILL_SLEEP_DELAY} ] then logprint "Ce script stoppera dans $KILL_SLEEP_DELAY" sleep $KILL_SLEEP_DELAY \ && logprint \ && logprint "<<STOP SYNCHRO>> Le processus est tué car il dépasse le temps imparti" \ && logprint \ && pkill -P $$ & # variante d'écriture : (sleep 6s; kill $$ ) & # logprint "PID processus kill : $! - delay: $SLEEP_DELAY" # On stocke le PID de process lancé en backgound pour pouvoir le tuer KILL_ME_PID=$! fi #------------------------------------------------------------- # Recupération sur erreur #------------------------------------------------------------- function traitement_erreur { kill $KILL_ME_PID > /dev/null 2>&1 logprint logprint "---------------------" logprint ">>> Erreur $? >>>>" logprint "---------------------" logprint # remote_report exit -1 } trap traitement_erreur SIGHUP SIGINT SIGTERM SIGKILL #------------------------------------------------------------- # Set rsync options #------------------------------------------------------------ # options rsync: # -a, --archive mode archivage; identique à -rlptgoD (pas -H) # -v, --verbose plus loquace # -S, --sparse traite les fichiers à trous efficacement # -P équivalent à --partial --progress # --progress montre l'avancement pendant le transfert # --partial conserve les fichiers partiellement transférés # --delete delete extraneous files from dest dirs # --delete-before receiver deletes before xfer, not during # --no-p disables permissions copying # --no-g disables group copying # --chmod=ugo=rwX ensures that all non-masked bits get enabled # --delete-excluded also delete excluded files from dest dirs # --exclude=PATTERN exclude files matching PATTERN # -s, --protect-args no space-splitting; wildcard chars only # --iconv=ISO-8859-1,utf-8 de Windows vers Linux # RSYNC_OPTIONS=" -avS --delete-before --bwlimit=$BAND_WIDTH_LIMIT -e \"ssh -p $SSH_PORT \" " # le -e \"ssh -p $SSH_PORT \" pose problème dans RSYNC_OPTIONS # le --chmod=u=rwX semble nécessaire pour le Raspi RSYNC_OPTIONS="-avS --no-p --no-g --stats" [ ${CHMOD_OPTION} ] && RSYNC_OPTIONS="${RSYNC_OPTIONS} ${CHMOD_OPTION}" [ ${DELETE_OPTION} ] && RSYNC_OPTIONS="${RSYNC_OPTIONS} ${DELETE_OPTION}" [ "${ICONV_OPTION}" ] && RSYNC_OPTIONS="${RSYNC_OPTIONS} ${ICONV_OPTION}" [ ${BAND_WIDTH_LIMIT} ] && RSYNC_OPTIONS="${RSYNC_OPTIONS} --bwlimit=$BAND_WIDTH_LIMIT" #LOGFILE="`basename $0`.`date +%Y\-%m\-%d\-%Hh%Mmn%S`.log" #logprint "------------------------------------------------------" logprint "Synchronisation" if [[ ${SYNC_OPTION} = "backup" ]] then logprint " de `hostname` ${LOCAL_WORKING_DIR}" logprint " vers $REMOTE_USER@$REMOTE_HOST:$SSH_PORT $REMOTE_WORKING_DIR" fi if [[ ${SYNC_OPTION} = "restore" ]] then logprint " De $REMOTE_USER@$REMOTE_HOST:$SSH_PORT $REMOTE_WORKING_DIR" logprint " Vers `hostname` ${LOCAL_WORKING_DIR}" fi logprint " bwlimit=$BAND_WIDTH_LIMIT, delay max=$KILL_SLEEP_DELAY" logprint " rsync options: ${RSYNC_OPTIONS}" #------------------------------------------------------------- # Build final command #------------------------------------------------------------- RSYNC_COMMAND="rsync $RSYNC_OPTIONS -e \"ssh -p $SSH_PORT\"" [ "${SYNC_OPTION}" = "backup" ] && RSYNC_COMMAND="${RSYNC_COMMAND} \"${LOCAL_WORKING_DIR}\" $REMOTE_USER@$REMOTE_HOST:\"$REMOTE_WORKING_DIR\"" [ "${SYNC_OPTION}" = "restore" ] && RSYNC_COMMAND="${RSYNC_COMMAND} $REMOTE_USER@$REMOTE_HOST:\"$REMOTE_WORKING_DIR\" \"${LOCAL_WORKING_DIR}\" " [ ${LOGFILE} ] && RSYNC_COMMAND="${RSYNC_COMMAND} >> $LOGFILE 2>&1" logprint "\n$RSYNC_COMMAND" # !!!! eval introduces potential vulnerabilities and unexpected behavior situations so must be used !!! eval ${RSYNC_COMMAND} || { logprint logprint "**************************************" logprint "\n\n Erreur sur RSYNC: $?\n\n" logprint "**************************************" exit -2 } [ "${CHECK}" ] && compareLocalAndRemoteDirs.sh -ld "${LOCAL_WORKING_DIR}" -rd "$REMOTE_WORKING_DIR" -rh $REMOTE_HOST -ru $REMOTE_USER -p $SSH_PORT #------------------------------------------------------------- # On tue le process de kill (qu'il existe ou pas) # afin qu'il n'interrompe pas rapport d'execution #------------------------------------------------------------- if [ ${KILL_SLEEP_DELAY} ] then # logprint "*** processus de kill $! tué" kill $KILL_ME_PID > /dev/null 2>&1 fi #------------------------------------------------------------- # fin des opération #------------------------------------------------------------- TIME_END_1970=$(date +%s) logprint logprint "******** $(date +%T) - $(hostname): Fin de $(basename $0) OK! ($(convertDuration.sh $(expr $TIME_END_1970 - $TIME_START_1970))) **********"
Ce script compare des repertoires qui euvent être locaux et/ou distants. Voir exemple plus bas.
#!/bin/bash # # Compare the content of a local and a remote directories # # V0.1 - 2014-11-23 # Missing: SSH port management #------------------------------------------------------------- # Define usage #------------------------------------------------------------- usage() { cat <<-'_usage_statement_' compareDirs.sh compares the top level content of 2 local or remote directories using du. Usage: compareDirs.sh [[user@]host1:]dir1 [[user@]host2:]dir2 Note: SSH port management is missing. _usage_statement_ } #------------------------------------------------------------- # Analyse les arguments de la ligne de commande. #------------------------------------------------------------- echo -e "\n$(basename $0) $@ \n" [[ $# != 2 ]] && { usage exit -1 } function getDirFromParam { idx=`expr index "$1" ':' ` if [ $idx == 0 ] then echo $1 else echo ${1:$idx} fi } function getHostFromParam { idx=`expr index "$1" ':' ` if [ $idx != 0 ] then ((idx--)) echo ${1:0:$idx} fi } function checkHostAndDir { dir=$1 host=$2 if [ $host ] then # echo "$host - $dir --> REMOTE" if ( ! ssh $host "pwd" > /dev/null 2>&1 ) then echo "ERROR: $host unreachable!" return 1 elif ( ! ssh $host "cd $dir" > /dev/null 2>&1 ) then echo "ERROR: $dir unreachable on $host!" return 2 fi else # echo "$dir --> Local" [ -d "${dir}" ] || { echo -e "Error $dir must be a directory!" return 3 } fi return 0 } dir1=$(getDirFromParam $1) host1=$(getHostFromParam $1) dir2=$(getDirFromParam $2) host2=$(getHostFromParam $2) # echo -e "dir1 : $dir1 - host1 : $host1\ndir2 : $dir2 - host2 : $host2" checkHostAndDir "$dir1" $host1 [ $? -eq 0 ] || exit -1 checkHostAndDir "$dir2" $host2 [ $? -eq 0 ] || exit -1 #------------------------------------------------------------- # VARS #------------------------------------------------------------- tmp_report_1="/tmp/`basename $0`-local-`date +%F-%T`.log" tmp_report_2="/tmp/`basename $0`-remote-`date +%F-%T`.log" #------------------------------------------------------------- # Recupération sur erreur #------------------------------------------------------------- function traitement_erreur { echo echo "---------------------" echo ">>> Erreur $? >>>>" echo "---------------------" echo rm -f $tmp_report_1 $tmp_report_2 exit 1 } trap traitement_erreur SIGHUP SIGINT SIGTERM SIGKILL #------------------------------------------------------------- # fonction de bilan de synchro # déclarée ici pour pouvoir être utilisée par le traitement d'erreur qui suit #------------------------------------------------------------- function local_report { function numFormat { echo $1 | sed ':a;s/\B[0-9]\{3\}\>/.&/;ta' } function stats_folder() { # Attention : le nom du dossier peut contenir des blancs # il faut donc le traiter entre "" local folder="$1" local items=`find "$folder" -iname '*' | wc -l` local taille=0 echo -e " $folder" # echo -e " $(basename $folder)" if [ $items -gt 0 ]; then taille=$(du -bc "$folder" | grep total | sed -e "s/[ \t]*total/ /g") taille_f=$(numFormat $taille) items_f=$(numFormat $items) echo -e " $items_f items, $taille_f octets" else echo vide fi } local curdir=$(pwd) local dir="$(readlink -f "$1")" echo -e "LOCAL report:" echo -e "$dir" cd "$dir" for f in * ; do # Attention : le nom du dossier peut contenir des blancs # il faut donc le passer entre "" #echo "---f: $f" [ -d "$f" ] && stats_folder "$f" done echo -e "Total: " stats_folder "$dir" # echo "--- pwd: $(pwd) " cd "$curdir" echo } function remote_report { local dir="$1" local host="$2" ssh $host " function numFormat { echo \$1 | sed ':a;s/\\B[0-9]\\{3\}\\>/.&/;ta' } function stats_folder() { # Attention : le nom du dossier peut contenir des blancs # il faut donc le traiter entre \"\" local folder=\"\$1\" local items=\`find \"\$folder\" -iname '*' | wc -l\` local taille=0 echo -e \" \$folder\" if test \$items -ne 0; then taille=\$(du -bc \"\$folder\" | grep total | sed -e \"s/[ \t]*total/ /g\") taille_f=\$(numFormat \$taille) items_f=\$(numFormat \$items) echo -e \" \$items_f items, \$taille_f octets\" else echo vide fi } dir=$(readlink -f "$dir") echo \"REMOTE report: \" echo $dir cd $dir for folder in * ; do # Attention : le nom du dossier peut contenir des blancs # il faut donc le passer entre \"\" [ -d \"\$folder\" ] && stats_folder \"\$folder\" done echo -e \"Total: \" stats_folder \"$dir\" echo " } echo # set -x if [ $host1 ] then remote_report "$dir1" $host1 >> $tmp_report_1 else local_report "$dir1" >> $tmp_report_1 fi if [ $host2 ] then remote_report "$dir2" $host2 >> $tmp_report_2 else local_report "$dir2" >> $tmp_report_2 fi sdiff -atw 140 $tmp_report_1 $tmp_report_2 rm -f $tmp_report_1 $tmp_report_2
#!/bin/bash
N73SM="n73sm"
RASPI="raspi"
BUREAU="HAL"
DIR_SCRIPTS_BUREAU="/home/Roge/scripts/"
DIR_SCRIPTS_N73SM="/media/hd2/scripts/"
DIR_SCRIPTS_RASPI="/media/hd1/roge/scripts/"
case `hostname` in
$N73SM)
# Scripts : N73SM --> Raspi
echo -e "Backup Scripts : N73SM --> Raspi"
./sshDirSynchro.sh --backup --check -rh $RASPI -ru roge \
-rd $DIR_SCRIPTS_RASPI \
-ld $DIR_SCRIPTS_N73SM
;;
$BUREAU)
# Scripts : Bureau --> N73SM
echo -e "Backup Scripts : Bureau --> N73SM"
./sshDirSynchro.sh --backup --check --delete -rh $N73SM -ru roge \
-rd $DIR_SCRIPTS_N73SM \
-ld $DIR_SCRIPTS_BUREAU
# execution sur N73SM pour alimenter le Raspi
ssh roge@n73sm "cd /media/hd2/scripts; ./$(basename $0)"
;;
*) # Unknown option
echo "ERROR: `basename $0` is not authorized to run on `hostname` host!"
exit -1
;;
esac
Le rapport suivant est produit par le script ci-dessus :
$ ./bkpScripts.sh Backup Scripts : Bureau --> N73SM 09:18:47: ------------------------------------------------------ 09:18:47: Synchronisation 09:18:47: de HAL /home/Roge/scripts/ 09:18:47: vers roge@n73sm:22 /media/hd2/scripts/ 09:18:47: bwlimit=, delay max= 09:18:47: rsync options: -avS --no-p --no-g --delete-before 09:18:47: ------------------------------------------------------ 09:18:47: 09:18:47: >> rsync -avS --no-p --no-g --delete-before -e "ssh -p 22" "/home/Roge/scripts/" roge@n73sm:"/media/hd2/scripts/" building file list ... done deleting sshDirectorySynchro.sh ./ archive/ archive/sshDirectorySynchro.sh sent 1429 bytes received 121 bytes 3100.00 bytes/sec total size is 175567 speedup is 113.27 09:18:47: ******** Fin du traitment OK! ********** 09:18:47: -------------- -------------- LOCAL report : | REMOTE report : -------------- -------------- LOCAL_WORKING_DIR: /home/Roge/scripts/ | REMOTE_WORKING_DIR: /media/hd2/scripts/ - archive: 21 fichiers, 133K | - archive: 21 fichiers, 120K - exemples: 6 fichiers, 24K - exemples: 6 fichiers, 24K - restore: 18 fichiers, 96K - restore: 18 fichiers, 96K - test: 1 fichiers, 4,0K - test: 1 fichiers, 4,0K -------------------------- -------------------------- Total: Total: - .: 57 fichiers, 329K | - .: 57 fichiers, 312K
Tester websync