How to verify copy process

Yesterday, I left long running copy process between two external USB hard-drives. I did not want to assume anything, so I created simple shell script to make sure that whole process went smoothly.

The idea description

The idea is to recursively compute file checksums in the source directory.

$ cd /source/directory
$ find . -type f -exec sha256sum '{}' \; > /tmp/checksums.file

Copy required data.

$ cp -r /source/directory /destination/directory

Verify file checksums in the destination directory.

$ cd /destination/directory
$ cat /tmp/checksums.file | sha256sum -c

Shell script

The following shell script can divided into four distinct sections: compute checksums, copy data, verify checksums and send mail message.

# simple "copy [directories] and verify [files]" shell script
# Sample usage: /from_directory /to_directory

# used commands
find_command=$(which find)
shasum_command=$(which sha256sum)
cat_command=$(which cat)
unlink_command=$(which unlink)

# copy command with additional arguments
copy_command=$(which cp)
copy_arguments="-rp" # recursive mode
                     # preserve mode, ownership, timestamps

# mail command and with used email address
mail_command=$(which mail)

if [ -d "$1" -a ! -d "$2" ]; then
  # first  directory          exists
  # second directory does not exists

  # compute 256-bit checksums
  (cd $1 && $find_command . -type f -exec $shasum_command '{}' \; > $shasum_log)

  # copy data
  $copy_command $copy_arguments "$1" "$2" > $copy_log

  # verify computed checksums
  (cd $2 && $cat_command $shasum_log | $shasum_command -c > $verify_log)

  # prepare message and send mail message
  if [ "$shasum_exit_code" -eq "0" ]; then
    mail_subject="Subject: ${0}: Success"
    mail_subject="Subject: ${0}: Error"
  echo                                     >  $mail_file
  echo "Command-line: ${0} ${1} ${2}"      >> $mail_file

  if [ -s "$copy_log" ]; then
    echo                                   >> $mail_file
    echo "Copy process"                    >> $mail_file
    $cat_command $copy_log                 >> $mail_file

  if [ "$shasum_exit_code" -ne "0" ]; then
    echo                                   >> $mail_file
    echo "Verify process"                  >> $mail_file
    $cat_command $verify_log | grep -v OK$ >> $mail_file

  $mail_command $mail_subject_argument "${mail_subject}" $mail_address < $mail_file

  # cleanup temporary files
  $unlink_command $mail_file
  $unlink_command $verify_log
  $unlink_command $copy_log
  $unlink_command $shasum_log
  echo "Problem with parameters\nCommand-line: ${0} ${1} ${2}" | $mail_command $mail_subject_argument "${0}" $mail_address
  exit 5

I have omitted checking mktemp exit code, but it can be easily added using similar code.

# compute 256-bit checksums
if [ "$?" -ne "0" ]; then
  echo "Cannot create temporary file" | $mail_command $mail_subject_argument "${0}" $mail_address
  exit 1
(cd $1 && $find_command . -type f -exec $shasum_command '{}' \; > $shasum_log)

The whole idea is quite simple, but you are not limited to it, as you can use any available file integrity checker.

Update - 18.02.2015

Thanks to Fernand Ka I have updated shell script to grep through the right file.

Please notice that locale settings take important role here as I am using grep command to exclude lines ending with OK string.

Milosz Galazka's Picture

About Milosz Galazka

Milosz is a system administrator working for a successful Polish company and a long time supporter of Free Software Foundation and Debian operating system.

Gdansk, Poland