rsync script for incremental backup which continues unfinished jobs

Question

I have a script which does incremental backups with rsync (script 1). I tried to add the feature to continue unfinished jobs in another script (script 2) but this version does a full copy of the source instead of doing incremental backup. What am I doing wrong? Here are the scripts:

script 1

#!/bin/zsh

readonly VOLUME_NAME="bckp_HD"
readonly SOURCE_DIR="/home/${USER}/srcdir"
readonly BACKUP_DIR="/media/${USER}/${VOLUME_NAME}/rsync_snapshots"
readonly DATETIME="$(date '+%Y_%m_%dT%H_%M_%S')"
readonly BACKUP_PATH="${BACKUP_DIR}/${DATETIME}${SOURCE_DIR}"
readonly LATEST_LINK="${BACKUP_DIR}/latest${SOURCE_DIR}"
readonly LOG_DIR="${HOME}/logs/rsync_log"
readonly LOG_FILE="${LOG_DIR}/${VOLUME_NAME}_${DATETIME}.log"

# Define the excluded paths
EXCLUDES=(
  "--exclude=.cache"
)


# Cleanup function
cleanup() {
  rm -f "$LOCK_FILE"
}

if [[ ! -d "${BACKUP_DIR}" ]]; then
  mkdir -p "${BACKUP_DIR}"
fi

if [[ ! -d "${LOG_DIR}" ]]; then
  mkdir -p "${LOG_DIR}"
fi


rsync -auv --delete --mkpath \
  "${SOURCE_DIR}/" \
  --link-dest "${LATEST_LINK}" \
  "${EXCLUDES[@]}" \
  --log-file="${LOG_FILE}" \
  "${BACKUP_PATH}"


rm -rf "${BACKUP_DIR}/latest"
ln -s "${DATETIME}" "${BACKUP_DIR}/latest"

script 2 (same as script 1 until line 29):

#!/bin/zsh

readonly VOLUME_NAME="bckp_HD"
readonly SOURCE_DIR="/home/${USER}/srcdir"
readonly BACKUP_DIR="/media/${USER}/${VOLUME_NAME}/rsync_snapshots"
readonly DATETIME="$(date '+%Y_%m_%dT%H_%M_%S')"
readonly BACKUP_PATH="${BACKUP_DIR}/${DATETIME}${SOURCE_DIR}"
readonly LATEST_LINK="${BACKUP_DIR}/latest${SOURCE_DIR}"
readonly LOG_DIR="${HOME}/logs/rsync_log"
readonly LOG_FILE="${LOG_DIR}/${VOLUME_NAME}_${DATETIME}.log"
readonly LOCK_FILE="${BACKUP_DIR}/.backup_in_progress"

# Define the excluded paths
EXCLUDES=(
  "--exclude=.cache"
)

# Cleanup function
cleanup() {
  rm -f "$LOCK_FILE"
}

if [[ ! -d "${BACKUP_DIR}" ]]; then
  mkdir -p "${BACKUP_DIR}"
fi

if [[ ! -d "${LOG_DIR}" ]]; then
  mkdir -p "${LOG_DIR}"
fi

# Handle unfinished backup
if [[ -f "${LOCK_FILE}" ]]; then
  echo "Found an unfinished backup. Resuming..."
  UNFINISHED_BACKUP_PATH=$(cat "$LOCK_FILE")

  rsync -auv --progress --mkpath \
    "${SOURCE_DIR}/" \
    --link-dest="${LATEST_LINK}" \
    "${EXCLUDES[@]}" \
    --log-file="${LOG_FILE}" \
    "${UNFINISHED_BACKUP_PATH}"

  cleanup
else
  echo "${BACKUP_PATH}" > "${LOCK_FILE}"

  # If latest snapshot exists, do incremental
  if [[ -L "${LATEST_LINK}" ]]; then
    rsync -auv --progress --mkpath \
      "${SOURCE_DIR}/" \
      --link-dest="${LATEST_LINK}" \
      "${EXCLUDES[@]}" \
      --log-file="${LOG_FILE}" \
      "${BACKUP_PATH}"
  else
    echo "No previous backup found. Running full backup."
    rsync -auv --progress --mkpath \
      "${SOURCE_DIR}/" \
      "${EXCLUDES[@]}" \
      --log-file="${LOG_FILE}" \
      "${BACKUP_PATH}"
  fi
  echo "Clean up the lock file"
  cleanup
fi

# Update the latest symlink
rm -rf "${BACKUP_DIR}/latest"
ln -s "${DATETIME}" "${BACKUP_DIR}/latest"

Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on Unix & Linux Meta, or in Unix & Linux Chat. Comments continuing discussion may be removed. — terdon
– terdon ♦, Commented Aug 15 at 8:41

Chris Davies · Accepted Answer · 2025-08-13 20:11:40Z

3

It looks like your definition of $LATEST_LINK may be wrong.

Assuming your shell is bash, then converting all the action commands (rm, ln, mkdir, echo… >…, rsync) into no-ops and running the result with bash -x it becomes clear that this condition:

[[ -L "${LATEST_LINK}" ]]

is executed as

[[ -L /media/user/bckp_HD/rsync_snapshots/latest/home/user/srcdir ]]

whereas I suspect it should be one of these:

[[ -L /media/user/bckp_HD/rsync_snapshots/latest ]]
[[ -d /media/user/bckp_HD/rsync_snapshots/latest/srcdir ]]
[[ -d /media/user/bckp_HD/rsync_snapshots/latest/home/user/srcdir ]]

To convert an action command into a no-op I might use something like this:

LOG() { printf '! %s !\n' "$*" >&2; }

# Later in the code, prefix all actions with "LOG"
LOG rsync…
LOG rm…

# Quote redirection operators
LOG echo something '>' a_file

edited Aug 13 at 20:11

answered Aug 13 at 8:18

Chris Davies

128k16 gold badges179 silver badges324 bronze badges

Thank you for your answer @Chris. Actually I would like to LATEST_LINK to be executed as [[ -L /media/user/bckp_HD/rsync_snapshots/latest/home/user/srcdir ]] as I want to retain the full path inside the backup. The first script works well with that definition (however without "continue backup feature"). So I think I would not change the definition of LATEST_LINK. What do you think?

ecjb
– ecjb

2025-08-13 08:47:47 +00:00
Commented Aug 13 at 8:47
1

That target isn't a link so the -L will fail. You could use -d

Chris Davies
– Chris Davies

2025-08-13 17:06:31 +00:00
Commented Aug 13 at 17:06

Add a comment |

Stack Exchange Network

rsync script for incremental backup which continues unfinished jobs

1 Answer 1

You must log in to answer this question.

Hot Network Questions

rsync script for incremental backup which continues unfinished jobs

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions