Rsync Include and Exclude Rules (Automated Incremental Backups with Rsync and Cron Part 1a)

When deciding what you want to backup, you may want find that you only want to backup certain parts of a directory. For example, you may want to backup your pictures, documents, etc. in your home folder but not temp files, config files, etc. No problem. In rsync, this is accomplished by using include and exclude rules.

If you don’t want to save everything in your /directory/to/copy/, you will most definitely want to use an exclude file, and you may use an include file as well. The reason you may not use an include file is that include may not do exactly what you think it will. For instance, if you only want to save your pictures and documents in your home folder, you can’t just tell rsync to include pictures and documents. This is because everything in /directory/to/copy/ is automatically included the backup. Therefore, if you tell rsync to backup your home folder and then also tell it to include pictures and documents, it will still backup your entire home folder. Include rules are not worthless, however, as they can override exclude rules. If this doesn’t make sense now, don’t worry the following examples will explain the weirdness more concretely.

Exclude Rules

Let’s start by excluding what we don’t want. If there are only a couple of things you want to exclude, you can list them right in your rysnc command. However, it will look cleaner, especially if you are excluding many things, to create an exclude file. To create your exclude file start by creating a text file and saving it wherever you stored your rsync script. I will use the name rsyncExclude, but you can use whatever you want. Inside the file, you simply list each file or directory you want to ignore by putting one item on each line. You can put comments in the file to help with organization by starting a line with the pound sign (#). The following is an example exclude file based on the one I use when backing up my home folder:

#Directories
/Desktop
/Downloads
/Public

#Firefox
.mozilla/*
.mozilla/firefox/*
.mozilla/firefox/*.default/*

You can view the file in its entirety on my GitHub.

This list tells rsync to exclude my Desktop, Downloads and Public folders as well as everything inside .mozilla, .mozilla/firefox, etc. That is what the ‘*’ does. It is a wild card. When used by itself it will cover everything in the specified directory.

Your rsync command will look something like this if you use an exclude file:

rsync -av --exclude-from='path/to/rsyncExclude' /directory/to/copy/ /destination/directory

If you are not going to use an exclude file, because you only want to exclude one (or a couple things), your command will look something like this:

rsync -av --exclude='path/to/fileOrDirectory/toExclude' /directory/to/copy/ /destination/directory

Multiple files or directories can be excluded by separating the paths with a comma.

Include Rules

It may seem strange that I told rsync to ignore everything inside .mozilla and then everything inside the firefox folder that is in .mozilla etc. Why not just do what I did for my Downloads folder and put .mozilla on its own line and be done with it? The answer has to do with the funky way that include and exclude rules interact. Rsync follows whatever rule it matches first. If it finds an include rule prior to an exclude rule, it will include what it is told to include, even if it subsequently comes across a matching exclusion rule. Therefore, if you put your include rules into the rsync command prior to your exclude rules, they will override the exclude rules. Since rysnc is not smart enough to include just the file(s) we specify in our include argument and exclude everything else, this is how you have to go about backing up only part of a directory. (E.g., I only want to backup the bookmarkbackups folder found in .mozilla and nothing else.)

Take a minute to look at this sample include file and compare it to the exclude file above. Then the following explanation will make more sense.

.mozilla/
.mozilla/firefox/
.mozilla/firefox/*.default/
.mozilla/firefox/*.default/bookmarkbackups/

Your rysnc command will look something like this when using both include and exclude files:

rsync -av --include-from='path/to/rsyncInclude' --exclude-from='path/to/rsyncExclude' /directory/to/copy/ /destination/directory

Were you able to figure out what’s going on with the include and exclude statements? Remember that if a match is found in the include rules, any matching exclude rule that rsync sees later will be ignored. Here is a rundown of what happens at each level when rsync comes across the .mozilla folder and its contents:

  1. include: .mozilla directory itself
  2. include: .mozilla/firefox directory
  3. exclude: everything else in .mozilla directory based on .mozilla/* in exclude file
  4. include: .mozilla/firefox/*.default directory
  5. exclude: everything else in .mozilla/firefox directory based on .mozilla/firefox/* in exclude file
  6. include: .mozilla/firefox*.default/bookmarkbackups/ and its contents
  7. exclude: everything else in .mozilla/firefox/*.default based on .mozilla/firefox/*.default/* in exclude file

A Quick Note About Syncing

If you are currently using the “–delete” argument in your rsync command, you can do one better when using an exclude file. Using “–delete-excluded” directs rsync to not only delete any files that are no longer in /directory/to/copy but also files in your exclude list. This way if you add more files or directories to your exclude list, rsync can automatically remove them from your backup.

This should get you started with using exclude and include rules for rsync. If you read this article as part of my tutorial series on automated, incremental backups, you are now ready for part 2. If you have a question or other feedback, leave it in the comments.

comments powered by Disqus