Page Comparison

Rsync is a data transfer tool. To use rsync you will be provide it with a source, and a destination. For each file in the source, rsync will first look to see if the file exists in the target. If not, then rsync will transfer a copy of the file to the target. If file does exist on the target, then rsync confirms that the files are the same. If not, then it determines the smallest amount of data that needs to be transferred to make the target file look like the source file. This makes rsync ideal for mirroring sets of files. Because rsync ignores files that are up to date it's also useful for resuming a transfer that might have gotten interrupted. However, this powerful piece of software can easily be used to create a mess, or damage files. If you wish to use rsync you should carefully read through this document to make sure you understand its proper use. It does also require a bit more familiarity with how files are stored, and accessed in UNIX. If you are still not comfortable with this, you should visit our New Users Guide which has some references for learning to work with UNIX. You might also refer to our UNIXary which defines some commonly used terms.

JUST TO SAY IT AGAIN, YOU MUST THINK WHILE/BEFORE USING THIS TOOL, AND MAKE SURE YOU ARE SYNCING THE RIGHT THINGS.

UNIX users (and Mac by use of Terminal.app) can use rsync by typing

...

will sync object a to object b where both a and b are in your current working directory. By object, it is meant either a file or a directory. For example,

rsync a b

will sync a to bUsing the syntax above src = a, tgt = b, and opts = "" (ie, we are not providing any additional options to modify the behavior of rsync). If a, and b are both files, then b is updated to look like a. If b is a directory, then b/a is updated to look like a. Keep in mind that "updateupdated" means "created" if it doesn't exist. If a is a directory, then rsync will fail because, by default, rsync will not transfer directories. You can get around this by using the -a option. In other words,

...

object meaning
a b sync object (file or directory) a to (file or directory b)
a/ b sync contents of directory a to b
a b/ sync a to directory b
a/ b/ sync directory a/ to b/

As before either a or b can be a short name which means it refers to something in your current working directory, or it can

...

Syncing to Another Directory or a Remote Host

Until now we've just assumed that we are syncing files in the same directory. However, rsync will also sync files between directories. For example, if you have modified files in your home directory, you might want to make sure the changed files are in your personal directory in your lab folder. In this case you could run,

...

which creates or updates /share/FooLab/bar1234/ProjectFiles/a. You can reverse this off course. Finally, of course, and use rsync to update date your current working directory with source that is in another directory. rsync allows either the source or the target on a remote host. This allows you to sync files to a remote host , using the syntax

rsync -a a remote_host:dir

...

and read about it. Also, --dry-run, --list-only, and --itemize-changes are good ways to see what rsync is doing, or to test what it might do. Finally, sometimes on remote systems

Rsync as a Backup Tool

This section will provide a mechanism for CCBB users to easily back their Mac or Linux laptops up. First, prep by logging into files.ccbb.utexas.edu using ssh, and then changing to your lab's bulk share. Please do not use home directories for this, as we do not have space to do this. Anybody that we have obligations to backup will also be a member of a lab with a lab share. Within your lab share you should have a personal folder which only you, and your lab head can access. Make a unique for your backup (say LAPTOP). Next on your system run the commands

mkdir ~/bin
mkidr ~/.ccbb-rsync-backup

Go into the directory ~/bin, and use an editor to create file called sync2files which contains the blocks of text between the -begin- and -end- lines below

– begin –

– end –

Once the script is done, run the command

chmod u+x sync2files

Next go into ~/.ccbb-rsync-backup, and create a file called include, and a file called exclude. This can be empty, but over time you may want to adjust the contents of these files. For example, you may notice cache files in your home directory, or other files that you don't want backed up (such as your music collection which is non-work related). Those can be put in the exclude file. Or if you wish to only back up specific items, or need to back up items other than ~, you can add entries to the include fileCURRENTLY THIS SCRIPT IS KNOWN TO WORK ON MAC. IF YOU HAVE PROBLEMS ON OTHER SYSTEMS, LET US KNOW.

Also, before syncing the contents of your laptop onto your lab share, you should consult with your PI, and make sure that this is an acceptable use of the space they purchased.

CCBB users in the Bull, Cannatella, Chen, Hillis, and Hofmann labs can take advantage of this to keep their Linux or Mac laptops up to date. To do this first go into your lab's folder /share/PILab (replace PI with the name of your lab's head). In this folder, should be another folder with your username. This folder is accessible only by you, and by your PI. Once you are in your personal folder, make a directory called SYNC2FILES, and go into it. Finally, make a folder for the device that you want back up (eg, LAPTOP). Please refer to the files server portion of our docs for info on how to do this.

Next go to your home directory, and download our backup script Rsync backup script. Run the command,

tar xvf ccbb-rsync-backup.tar

which will extract the backup script, and create the following

~bin/sync2files
~/ccbb-rsync-backup/exclude
~/ccbb-rsync-backup/include
~/ccbb-rsync-backup/sync2files

(and others, but these are the ones that you will actually care about). First, use a text editor to edit ~/ccbb-rsync-backups/sync2files. At the top, you should provide your lab head, and your eid. Also, provide the folder name that you created above. These 3 things ensure that

Some caveats:

• This script is not 100% tested, so you will need to make sure that it is actually working. If there are problems, please mention them so that we can make fixes.

• This is not necessary if you are only using a cluster account, or another UNIX server that we maintain, because these are already being backed up. This script is meant only for people who want easy backups of their laptops without having to worry about dragging, and dropping specific files to the server.

• Much more needs to be added to the exclude file. For sure, we do not have the space for you to back up your personal music collection, iPhoto library, and other non-work related files. Also, we cannot store security sensitive files (see our ISORA page for more info), and in fact you should not be storing them on your laptop or desktop anyways. If someone can send us information as to where things like iTunes store their files, or where other files that can be safely ignore are stored, then we can make this process better by making OS specific exclude files.

Once you have your files ready to go, then you can type

~/bin/sync2files

provide your password.