Thursday, February 28, 2008

Rsync and Wget

This tool helps you to copy files or directories between a local and a remote host.

rsync attempts to identify differences between source and destination files prior to initiating a transfer, and (assuming differences exist) tries only to copy the changes, rather than the entire file.

It woks like

On the same machine

rsync -a sourceDir/ destinationDir/
rsync care about the trailing slash, but only on the source argument

say dir1 and dir2 are two directories and file1 is the file that reside under dir1

rsync -a dir1 dir2 ==> will give the output as dir2/dir1/file1
rsync -a dir1/ dir2 ==> will give the output as dir2/file1

rsync -a --delete sourceDir/ destinationDir/ ==> will delete extra file from destinationDir which does not have presence in sourcedir

For Copying data by Rsync, it should be installed on both hosts.

Example:- How to copy data between localhost and remote host

You are sitting on localhost machine where you want to copy data from a remote machine

rsync -avz -e ssh userName@remoteHost:/remote/outgoingData/ /local/incomingData

here "/" after the outgoing data means that the localhost already have the directory and you just just want the data from the outgoingData directory.

If we remove the "/" then it means that you want to create the outgoingData directory first on
the remote machine then want to copy data from it. It should be like below:-

rsync -avz -e ssh userName@remoteHost:/fullPath/outgoingData/ /fullPath/incomingData/outgoingData

Mirroring the websites:-
you are giving the below command from (you want the backup to this server)
rsync -avz -e ssh /var/www/

rsync -az -e ssh --delete ~/public_html/'~/public_html' (Mirror web site (using compression and encryption))

rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ (Synchronize current directory with remote one(

wget -c [Continue downloading a partially downloaded file ]
wget -r -nd -np -l1 -A '*.jpg' [Download a set of files to the current directory ]
wget ftp://remote/file[1-9].iso/ [FTP supports globbing directly ]
wget -q -O- | grep 'a href' | head [Process output directly ]
echo 'wget url' | at 01:00 [Download url at 1AM to current dir]
wget --limit-rate=20k url [Do a low priority download (limit to 20KB/s in this case) ]
wget -nv --spider --force-html -i bookmarks.html [Check links in a file ]
wget --mirror [Efficiently update a local copy of a site (handy from cron)]

copy files from remote server called server1 into /backup directory:
rsync -arvz -e ssh user@server1:/var/www/html /backup

a = archive - means it preserves permissions (owners, groups), times, symbolic links, and devices.
r = recursive - means it copies directories and sub directories
v = verbose - means that it prints on the screen what is being copied

Use of "/" at the end of path:
When using "/" at the end of source, rsync will copy the content of the last folder.
When not using "/" at the end of source, rsync will copy the last folder and the content of the folder.
When using "/" at the end of destination, rsync will paste the data inside the last folder.
When not using "/" at the end of destination, rsync will create a folder with the last destination folder name and paste the data inside that folder.

No comments:

Post a Comment