Rsync and Wget

This tool helps you to copy files or directories between a local and a remote host.

rsync attempts to identify differences between source and destination files prior to initiating a transfer, and (assuming differences exist) tries only to copy the changes, rather than the entire file.
http://www.samba.org/rsync/

It woks like

On the same machine

rsync -a sourceDir/ destinationDir/

rsync care about the trailing slash, but only on the source argument










say dir1 and dir2 are two directories and file1 is the file that reside under dir1

command:-
rsync -a dir1 dir2 ==> will give the output as dir2/dir1/file1
command:-
rsync -a dir1/ dir2 ==> will give the output as dir2/file1

command:-
rsync -a --delete sourceDir/ destinationDir/ ==> will delete extra file from destinationDir which does not have presence in sourcedir



For Copying data by Rsync, it should be installed on both hosts.

Example:- How to copy data between localhost and remote host

You are sitting on localhost machine where you want to copy data from a remote machine

rsync -avz -e ssh userName@remoteHost:/remote/outgoingData/ /local/incomingData

here "/" after the outgoing data means that the localhost already have the directory and you just just want the data from the outgoingData directory.

If we remove the "/" then it means that you want to create the outgoingData directory first on
the remote machine then want to copy data from it. It should be like below:-

rsync -avz -e ssh userName@remoteHost:/fullPath/outgoingData/ /fullPath/incomingData/outgoingData


Mirroring the websites:-
you are giving the below command from server2.xyz.com (you want the backup to this server)
rsync -avz -e ssh userName@server1.xyz.com:/var/www/ /var/www/


rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html' (Mirror web site (using compression and encryption))

rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ (Synchronize current directory with remote one(

WGET

wget -c http://www.example.com/big.file [Continue downloading a partially downloaded file ]

wget -r -nd -np -l1 -A '*.jpg' http://www.example.com/dir/ [Download a set of files to the current directory ]

wget ftp://remote/file[1-9].iso/ [FTP supports globbing directly ]

wget -q -O- http://www.pixelbeat.org/timeline.html | grep 'a href' | head [Process output directly ]

echo 'wget url' | at 01:00 [Download url at 1AM to current dir]

wget --limit-rate=20k url [Do a low priority download (limit to 20KB/s in this case) ]

wget -nv --spider --force-html -i bookmarks.html [Check links in a file ]

wget --mirror http://www.example.com/ [Efficiently update a local copy of a site (handy from cron)]

copy files from remote server called server1 into /backup directory:
rsync -arvz -e ssh user@server1:/var/www/html /backup



a = archive - means it preserves permissions (owners, groups), times, symbolic links, and devices.
r = recursive - means it copies directories and sub directories
v = verbose - means that it prints on the screen what is being copied


Use of "/" at the end of path:
When using "/" at the end of source, rsync will copy the content of the last folder.
When not using "/" at the end of source, rsync will copy the last folder and the content of the folder.
When using "/" at the end of destination, rsync will paste the data inside the last folder.
When not using "/" at the end of destination, rsync will create a folder with the last destination folder name and paste the data inside that folder.