Linux- copy methods

Linux- copy methods

wget

Download a file

wget http://www.example.com/myfile.html

Download a file whose URL contains a “&” (note quotes)
wget http://www.example.com/min.php?page=home&user=ma

Draw an entire site (recursive download)
wget-r http://www.example.com/

Resume a download
wget-c http://www.example.com/xyz.html

Download files only an extension (here,. Jpg)
wget-A.jpg http://www.example.com/

Limit download speed
wget – limit-rate = 30k http://www.example.com/

Download via ftp (With authentication)
wget-r l4 ftp://username:password @ example.com /

If there are a list of pages, files, and so on that I want to grab for the script, I can list one URL-formatted item per line within a file. For example, if the file was ~/bin/getme, I would use:

wget -i ~/bin/data/getme

To download a copy of a complete web site, up to five levels deep, you use the -r option (for recursive):

wget -r site.com -o logfile

Instead of having the progress messages displayed on the standard output, you can save it to a log file with the -o option:

SCP

Copy the file "foobar.txt" from a remote host to the local host

$ scp your_username@remotehost.edu:foobar.txt /some/local/directory

Copy the file "foobar.txt" from the local host to a remote host

$ scp foobar.txt your_username@remotehost.edu:/some/remote/directory

Copy the directory "foo" from the local host to a remote host's directory "bar"

$ scp -r foo your_username@remotehost.edu:/some/remote/directory/bar

Copy the file "foobar.txt" from remote host "rh1.edu" to remote host "rh2.edu"

$ scp your_username@rh1.edu:/some/remote/directory/foobar.txt \
your_username@rh2.edu:/some/remote/directory/

Copying the files "foo.txt" and "bar.txt" from the local host to your home directory on the remote host

$ scp foo.txt bar.txt your_username@remotehost.edu:~

Copy multiple files from the remote host to your current directory on the local host

$ scp your_username@remotehost.edu:/some/remote/directory/\{a,b,c\} .

$ scp your_username@remotehost.edu:~/\{foo.txt,bar.txt\}  .

RSYNC

$ rsync options source destination

Source and destination could be either local or remote. In case of remote, specify the login name, remote server name and location.

Synchronize Two Directories in a Local Server

$ rsync -zvr /var/dir1/ /root/dir2

-z  :enable compression

-v :verbose

-r    : recursive

-a : preserve time stamp

-d : sync the directory tree structure

Preserve timestamps during Sync using rsync -a

rsync -azv /var/dir1/ /root/dir2

rsync -arzv /var/dir1/ /root/dir2       (this will do recursively)

a = archive - means it preserves permissions (owners, groups), times, symbolic links, and devices.
r = recursive - means it copies directories and sub directories
v = verbose - means that it prints on the screen what is being copied 

Synchronize Only One File

To copy only one file, specify the file name to rsync command

rsync -v /var/singlefile /root/myfile/

Synchronize only the Directory Tree Structure (not the files)

rsync -v -d abc@192.168.200.10:/var/lib/ .

Use of "/" at the end of path:
When using "/" at the end of source, rsync will copy the content of the last folder.
When not using "/" at the end of source, rsync will copy the last folder and the content of the folder.

When using "/" at the end of destination, rsync will paste the data inside the last folder.
When not using "/" at the end of destination, rsync will create a folder with the last destination folder name and paste the data inside that folder.

Synchronize with the Directory Name

What if you wanted to not just copy all of the files under /myDir/, but also the /myDir/ directory name. Just leave off the last "/" after the "/myDir" directory name. This will tell rsync to copy the directory name and all files under that directory. On the remote server you will then see the directory structure /backups/myDir/.

rsync -avz /myDir user@remotemachine:/backups/

Incremental Backups using rsync

The following example will make an incremental backup of the directory /data/working/ and put a copy of any file that changes into a dated directory in /BACKUP/ . This can be used to keep a daily backup tree of any changed files and not have to overwrite the previous days files. Note that this method does need to copy the entire file if it changes as the new files are made in the directory named under current day.

rsync --backup --backup-dir=`date +%Y.%m.%d` -a /data/working/ /BACKUP/

LFTP

#!/bin/bash
HOST="your.ftp.host.dom"
USER="username" 
PASS="password"
LCD="/path/of/your/local/dir"
RCD="/path/of/your/remote/dir" 
lftp -c "set ftp:list-options -a;
open ftp://$USER:$PASS@$HOST;
lcd $LCD;
cd $RCD; 
mirror --reverse \
        --delete \
        --verbose \
        --exclude-glob a-dir-to-exclude/ \
        --exclude-glob a-file-to-exclude \
        --exclude-glob a-file-group-to-exclude* \
        --exclude-glob other-files-to-exclude"

Upload a file

lftp -e 'put /local/path/yourfile.txt; bye' -u user,password ftp.remote.com 

Download a file

lftp -e 'set net:timeout 10; get yourfile.txt; bye' -u user,password ftp.remote.com 

Mirroring / Recursive upload

When using the mirror command, LFTP is recursive by default.

Mirror everything from the /local/path to the root of the remote FTP site, including all subdirectories and their files. The -R switch means “reverse mirror” which means “put” [upload]. Remote path is simply /, the root.

lftp -e 'mirror -R /local/path/ /' -u user,password ftp.remote.com

Recursive Download

To mirror remote to local, just omit the -R param and swap remote path with local. Be careful with this. Don’t overwrite your local changes.

lftp -e 'mirror / /local/path/' -u user,password ftp.remote.com  

Special Characters in Username or Password

Special characters in your username or password must be escaped with backslashes:

lftp -e 'put /pathToyourfile.txt; bye' -u user,password\!\! ftp.remote.com