Linux- copy methods
wget
Download a file
Download a file whose URL contains a “&” (note quotes)
wget http://www.example.com/min.php?page=home&user=ma
wget http://www.example.com/min.php?page=home&user=ma
Draw an entire site (recursive download)
wget-r http://www.example.com/
wget-r http://www.example.com/
Resume a download
wget-c http://www.example.com/xyz.html
wget-c http://www.example.com/xyz.html
Download files only an extension (here,. Jpg)
wget-A.jpg http://www.example.com/
wget-A.jpg http://www.example.com/
Limit download speed
wget – limit-rate = 30k http://www.example.com/
wget – limit-rate = 30k http://www.example.com/
Download via ftp (With authentication)
wget-r l4 ftp://username:password @ example.com /
wget-r l4 ftp://username:password @ example.com /
If there are a list of pages, files, and so on that I want
to grab for the script, I can list one URL-formatted item per line within a
file. For example, if the file was ~/bin/getme, I would use:
wget -i ~/bin/data/getme
wget -i ~/bin/data/getme
To download a copy of a complete web
site, up to five levels deep, you use the -r option (for recursive):
wget -r site.com -o logfile
Instead
of having the progress messages displayed on the standard output, you can save
it to a log file with the -o option:
SCP
Copy
the file "foobar.txt" from a remote host to the local host
$
scp your_username@remotehost.edu:foobar.txt /some/local/directory
Copy the file "foobar.txt" from the local host to a remote host
$
scp foobar.txt your_username@remotehost.edu:/some/remote/directory
Copy the directory "foo" from the local host to a remote host's directory "bar"
Copy the file "foobar.txt" from remote host "rh1.edu" to remote host "rh2.edu"
$
scp your_username@rh1.edu:/some/remote/directory/foobar.txt \
your_username@rh2.edu:/some/remote/directory/
your_username@rh2.edu:/some/remote/directory/
Copying the files "foo.txt" and "bar.txt" from the local host to your home directory on the remote host
$
scp foo.txt bar.txt your_username@remotehost.edu:~
Copy multiple files from the remote host to your current directory on the local host
$
scp your_username@remotehost.edu:~/\{foo.txt,bar.txt\} .
RSYNC
$ rsync options source destination
Source
and destination could be either local or remote. In case of remote, specify the
login name, remote server name and location.
Synchronize Two Directories in a Local Server
$ rsync -zvr /var/dir1/ /root/dir2
-z :enable compression
-v :verbose
-r : recursive
-a
: preserve time stamp
-d
: sync the directory tree structure
Preserve timestamps during Sync using rsync -a
rsync -azv /var/dir1/ /root/dir2
rsync -arzv /var/dir1/ /root/dir2 (this will do recursively)
a = archive - means it preserves permissions (owners, groups), times, symbolic links, and devices. r = recursive - means it copies directories and sub directories v = verbose - means that it prints on the screen what is being copied
Synchronize Only One File
To
copy only one file, specify the file name to rsync command
rsync -v /var/singlefile /root/myfile/
Synchronize only the Directory Tree Structure (not the files)
rsync -v -d abc@192.168.200.10:/var/lib/ .
When using "/" at the end of source, rsync will copy the content of the last folder.
When not using "/" at the end of source, rsync will copy the last folder and the content of the folder.
When using "/" at the end of destination, rsync will paste the data inside the last folder.
When not using "/" at the end of destination, rsync will create a folder with the last destination folder name and paste the data inside that folder.
Synchronize
with the Directory Name
What if you wanted to not just copy all of the
files under /myDir/, but also the /myDir/ directory name. Just leave off the
last "/" after the "/myDir" directory name. This will tell
rsync to copy the directory name and all files under that directory. On the
remote server you will then see the directory structure /backups/myDir/. rsync -avz /myDir user@remotemachine:/backups/
Incremental
Backups using rsync
The following example will make an incremental backup of the directory /data/working/ and put a copy of any file that changes into a dated directory in /BACKUP/ . This can be used to keep a daily backup tree of any changed files and not have to overwrite the previous days files. Note that this method does need to copy the entire file if it changes as the new files are made in the directory named under current day.
rsync --backup --backup-dir=`date +%Y.%m.%d` -a /data/working/ /BACKUP/
LFTP
#!/bin/bash HOST="your.ftp.host.dom"USER="username" PASS="password"LCD="/path/of/your/local/dir"RCD="/path/of/your/remote/dir" lftp -c "set ftp:list-options -a;open ftp://$USER:$PASS@$HOST;lcd $LCD;cd $RCD; mirror --reverse \ --delete \ --verbose \ --exclude-glob a-dir-to-exclude/ \ --exclude-glob a-file-to-exclude \ --exclude-glob a-file-group-to-exclude* \ --exclude-glob other-files-to-exclude"
Upload a file
lftp -e 'put /local/path/yourfile.txt; bye' -u user,password ftp.remote.com
Download a file
lftp -e 'set net:timeout 10; get yourfile.txt; bye' -u user,password ftp.remote.com
Mirroring / Recursive upload
When using the
mirror command, LFTP is recursive by default.
Mirror everything from the
/local/path to the root of the remote FTP site,
including all subdirectories and their files. The -R switch means “reverse
mirror” which means “put” [upload]. Remote path is simply /, the root.lftp -e 'mirror -R /local/path/ /' -u user,password ftp.remote.com
Recursive Download
To mirror remote to local, just omit the
-R param and swap remote
path with local. Be careful with this. Don’t overwrite your local changes.lftp -e 'mirror / /local/path/' -u user,password ftp.remote.com
Special Characters in Username or Password
Special characters in your username or password must be escaped with backslashes:lftp -e 'put /pathToyourfile.txt; bye' -u user,password\!\! ftp.remote.com
