piotrek piotrek - 5 months ago 24
Linux Question

bash: sending compressed files while compressing others

i have a simple bash script to download a lot of logs files over pretty slow network. i can compress logs on the remote side. basically it's:

ssh: compress whole directory
scp: download archive
ssh: rm archive

using lzma gives great compression but compressing the whole directory is slow. is there any tool or easy way to write a script that allows me to compress a single files (or a bunch of files) and start downloading them while other files/chunks are still being compressed? i was thinking about launching compressing for every single file in the background and in the loop downloading/rsync files with correct extension. but then i don't know how to check if compressing process finished its work


best solution i found here. in my case it was:

ssh -T user@example.com 'tar ... | lzma -5 -' > big.compressed