pingul pingul - 7 days ago 4x
Bash Question

How do I copy with scp with a wildcard(*) in the destination path?

Is it possible to get the value of what a wildcard matched to? I.e. I want to do

scp -r user@host:some/path/with/*/file.txt cp/to/*/file.txt

With the command above, I want

some/path/with/TEST/file.txt --> cp/to/TEST/file.txt
some/path/with/OTHER/file.txt --> cp/to/OTHER/file.txt

I'll add some more data of my actual setup, which might make it clearer.

On my server I have the structure

server $ pwd
server $ ls

is a directory, containing files like
etc. I could copy everything over to my own computer like

pingul $ scp -r 'user@host:sixtrack/simulations/*' .

However, I would like to not copy over every file, instead just some. Because all files are named the same and only separated through the directory, I have to respect the structure. As such, I would like
to go into my local folder


pingul $ pwd
pingul $ scp -r 'user@host:sixtrack/simulations/*/summary.dat' */.

Does not accomplish what I want. Adding the
option as suggested prints out

Executing: cp -r -- run0001/. run0100/.
Executing: cp -r -- run0002/. run0100/.
Executing: cp -r -- run0003/. run0100/.

and I only get one
. All other directories are empty (i.e.
, ...)

Is what I want possible?


It depends on whether you want to expand * before running a command or after running it and whether you want to do it interactively or not.

If you want to expand it before running a command interactively you can either use insert-completions (M-*) or glob-expand-word (C-x *) described in man bash:

   glob-expand-word (C-x *)
          The word before point is treated as a pattern for
          pathname expansion, and the list of matching filenames
          is inserted, replacing the word.  If a numeric argument
          is sup- plied, an asterisk is appended before pathname

   insert-completions (M-*)
          Insert all completions of the text before point that
          would have been generated by possible-completions.

To use these functions put a cursor before or after * and press either Control-x * or Alt-*:

$ pwd
$ ls
$ cp /tmp/expand-*/*

Now put your cursor after the last *, don't press Enter but C-x * and you'll get this

$ cp /tmp/expand-glob/FILE

If you want to expand * to test command in the script then neither scp nor cp is a good option because the cannot run in dry-run mode. You should go with something like rsync that would show what files it would transfer if it was actually run like this:

$ rsync  -vn --relative /tmp/expand-*/* .


How about this:

$ rsync -avn -R --rsync-path="cd sixtrack/simulations && rsync"  user@host:run*/summary.dat .

-n stands for dry-run. With this option rsync will only print how will recreate a remote directory structure in current directory. In your case it will be something like:

receiving incremental file list
drwxr-xr-x          4,096 2016/10/13 12:45:36 run0001
-rw-r--r--              0 2016/10/13 12:23:18 run0001/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run00010
-rw-r--r--              0 2016/10/13 12:23:18 run00010/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0002
-rw-r--r--              0 2016/10/13 12:23:18 run0002/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0003
-rw-r--r--              0 2016/10/13 12:23:18 run0003/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0004
-rw-r--r--              0 2016/10/13 12:23:18 run0004/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0005
-rw-r--r--              0 2016/10/13 12:23:18 run0005/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0006
-rw-r--r--              0 2016/10/13 12:23:18 run0006/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0007
-rw-r--r--              0 2016/10/13 12:23:18 run0007/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0008
-rw-r--r--              0 2016/10/13 12:23:18 run0008/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0009
-rw-r--r--              0 2016/10/13 12:23:18 run0009/summary.dat