Tony Tony - 6 months ago 12
Bash Question

Script to distribute a large number of files in to smaller groups

I have folders containing large numbers of files (e.g. 1000+) of various sizes which I want to move in to smaller groups of, say, 100 files per folder.

I wrote an Apple Script which counted the files, created a numbered subfolder, and then moved 100 files in to the new folder (the number of files could be specified) which looped until there were less than specified number of files which it moved in to the last folder it created.

The problem was that it ran horrendously slowly. I'm looking for either an Apple Script or shell script I can run on my MacBook and/or Linux box which will efficiently move the files in to smaller groups.

How the files are grouped is not particularly significant, I just want fewer files in each folder.

Answer

This should get you started:

DIR=$1
BATCH_SIZE=$2
SUBFOLDER_NAME=$3
COUNTER=1

while [ `find $DIR -maxdepth 1 -type f| wc -l` -gt $BATCH_SIZE ] ; do
  NEW_DIR=$DIR/${SUBFOLDER_NAME}${COUNTER}
  mkdir $NEW_DIR
  find $DIR -maxdepth 1 -type f | head -n $BATCH_SIZE | xargs -I {} mv {} $NEW_DIR
  let COUNTER++
if [ `find $DIR -maxdepth 1 -type f| wc -l` -le $BATCH_SIZE ] ; then
  mkdir $NEW_DIR
  find $DIR -maxdepth 1 -type f | head -n $BATCH_SIZE | xargs -I {} mv {} $NEW_DIR
fi
done

The nested if statement gets the last remaining files. You can add some additional checks as you see needed after you modify for your use.

Comments