Alex Bueno Alex Bueno - 1 month ago 4x
Linux Question

Listing files while working with them - Shell Linux

I have a database server that it basic work is to import some specific files, do some calculations and provide data in a web interface.

It's planned for next weeks a hardware replacement, it needs to migrate the database. But there's one problem in it: the actual database is corrupted and show some errors in web interface. This is due to server freezing while importing/calculating, that's why the replacement.

So I'm not willing to just dump the db and restore in the new server. Doesn't make sense to still use the corrupted database and while dumping the old server goes really slow. I have a backup from all files to be imported (the current number is 551) and I'm working on a script to "re-import" all of them and have a nice database again.

The actual server takes ~20 minutes to import each new file. Let's say that new server takes 10 for each file due to its power... It's a long time! And here comes the problem: it receives new file hourly, so there will be more files when it finishes the job.

Restore script start like this:

for a in $(ls $BACKUP_DIR | grep part_of_filename); do

Question is: does this "ls" will have new file names when they come? File names are timestamp based, so they will be in the end of the list.

Or does this "ls" is execute once and results goes to a temp var?



ls will execute once, at the beginning, and any new files won't show up.

You can rewrite that statement to list the files again at the start of each loop (and, as Trey mentioned, better to use find, not ls):

while all=$(find $BACKUP_DIR/* -type f | grep part_of_filename); do
  for a in $all; do

But this has a major problem: it will repeatedly process the same files over and over again.

The script needs to record which files are done. Then it can list the directory again and process any (and only) new files. Here's one way:

touch ~/done.list
# loop while f=first file not in done list:
#   find                       list the files; more portable and safer than ls in pipes and scripts
#   fgrep -v -f ~/done.list    pass through only files not in the done list
#   head -n1                   pass through only the first one
#   grep .                     control the loop (true iff there is something)
while f=`find * -type f | fgrep -v -f ~/done.list | head -n1 | grep .`; do
  <process file $f>
  echo "$f" >> ~/done.list