dvsoukup dvsoukup - 1 month ago 10
Bash Question

nodejs - scan directory forever for new files

Wondering what the best approach is here for my situation. Currently operating a Linux AWS instance, which runs Apache, hosting a Nodejs application.

Here is a brief scenario of the "flow" of things:


  1. User uploads a file via http post (i have this working)

  2. File has some validation applied to it (correct fields and what not..)

  3. nodejs writes file to some directory on the linux box

  4. nodejs executes a shell command, which executes a shell script. This shell script picks up the file just written to process it through another 3rd-party application on the Linux box.

  5. Once this "other" application finishes processing it, it spits out a processed file on the other end to some directory.

  6. Back in nodejs, pick up new file and process data down into the database



So what I need help doing here is the part AFTER the file is processed... between steps 5 and 6. What is the best way of going about this?


  1. Can I just do some infinite running loop in nodejs that searches every X seconds if files exist in that directory, pick up and process, then delete?



OR


  1. Better to setup a cron job on the linux box that would be performing that duty? I'd need a way to let the nodejs application "know" that a file is ready though to send into the database.



If I go with choice 1, how would that look like in the node js app? Something like?

while(true){ //forever loop
//do stuff
process.sleep(30 seconds)
}


If option 2 is better, how can I let the nodejs app know to process the file? Maybe just call an Express endpoint as a signal to go out and look in the directory?

I've got everything else working (db connectivity, file reading/writing, etc...). My brain is just stuck on this piece of trying to understand how to more or less get these two different applications to notify the other.

Answer

Wound up using "chokidar" to do this! What a great library! Greatly improves upon the default "fs.watch" that comes with nodejs. Worked perfectly for what I needed to do. Thanks marekful for the suggestion!