I'm trying to write a script that will do a few things in the following order:
for url in $urllist; do
content="$(curl -L -s "$url" | grep -iF "$keywords" /dev/null)"
Here's how I would do it:
#!/bin/bash keywords="$(<./keywords.txt)" while IFS= read -r url; do curl -L -s "$url" | grep -ioF "$keywords" | while IFS= read -r keyword; do echo "$url: $keyword" done done < ./url_list.txt
What did I change:
$(<./keywords.txt)to read the
keywords.txt. This does not rely on an external program (
catin your original script).
forloop that loops over the url list, into a
whileloop. This guarentees that we use
Θ(1)memory (i.e. we don't have to load the entire url list in memory).
/dev/nullalone is meaningless, since it will find nothing there. Instead, I invoke
grepwith no arguments so that it filters its
stdin(which happens to be the output of
curlin this case).
grepso that it outputs only the matched keyword.
curl. Instead I run the command directly and feed its output to a while loop. This is necessary because we might get more than keyword match per url.