JKaiser JKaiser - 6 months ago 30
Linux Question

Automation of Lynx

I want to build a custom Wordlist for Research and therefor im scraping special Websites.
I want to:
Use Lynx from Commandline with this Command:

lynx -dump -listonly "example.com" | grep -E -o "http://example.com.*|http://www.example.com.*" | sort | uniq > output.txt


I dont want to write 5000 x this command. I would like to use a .txt file with Websites which will be read by the lynx command and all "output" should go in one file.
Can you help me to solve this? Im a noob at Linux/Ubuntu and know nothing about coding. Sorry for my language, im german.

Answer

Do you mean something like

cat yourdomains.txt | while read domain
do
    lynx -dump -listonly "$domain" | \
       grep -E -o "http://${domain}.*|http://www.${domain}.*" 
done | sort | uniq > output.txt

?