Thomas Thomas - 1 month ago 18
Linux Question

how to send a request to Google with curl

I work on Linux and try to use

curl
to send requests to Google and save its reply as a html file.


When I use Google to search something, such as a string "abc", I find that the link of Google is: https://www.google.lu/#q=abc


So I try like this:


curl https://www.google.lu/#q=abc -o res.html


But the
res.html
is just the main page of Google, instead of the result of searching "abc".


How to do it?

Answer

Anything after the # is handled client side with JavaScript, which is why it doesn't work with curl.

You can instead use the traditional, non-AJAX interface on https://www.google.com/search?q=abc

It appears to block you unless you also spoof the user agent, so all in all:

curl \
  -A 'Mozilla/5.0 (MSIE; Windows 10)'  \
  -o res.html \
  https://www.google.com/search?q=abc 
Comments