Ava Barbilla Ava Barbilla - 3 years ago 179
HTTP Question

cURL - Submit form not working as expected

I am writing a bash script that should simulate:

  1. Typing into a search bar.

  2. Submit its content.

  3. Fetch the results page.

Given this basic example:

<input name="search" type="TEXT">
<input name="submit" value="SUBMIT" type="SUBMIT" align="center">

I would use
curl -s -d "search=value%&submit=SUBMIT" <URL> > export.html
to submit this form and write the results into

Given the next form:

<form action="/search.asp" method="GET" name="mmssearch">
<div class="search-group">
<input class="search-input" type="text" value="" placeholder="Enter search value here..." name="_search" autocomplete="off">
<input id="form-submit" class="search-submit" type="submit" value="">

I would use
curl -s -d "_search=value" <URL> > export.html
, however, the form submission outputs the same
as before the search, which means nothing has changed; it returns the homepage instead of the results page.

How would I go about submitting this form? Did I forget something?

NB: The target URL itself contains several forms (amongst mobile search bars, login forms & others) of which some are very if not to say exactly identical, hence, it would be reasonable telling the script to use this specific form.

Answer Source

I accidentally used he wrong link. The correct approach that worked for me in order to parse the results page via the GET method is:

curl -L '<URL>/search.asp?_search=value'

I needed the -L flag to follow the redirects, as thankfully stated by @TravisClarke. The <URL> in this case stands for Base URL. The ? is pretty straight forward for experienced users; it passes a set of parameters to the target URL which in turn should output the desired result page.

There is no need for -d or --data flag as the parameters are passed directly through the link, as opposed to POST which should require -d and sometimes --cookie as well.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download