Konstantin Konstantin - 2 months ago 24
Ruby Question

Tool to help automatize web tasks?

I would like to automatize web tasks which normally should be achieved by a web browser, for example video upload to a certain video sharing website. I would like to make a pure command line program, which apply simple cURL command calls, because those calls are accessible for example in FireBug and Chrome / Chromium dev console / Network pane. So I woudn't like to use libcurl, or similar libraries. I would prefer programming in Ruby.

Task is straightforward: I upload a video while watching the dev tool network pane, and tracking the communication between the browser and the server. Copying POST and GET requests by "copy as cURL" menu. Applying some modification on the copied cURL command, eg. removing some header lines, which sends cookies, and substitute them with the cookies in a "cookie jar" text file (-c option in cURL). And later sending the needed cookies by applying that text file again (-b option in cURL). In the past I managed to make such Ruby scripts and they are simply working I can use those websites services by pure command line, so I can upload files from my VPS which is very fast unlike uploading from home machine.

Unfortunately the website I want to automatize apply a lot of redirections even at the login stage (for example 4 consecutive redirects), which aren't tracked by Chrome dev tool, so I can't see what really happen and when the needed cookies are stored, and which request is responsible for getting those cookies. Sometimes tricky javascript calls are applied by the website to store a cookie which is needed for the video upload and even exporting the video.

So my question is that besides Chrome dev tools and FireBug is there any automated and handy tool which can help achiving similar tasks?


Maybe BrowserAutomationStudio will help:

This program can record your browser actions and replay them as a standalone bot.