Vasanth Nag K V Vasanth Nag K V - 1 year ago 93
Java Question

How to download a Web page source using Selenium

I have just started using Selenium Web Driver and I am stuck with a problem: I want to download a web page's source to my Java program. I have tried using

driver but the result I got does not exactly match the result I got when I manually did the following:

right click on the browser -> view page source.

I am not able to figure out what the problem is. Is there a different API for my purpose or am I using the wrong driver here? Should I use a chrome driver instead of the
driver? If yes, how to use the chrome driver?

Here is what I am doing:

WebDriver driver = new HtmlUnitDriver();

Answer Source

I've just check out Fluent Selenium which uses Firefox WebDriver. It's a testing framework, so don't be surprised by presence of asserting methods. It can be used for crawling. Worked perfectly for me with very little configuration. It requires Maven to run, here is my working example:

package fluent;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.seleniumhq.selenium.fluent.FluentWebDriver;
import org.seleniumhq.selenium.fluent.Period;
import org.seleniumhq.selenium.fluent.TestableString;

import java.util.concurrent.TimeUnit;

import static org.openqa.selenium.By.className;

public class Test {
    public static void main(String[] args) {
        WebDriver driver = new FirefoxDriver();
        FluentWebDriver fwd = new FluentWebDriver(driver);

        driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);


        TestableString test = fwd.div(className("heading")).within(Period.secs(3)).getText();

        System.out.println("header: " + test.toString());


        System.out.println("all is fine!");

My pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns=""



        <!-- If you're needing Coda Hale's Metrics integration (optional) -->




FluentLenium seems being a little more popular.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download