John Roberts John Roberts - 7 months ago 60
Javascript Question

Get Dynamically Loaded HTML After Scrolling To Bottom

I have the following code:

import java.io.IOException;

import com.machinepublishers.jbrowserdriver.JBrowserDriver;
import com.machinepublishers.jbrowserdriver.Settings;
import com.machinepublishers.jbrowserdriver.Timezone;

public class ReadMovies {

public static void main(final String[] args) throws IOException {
final JBrowserDriver driver = new JBrowserDriver(Settings.builder().timezone(Timezone.AMERICA_NEWYORK).build());
driver.get("http://www.cinesift.com/#/");

driver.executeScript("window.scrollTo(0, document.body.scrollHeight)");

System.out.println(driver.getPageSource());

driver.quit();
}
}


However, the
executeScript()
seems to be having no effect - the program still prints the original source as when the page was first loaded (so no additional elements were loaded). Does anyone know what's wrong?

Answer

There are additional movies loaded when you scroll the page down. If you want to load all of the or certain amount of movies, you would have to continuously scroll down and wait.

Naive implementation:

int DESIRED_COUNT = 100;
int currentCount = driver.findElements(By.cssSelector("#films .film")).size();

while (currentCount < DESIRED_COUNT) {
    driver.executeScript("window.scrollTo(0, document.body.scrollHeight)");
    Thread.sleep(1000);
    currentCount = driver.findElements(By.cssSelector("#films .film")).size();
}

// now get the page source
System.out.println(driver.getPageSource());

The Thread.sleep() wait should probably be replaced with an Explicit Wait via WebDriverWait.

Comments