John Roberts John Roberts - 1 year ago 116
Javascript Question

Get Dynamically Loaded HTML After Scrolling To Bottom

I have the following code:


import com.machinepublishers.jbrowserdriver.JBrowserDriver;
import com.machinepublishers.jbrowserdriver.Settings;
import com.machinepublishers.jbrowserdriver.Timezone;

public class ReadMovies {

public static void main(final String[] args) throws IOException {
final JBrowserDriver driver = new JBrowserDriver(Settings.builder().timezone(Timezone.AMERICA_NEWYORK).build());

driver.executeScript("window.scrollTo(0, document.body.scrollHeight)");



However, the
seems to be having no effect - the program still prints the original source as when the page was first loaded (so no additional elements were loaded). Does anyone know what's wrong?

Answer Source

There are additional movies loaded when you scroll the page down. If you want to load all of the or certain amount of movies, you would have to continuously scroll down and wait.

Naive implementation:

int DESIRED_COUNT = 100;
int currentCount = driver.findElements(By.cssSelector("#films .film")).size();

while (currentCount < DESIRED_COUNT) {
    driver.executeScript("window.scrollTo(0, document.body.scrollHeight)");
    currentCount = driver.findElements(By.cssSelector("#films .film")).size();

// now get the page source

The Thread.sleep() wait should probably be replaced with an Explicit Wait via WebDriverWait.