earthw0rmjim earthw0rmjim - 1 month ago 13
Java Question

Parsing dynamically growing web pages with jsoup

I'm writing an Android app that parses a web page, filters the image links from it and load them in a

WebView
.

It works fine for static pages, but i have no idea how to handle pages that dynamically add content as i scroll down, such as 9gag, imgur, Facebook etc.

Is there a solution for this? I guess the dynamic content is handled by JavaScript. Maybe there's a way to call this JavaScript code before parsing the page?

I'd appreciate any advice.

Thanks in advance.

Answer

You should try looking at the requests that dynamic pages make. All of them use a pattern of dynamic pagination, or a cursor. Imgur for example issues requests with an url like this.

https://imgur.com/gallery/hot/viral/page/4/hit?set=0

Where you specify the page and the set is the portion of the page (Normaly they go up to 3)

Comments