Yuv Yuv - 1 year ago 129
Ajax Question

GreaseMonkey/Javascript: continue running after paginating

I am writing a script to scrape information from a multi-page output. The output is paginated plain vanilla in the URL: http://example.com/results.php?records_per_page=100&page=1 and the results are AJAX-loaded.

My script includes the waitForKeyElements utility and starts scraping only after the results have loaded. After scraping, it triggers the next page, but does not start scraping the next page unless I manually reload the page.

The essence of my script:

// ==UserScript==
// @name example
// @namespace http://example.com
// @description example
// @version 1
// @grant GM_xmlhttpRequest
// @grant GM_getValue
// @grant GM_setValue
// @include https://example.com/results.php?record_per_page=*&page=*
// @require https://code.jquery.com/jquery-2.2.4.min.js
// ==/UserScript==

function iterateResults () {
// iterate through all content DIVs on the page
for (i = 1; i<=$('div.row_content).length; i++) {
// parse each DIV
a = parse($(div.row_content[i]);
// issue an AJAX request to my server to save the parsed object
save (a);

function nextPage() {
patt = /^(.*&page=)([0-9]+)(.*)$/;
match = document.location.href.match(patt);
p = 1+match[2];
document.location = nextURL;

* Wait until all AJAX elements have loaded
function Timer() {
timer = GM_getValue("timer","")
timer = window.setTimeout(function(){

waitForKeyElements ("div.results_lists", Timer, false);

This works well on page=1, but stops after loading page=2 with document.location.

What am I doing wrong?

Yuv Yuv
Answer Source

I ended up finding the answer, and it was very simple. The pages are loaded via AJAX, and adding a dummy function triggered the scraping on the second and subsequent pages:

window.addEventListener ("hashchange", fireOnNewPage,  false);

function fireOnNewPage () {
  // console.log ("Fired");

PLEASE READ BEFORE DOWNVOTING: This answer has received a number of down and up votes. If you don't tell me why you down voted, I cannot improve on it and I do not know what I did wrong. Kindly share your thought. In the meantime, my script is in production and works well as answered here.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download