Elong Chen Elong Chen - 2 months ago 21
R Question

r trycatch for conduct a forever loop for web crawling Q

I recently learn to web crawling.

I use the While(TRUE) to let my function of web crawling to run forever.

However, my internet is not stable, so it sometimes shows error when my internet failed.

I tried to figure it out use the trycatch function

rf() is my function of web crawling.

if(tryCatch(rf(), error=function(cond) FALSE)==FALSE){
Sys.sleep(60+sample(1:5,1))
rf()
}


I ask R to run it again after a while because the Internet usually would be fine after a while.

However, this code is not good enough because if the internet failed for the second time then R stop.

I thought if i use repeat and it might be better, as followed.

Am I right?

repeat {
if(tryCatch(rf(), error=function(cond) FALSE)==FALSE){
Sys.sleep(60+sample(1:5,1))
rf()
}
Sys.sleep(900+sample(1:30,1))#second
}

Answer

If you want to wait a longer time period if the requests fails, try this code:

while(TRUE) {
  tryCatch({
    rf()
    print("wait normal")
    Sys.sleep(60+sample(1:5,1))
  }, error = function(e) { 
    print("wait longer") 
    Sys.sleep(900+sample(1:30,1))
    }
  )
}

Note that your rf() function must be in a try statement so that errors are catched and not raised further.