Joseph Joseph - 7 months ago 75
PHP Question

How to reduce load time of curl and parse html

i need to reduce load time of my script. It is curl and simple parse dom.
This is my script, i need help :(
It lasts about 2 minutes, i need to parse many different pages!

require_once ("simple_html_dom.php");

function curl ($page){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "URL");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
return $result;
$start = microtime(true);
$text = "here text of last page";

$i = 0;
while(strpos(str_get_html(curl($i)), $text) == null){
$html = str_get_html(curl($i));
foreach($html->find('div#box-container-inner') as $e){
PRINT etc... only for test
echo "parsata la pagina ".($i+1)."<br>";
$time_elapsed_secs = (microtime(true) - $start)/60;
echo $time_elapsed_secs;


You appear to be running CURL twice in each loop (once to evaluate the while-loop condition, once to set $html) and converting the resulting string into an object for each loop. That's four potentially-intensive processes each time you loop that you can knock down to two per loop.

Instead, you can assign the $html variable to the result of str_get_html(curl($i)) while within the while-loop evaluation:

while(strpos(($html = str_get_html(curl($i))), $text) === false) {
//    $html = str_get_html(curl($i));