Afzaal Afzaal - 11 months ago 89
MySQL Question

how to build web crawler/spider Like Google php mysql

I am wondering that how to build a superfast Web crawler or spider that works same as google's bot did,
that is pretty difficult but where to get started any idear will be appreciated

Answer Source

You have to use this script for php search engine.

<script src="jquery.js"></script>
<script language="javascript">

function getCrawl(x)

$.get("json.php", {site:x}, function(data){

var ans = data;
var b = $("#output").html();

for(var y=0;y!=ans.length;y++)
b += "<a href='javascript:getCrawl(\""+ans[y]+"\")'>"+ans[y]+"</a><br />";


}, "json")



<div id="output">

$target_url = "";

$html = new simple_html_dom();
foreach($html->find('a') as $link){
echo "<a href='javascript:getCrawl(\"".$link->href."\")'>".$link->href."</a><br />";