google spiders

Networking/Security Forums -> Beginners // Misc. Computer Questions

Author: daffyduck PostPosted: Wed Dec 08, 2010 6:41 pm    Post subject: google spiders
hi can anyone explain to me how google spiders work??

Author: PhiBerLocation: Your MBR PostPosted: Wed Dec 08, 2010 6:56 pm    Post subject:
Sure, someone can explain but it is easier to simply search Google for what has already been written:

How Internet Spiders Work
How Spiders Work

Author: DaveShield PostPosted: Thu Dec 23, 2010 12:44 am    Post subject:
Hey PhiBer - I'll take you up on that and give explanation a shot... Smile

First, GoogleBot starts by fetching a specific url. The bot examines the code of the page, paying particular attention to title, meta description, headers, body copy and ALT text. The info in each of these elements is used to determine relevancy.

Next, the Bot gathers up all the links on the page - looks for <a href="etc, etc." Those links are put into the queue so that each of those urls can, in turn, be analyzed - this is the act of "crawling." The bot does the same with each of those urls.

After the bot examines the page code, notes the keyword relevancy, puts the links on the page into queue, the page code is then put through a SPAM filter. This looks for anomalous code or any attempt to game the engine, i.e., white text on white background, for a simple example.

Once through the SPAM filter, the url/page is categorized based on keyword relevancy and placed into the Google Index - the page is now said to have been "indexed."

After a page has been indexed, it is available to show up in search engine results. Just being indexed, however is no guarantee that the page will show up at all, much less high up in the results. A site must meet relevancy criteria to be displayed. Google uses various criteria to rank sites - a completely separate issue than how the bot works.

I'm sure I'm missing some technical details, but those are the basics:

Look at the page => examine title, meta desc, header, body copy, ALT => grab/queue links => SPAM scrubber => Index.

Hope that helps!


Author: Fire AntLocation: London PostPosted: Wed Feb 09, 2011 2:47 pm    Post subject:
Hi Carl,

Thanks for posting about Fetch Googlebot, not only did I learn something but I can now include it in my current research.


Fire Ant

Author: Allnsmth PostPosted: Mon Feb 21, 2011 11:59 am    Post subject:
Google spider read meta tag from your site and index your site if you have unique and related content your keywords..!

Networking/Security Forums -> Beginners // Misc. Computer Questions

output generated using printer-friendly topic mod, All times are GMT + 2 Hours

Page 1 of 1

Powered by phpBB 2.0.x © 2001 phpBB Group