An internet spider is a robot that crawls around the world wide web. They are also sometimes called crawlers. They use an algorithmic programme that follows links throughout the net searching for new content. This then fetches the new webpages and adds them to the search engine indexes. Google is a crawler-based search engine, as it relies on spiders to automatically create its listings.
Some spiders have even been given names, such as Mozilla for Netscape, Scooter for Alta Vista and Slurp for Hotbot. They leave evidence of their visits just like human surfers in analytics, code and stats.
Spiders enter and leave websites through links, which act as portals throughout the net. That's why it's important to have lots of incoming links to your website to encourage spider activity. If you provide lots of new content for spiders to feed on, they will remember to visit your site more frequently.
Spiders only see text on the webpages, therefore pictures and Flash programmes are invisible to them. You can add alt tags to your pictures which are written descriptions behind them, enabling spiders to understand your images.
Spiders are programmed to look for new content with links, tags and keywords. They particularly relish appropriately selected keywords combined with extremely relevant links and their destinations. They don't like hidden or invisible keywords, as they think they're being fooled. If your site's navigation is complete, spiders will visit every page, indexing anything that's new. If you treat spiders well, they are more likely to return.
@Diana
Well done my friend......you have done your homework...I am very impressed....even us old timers can learn new things about technology
Thanks Diana. msn emoticon (8)
Seeing the word "crawl" is mentioned in there I think the Doom will stay here next to PC.