How the Spider ("Bot") Searches the Pages
Once you submit your page, the search engine will add your URL to a list of sites to be spidered. The search engine will read all the words on the page and filter out parts it does not want when your URL is spidered. The search engine can take anywhere from one day to a number of months to spider your site. The benefits of placement usually take two to six months but the time spent waiting is well worth it. You’ll get more hits from higher quality prospects (target group) and more opportunities to convert these hits to sales.
The “Spidering” process involves reading all of the text and links on a website and saving that information to the hard drive of the respective search engine. The links are followed by the Spider and weighed for relevancy. This relevancy (number and their rankings) helps determine your websites’ popularity and corresponding ranking.
Certain search engines cannot read text within frames and require that a <NOFRAMES> tag be used for this text.