Part I: Theory
2. Web Crawlers and Search Engines
REFERENCES WITH LINKS
Chakrabarti, S., Joshi, M.M., Punera, K. & Pennock, D.M. (2002). The structure of broad topics on the Web, WWW2002.
Koster, M. (1994). A standard for robot exclusion.
See also the Ethical Issues for Crawlers section and the Personal Crawlers section.
  
|