PhD Thesis: Effective Web Crawling

•  
• Download the full thesis
• Download the thesis by chapters

Title: Effective Web Crawling 数据挖掘实验室

Author: Carlos Castillo
Advisor: Ricardo Baeza-Yates
Comitee: Mauricio Marin, Alistair Moffat, Gonzalo Navarro, Nivio Ziviani
Year: 2004
数据挖掘研究院

Ph.D. Thesis: EFFECTIVE WEB CRAWLING by Carlos Castillo [4Mb, 180 pags.]

数据挖掘研究院

Thesis divided by chapters. Main chapters are marked in bold.

数据挖掘实验室

• Front matter: cover, indexes, etc.

• Introduction
• Related work, bibliographic review of Web crawling: state of the art in Web crawling, survey.
  
• A new crawling model and architecture: framework and classification of Web crawlers.
• Scheduling algorithms for effective Web crawling: long-term and short-term scheduling.
• Crawling the infinite Web, when to stop crawling: dynamic Web sites can be unbounded. 数据挖掘实验室
  
• Cooperation schemes for Web servers: to improve their representation in the search engine.
• Crawler implementation: algorithms and data structures.
• Web characterization: study of the Chilean Web.
  
• Conclusions

• Appendix: Practical Web crawling problems, Web crawling in practice: practical issues and caveats of Web crawling.

Abstract was published by ACM SIGIR Forum: "Effective Web Crawling (Doctoral Abstract)". ACM SIGIR Forum 55 Vol.39 No. 1, pp. 55-56. June 2005.

数据挖掘研究院

This thesis is part of the WIRE project for developing an open-source Web Information Retrieval Environment. 数据挖掘研究院