Link analysis home page
Part I: Theory
1. Introduction
2. Crawlers and search engines
3. Theoretical perspectives
4. Sampling & correlations
Part II: Web structure
5. Link structures in the web graph
6. Content structure of the web
Part III: Academic links
7. Universities – link types
8. Universities - link models
9. Universities - international
10. Departments and disciplines
11. Journals and articles
Part IV: Applications
12. Site design & search engines
13. Health check for universities
14. Personal home pages
15. Academic network analysis
16. Business web sites
Part V: Tools and techniques
17. Search engines & Archive
18. Personal crawlers
19. Data cleansing
20. Cybermetrics database
21. Embedded link analysis
22. Social network analysis
23. Network visualisation
24. Academic web indicators
Part VI: Summary
25. Summary & future directions
26. Glossary
Online Appendix
Ethical issues for crawlers

Reviews of this book

- follow-up book (2009):
Introduction to Webometrics


Part V: Tools and Techniques

18. Personal Crawlers and SocSciBot

SocSciBot and Resources

  • The SocSciBot home page including links to free downloads of:
    • The crawler SocSciBot
    • The SocSciBot Tools program for link analysis of SocSciBot crawls
    • Several tutorial introductions to SocSciBot and SocSciBot Tools, and brief user guides
    • Cyclist: a search engine for SocSciBot crawl
    See also Webometric Analyst - not a crawler but able to peform link analysis with data from search engines, and raw data from search engines plus some processing is available online here

Other Personal Crawlers

Customisable crawlers (for programmers only)

Ready-made personal crawlers

  • See online collections (e.g., tucows) of mainly shareware personal crawlers (offline browsers).