Link analysis home page
Part I: Theory
1. Introduction
2. Crawlers and search engines
3. Theoretical perspectives
4. Sampling & correlations
Part II: Web structure
5. Link structures in the web graph
6. Content structure of the web
Part III: Academic links
7. Universities – link types
8. Universities - link models
9. Universities - international
10. Departments and disciplines
11. Journals and articles
Part IV: Applications
12. Site design & search engines
13. Health check for universities
14. Personal home pages
15. Academic network analysis
16. Business web sites
Part V: Tools and techniques
17. Search engines & Archive
18. Personal crawlers
19. Data cleansing
20. Cybermetrics database
21. Embedded link analysis
22. Social network analysis
23. Network visualisation
24. Academic web indicators
Part VI: Summary
25. Summary & future directions
26. Glossary
Online Appendix
Ethical issues for crawlers

Reviews of this book

- 2009 follow-up book:
Introduction to Webometrics

  

Part V: Tools and Techniques

18. Personal Crawlers and SocSciBot

SocSciBot and Resources

  • The SocSciBot home page including links to free downloads of:
    • The crawler SocSciBot
    • The SocSciBot Tools program for link analysis of SocSciBot crawls
    • Several tutorial introductions to SocSciBot and SocSciBot Tools, and brief user guides
    • Cyclist: a search engine for SocSciBot crawl
    See also Webometric Analyst - not a crawler but able to peform link analysis with data from search engines, and raw data from search engines plus some processing is available online here http://tools.issuecrawler.net/beta/scrapeGoogle/.

Other Personal Crawlers

Customisable crawlers (for programmers only)

Ready-made personal crawlers

  • See online collections (e.g., tucows) of mainly shareware personal crawlers (offline browsers).