Core Architecture & Crawl Topology:
- Goal: Scalable, polite, and fault-tolerant crawling of billions of URLs with deduplication and high freshness.
1. Frontier & URL Scheduling:
- URL Frontier manages pending URLs: prioritized by page rank and freshness necessity.
- Politeness Module: Enforce strict host-level rate limiting using a two-tier queue system (FIFO priority queues mapped to host-specific delay queues) so a single domain is never hit faster than its robots.txt guidelines.
2. Distributed Crawl Workers:
- Multi-threaded worker pods pull URLs, download robots.txt rules, execute DNS lookups via local DNS caches, and fetch raw HTML.
- Handle redirects, socket timeouts, and HTTP status codes gracefully.
3. Parsing & Duplicate Detection:
- Content Parser extracts text and outbound links.
- Duplicate Content Detection: Compute 64-bit SimHash or MinHash of document content to detect mirror/duplicate pages without full string comparisons.
- URL Deduplication: Maintain a Bloom Filter backed by a distributed database (Bigtable or Cassandra) to check if a URL has already been visited.
4. Storage & Indexing Feed:
- Store raw compressed web pages in a distributed object store (S3 or HDFS) and stream parsed text tokens into an indexing pipeline via Kafka.