TL;DR — Web Crawler

Design a distributed web crawler that can fetch billions of web pages efficiently. The crawler must respect robots.txt rules, enforce politeness policies (not overloading any single host), deduplicate content, and prioritize URLs based on importance and freshness requirements. Key features: Crawl web pages starting from a seed set of URLs. Extract and follow hyperlinks to discover new pages.