Now showing items 1-1 of 1

    • IRLbot: design and performance analysis of a large-scale web crawler 

      Lee, Hsin-Tsang (Texas A&M University, 2008-10-10)
      This thesis shares our experience in designing web crawlers that scale to billions of pages and models their performance. We show that with the quadratically increasing complexity of verifying URL uniqueness, breadth-first ...