Dopious
Senior Member
Founding Member
Sapphire Member
Patron
Bronze Star
Bronze Star
Bronze Star
Bronze Star
Bronze Star
- Joined
- Apr 5, 2025
- Messages
- 1,239
- Reaction Score
- 3,819
Good article on the subject, summarized by AI:
The article, written by Andrew Chan, details his project to crawl one billion web pages in 24 hours for under $500. Inspired by a similar project from 2012, Chan wanted to see how web crawling has changed with modern hardware and the evolution of the web. He ran into several challenges, including CPU bottlenecks from parsing large web pages and the overhead of SSL handshakes, which are now used by the vast majority of websites.
The project was ultimately successful, crawling a billion pages in just over 24 hours for about $462. Chan concludes by reflecting on the surprising amount of the web that is still accessible without executing JavaScript and poses questions for future work, such as how to handle dynamically rendered content.
Source: https://andrewkchan.dev/posts/crawler.html
The article, written by Andrew Chan, details his project to crawl one billion web pages in 24 hours for under $500. Inspired by a similar project from 2012, Chan wanted to see how web crawling has changed with modern hardware and the evolution of the web. He ran into several challenges, including CPU bottlenecks from parsing large web pages and the overhead of SSL handshakes, which are now used by the vast majority of websites.
The project was ultimately successful, crawling a billion pages in just over 24 hours for about $462. Chan concludes by reflecting on the surprising amount of the web that is still accessible without executing JavaScript and poses questions for future work, such as how to handle dynamically rendered content.
Source: https://andrewkchan.dev/posts/crawler.html