https://thomasarie.com/search-engine-robots-and-bandwidth/
Search Engine Robots and Bandwidth