Go package net/http for web scraping or spidering. Understanding of robots.txt https://support.google.com/webmasters/answer/6062608?hl=en