What is meant by a consumer-agent in robots.txt is the particular form of World wide web crawler that you give the command to crawl. This World-wide-web crawler generally varies according to the search engine utilised. You might have use of a great number of free robots.txt generator tools online, most https://seotoolstube.com/