10. Gzip Compression: Enable Gzip compression in your server to reduce the dimension of data files transferred amongst the server as well as user’s browser. txt file is then parsed and will instruct the robot concerning which internet pages are certainly not for being crawled. Like a internet search engine https://www.youtube.com/watch?v=7OfVkYHB1OY