10. Gzip Compression: Help Gzip compression on your own server to reduce the sizing of documents transferred in between the server and the consumer’s browser. txt file is then parsed and may instruct the robotic as to which web pages usually are not to become crawled. Like a internet search https://www.youtube.com/watch?v=aKirkq1uHGM