To avoid undesirable material inside the search indexes, webmasters can instruct spiders to not crawl specified documents or directories in the common robots.txt file in the root directory of the domain. Furthermore, a web page may be explicitly excluded from a internet search engine's database by using a meta tag https://torontolawfirmitsupport.com