in order to avoid unwanted content material during the search indexes, site owners can instruct spiders not to crawl particular documents or directories throughout the conventional robots.txt file in the foundation Listing in the domain. In addition, a page could be explicitly excluded from the search engine's databases by using a meta tag unique t