to stop unwanted material while in the search indexes, webmasters can instruct spiders never to crawl specified information or directories from the regular robots.txt file in the root directory with the area. Moreover, a website page is often explicitly excluded from a search engine's database by utilizing a meta tag unique to robots (generally ).