To stay away from unwanted content in the look for indexes, site owners can instruct spiders never to crawl certain files or directories with the common robots.txt file in the root directory from the domain. In addition, a site can be explicitly excluded from a search engine's database through the use of a meta tag certain to robots (ordinarily ).