Sometimes, when you develop your client website and publish the site as staging site. You want to make sure the site is not going to be crawled or indexed by Google or search engine. Otherwise it is going to looks bad, the main reason is because the site is not live and approved yet. So all the contents should be privately viewed by clients and yourself only.
In this tutorial, you will learn how you can create your own robots file and place it on the root of your website.
To block the site is pretty simple, what you need to do is to create a robots.txt and type in the following text:
User-agent: * Disallow: /
Alternatively you can place this meta tag information on your header meta tags as well
<meta name="robots" content="noindex"/>
If you want to target Google search engine only, you can use the following meta tags.
<meta name="googlebot" content="noindex"/>