Prevent Search Engine Crawling of your WordPress Website

Working on a staging WordPress website for which you want to prevent URL indexing and its visibility of its pages on the search engine? This guide will help to block such crawling bots on your website. There are certain other reasons for preventing crawlers on the website like project management or internal guide within the company. This can be done with simple steps discussed below.

Procedure

1. Log in to WordPress wp-admin Dashboard
2. Visit Settings  > Reading and check the box next to the Search Engine Visibility option and save it.
   
3. Login to FTP or File Manager and create/edit robots.txt file in the website’s root directory with the following content.
   User-agent: *
Disallow: /
   These above lines will ask robots not to index your website’s pages.
4. Add the no-index nofollow headers rules in .htaccess file and save it.
   Header set X-Robots-Tag "noindex, nofollow"

With these small changes we can stop the crawlers to index the webpages from your website. By this way we are telling the site crawlers eg google, bing to not crawl the website for any link indexing. However, it’s completely upto the site to crawl or not.

You can use directly the 3rd step if you wish (WordPress admin dashboard has inbuilt option, however, it’s good to manually verify the same) so for any type of website be in HTML based or PHP etc.  The same can be edited in more ways to prevent crawling of any particular folder if you want eg. if you added a subdomain in another folder /example , you can add that in Disallow URL instead of only root path i.e / . To prevent media files getting searched in google or indexed, this can be used with mentioned the media file folder.

 

Once you are done with the development work or you want to index the site, you can remove these settings so that search engines will start crawling.