How To Create Robots .txt File For WordPress
The Robots Exclusion Standard was created in 1994 to ensure that site owners could recommend search engines how you can crawl your site . The main difference being that the robots .txt file will stop search engines from viewing a webpage or directory , meanwhile the robots meta tag merely controls whether it is indexed .Setting a robots .txt file in the root of the domain enables you to stop search engines indexing sensitive files as well as directories . For instance , you can stop the search engines from crawling the pictures folder or from indexing a PDF file which is positioned in a secret folder .
Leading searches will stick to the rules which you set . Remember , in spite of this , that the guidelines you explain in the robots .txt file will never be enforced . Crawlers for risky software and also poor search engines may not adhere to the guidelines and also index whatever they want . Fortunately , major search engines stick to the standard , includingGoogle , Bing , Yandex , Ask , and Baidu .
In this post , I would really like to show you How To Create Robots .txt File For WordPress and also provide you with what files as well as directories you might want to hide from search engines for a WordPress site .
The Standard Guidelines of the Robots Exclusion Standard
A robots .txt file could be produced in secs . All you need to do is open a text editor and also save a blank file as robots .txt . As soon as you have added some guidelines to the file , save the file and also upload it to the root of the domain i .e . www .yourwebsite .com/robots .txt . Please make sure you upload robots .txt to the root of the domain -even though WordPress is installed in a subdirectory .
I would recommend file permissions of 644 for the file . Most web hosting setups would setup that file with those permissions as soon as you upload the file . Its also wise to check out the WordPress plugin WP Robots Txt- that allows you to change the robots .txt file instantly by means of the WordPress admin area . It is going to save you from having to re-upload the robots .txt file by means of FTP each and every time you change it .
Search engines will look for a robots .txt file at the root of the domain when they crawl your site . Please bear in mind that another robots .txt file will have to be configured for each subdomain and also for other protocols like https ://www .yourwebsite .com .
It will not take long to obtain the full knowledge of the robots exclusion standard , because there are just a few principles to understand . All these guidelines are generally known as directives .
