How to Stop Search Engines From Crawling a WordPress Site

How to Stop Research Engines from Crawling a WordPress Website


Not long ago, one of our people asked us how they can end lookup engines from crawling and indexing their WordPress internet site.

There are a number of situations when you would want to prevent lookup engines from crawling your web page and listing it in lookup final results.

In this article, we will show you how to stop search engines from crawling a WordPress web page.

How to Stop Search Engines From Crawling a WordPress Site

Why End Search Engines From Crawling a WordPress Website?

For most WordPress internet websites, lookup engines are the biggest resource of visitors. You could talk to, why would everyone want to block lookup engines?

Below are some conditions when you won’t want research engines to index your web page:

  • There are also quite a few people who use WordPress to make private weblogs. They don’t want them indexed by lookup engines due to the fact they are private.
  • You can use WordPress for job management or an intranet. In these circumstances, you won’t want your inner paperwork to be publicly accessible.

A prevalent misconception is that if you never have links pointing to your domain, then lookup engines will possibly never ever discover your web site. This is not absolutely genuine.

For case in point, there may possibly be links pointing to your web site because the area identify was formerly owned by another person else. Also, there are countless numbers of web pages on the world-wide-web that basically list domain names. Your web page may possibly surface on just one of those.

With that becoming claimed, let’s acquire a look at how to cease research engines from crawling your site. We’ll protect 4 procedures:

Approach 1: Asking Lookup Engines not to Crawl Your WordPress Site

This is the easiest strategy but does not absolutely shield your web site from becoming crawled.

WordPress will come with a developed-in attribute that enables you to instruct research engines not to index your internet site. All you want to do is stop by Options » Reading through and look at the box next to ‘Search Engine Visibility’.

Search Engine Visibility Setting in WordPress

When this box is checked, WordPress provides this line to your website’s header:


WordPress also modifies your site’s robots.txt file and adds these strains to it:

Person-agent: *
Disallow: /

These strains check with robots (internet crawlers) not to index your webpages. Nevertheless, it is absolutely up to look for engines to acknowledge this request or ignore it. Even while most research engines respect this, there’s however a probability that some internet pages or visuals from your website could get indexed.

If you want to make it impossible for research engines to index or crawl your site, then you will will need to password guard your WordPress website employing Methods 3 or 4.

Process 2: Inquiring Look for Engines not to Crawl Personal Web pages

You could want lookup engines to crawl and index your web-site, but not incorporate particular posts or webpages in lookup final results web pages.

The least difficult way to do that is applying the All in A single Web optimization (AIOSEO) plugin. It is the most effective Website positioning instrument for WordPress and is trusted by over 3 million organizations.

For this tutorial, we’ll be working with the AIOSEO free edition as it consists of the Web optimization Analysis software. There is also a high quality edition of AIOSEO that features a lot more features like sitemap equipment, redirection manager, schema markup, robots.txt editor, and more.

The to start with thing you are going to have to have to do is install and activate the AIOSEO plugin on your web-site. You can master how to set up and configure the plugin by following our step by step manual on how to established up All in 1 Search engine optimisation for WordPress.

At the time the plugin is set up, you can use it to request search engines not to index specified posts and internet pages. Once more, it is absolutely up to look for engines to acknowledge this ask for or ignore it.

Simply just edit the submit or webpage that you do not want to be indexed. You want to scroll down to AIOSEO Settings at the bottom of the WordPress editor and then click on the ‘Advanced’ tab.

AIOSEO Advanced Settings

Notice that the post is employing the default robots settings. To adjust this, you want to switch the ‘Use Default Settings’ toggle to the off position.

Asking Search Engines Not to Index a Single Page

Now you can click the ‘No Index’ checkbox. When the post is posted or updated, research engines will be questioned not to index it.

Method 3: Password Preserving an Entire Internet site Utilizing cPanel

If your WordPress web hosting supplier gives cPanel obtain to manage your web hosting account, then you can guard your whole web page working with cPanel. All you have to do is log in to your cPanel dashboard and then click on the ‘Directory Privacy’ icon in the ‘Files’ segment.

Using cPanel's Directory Privacy Feature

Following, you need to come across the folder in which you installed WordPress. Usually, it is the public_html folder. After that, you will need to click on the ‘Edit’ button next to that folder.

Edit the Root Folder

Note: If you have various WordPress web sites mounted beneath general public_html directory, then you need to simply click on the public_html url to look through those internet sites, and then edit the folder for the web-site you want to password defend.

This provides you to a display screen wherever you can convert on password safety.

Just check out the box that suggests ‘Password safeguard this directory’ and simply click the ‘Save’ button. If you like, you can also customize the identify for the secured listing.

Check the Box to Password Protect This Directory

You will see a affirmation message saying that the access permissions for the listing have been improved.

Upcoming, you need to click on the ‘Go Back’ button.

cPanel's Directory Privacy Success Message

You’ll be taken to a screen exactly where you can make a username and password that will will need to be used when accessing this listing.

You require to enter a username and password and then confirm the password. Make certain to notice your username and password in a secure put, such as a password manager app.

Add a Username and Password

At the time you click on the ‘Save’ button you have successfully additional password defense to your WordPress web-site.

Now, whenever a consumer or look for motor visits your website they will be prompted to enter the username and password you made before to see the web site.

A Username and Password Are Now Needed to Access Your Website

Method 4: Password Defending WordPress With a Plugin

If you are employing a managed WordPress internet hosting alternative, then you might not have accessibility to cPanel. In that situation, you can use a WordPress plugin to password safeguard your internet site.

In this article are the two most popular alternatives:

  1. SeedProd is the #1 coming quickly and maintenance method plugin for WordPress, applied on over 800,000 websites. It will come with comprehensive obtain management and permissions features that you can use to conceal your internet site from absolutely everyone including search engines. We have a stage by move how-to guide for SeedProd.
  2. Password Secured is a really uncomplicated way to password secure your WordPress site with a one password (no user generation desired). See our step by step guidebook on how to password secure a WordPress web page.

We hope this short article helped you quit look for engines from crawling or indexing your WordPress site. You may also want to study how to velocity up your internet site general performance, or see our greatest action by stage WordPress safety tutorial for newcomers.

If you preferred this write-up, then remember to subscribe to our YouTube Channel for WordPress video tutorials. You can also obtain us on Twitter and Facebook.



Leave a Reply

Your email address will not be published.