Martin advises using “noindex” to allow search engines to read your web page’s content while stopping it from appearing in search results. Split says this will enable users to see a page you don’t want displayed in search results.
Disallow robots.txt file recap
Adding “disallow” to your site’s robots.txt file denies search engines access to specific files, URLs, or sensitive website areas, resulting in search engines not crawling or indexing the content.
When to use it:
Splitt advises using “disallow” to completely block search chile mobile numbers list engines from crawling and indexing a page, saying it helps protect sensitive data information and not displaying non-relevant content like log-in or thank-you pages, which can help preserve your site’s crawl budget.
Split says to avoid this mistake
Splitt recommends not using “disallow” in the robots.txt file and “noindex” directive in the page’s meta tag for the same page, saying it is a common mistake and can cause search engines to index the page but with limited information.
Instead, Martin advises using the “noindex” directive and not the “disallow” directive on the page to stop it from appearing in search results.
You can review your websites disallow and noindex directives using Google’s Search Console to ensure search engines interpret them correctly.