Does Robots.txt stop a page from being indexed?

No. Robots.txt mainly controls crawler requests. A page can still be discovered through external references, so page-level indexing controls handle a different task.

Where should a Robots.txt file be placed?

Robots.txt should be placed at the root of the relevant host, such as the main domain. A file in a subfolder does not set rules for the whole host.

Can Robots.txt give different instructions to different crawlers?

Yes. Robots.txt can identify a crawler with a user-agent rule and apply access instructions to that crawler. Supported directives can vary between crawlers.

Glossary

What Is Robots.txt?

Robots.txt is a plain-text file that gives search engine crawlers instructions about which areas of a website they may request. The file sits at the website’s root address and can guide crawler access to folders, files, or URL patterns. Robots.txt does not remove pages from search results or replace page-level indexing controls.

Quick Facts About Robots.txt

Key Takeaways About Robots.txt

It gives crawlers access instructions through text-based rules.
The file belongs at the root of a website.
Robots.txt is not the same as a noindex tag.

How Robots.txt Works

Hands type at a desk around a sheet labeled robots.txt, with laptops, a gavel, and office items around it

Robots.txt uses plain-text directives to identify crawlers and set access rules for selected paths. A crawler reads the file before requesting URLs, then follows the rules it supports. The file can help keep private, duplicate, or low-value areas out of routine crawling.

Related glossary terms: Noindex tag, Crawl budget.

Robots.txt operates at the host level, so each domain or subdomain may have its own file. The file does not guarantee that a blocked URL will disappear from search results, because search engines may learn about that URL through other signals. A noindex tag controls indexing on a page, while XML sitemap files help identify URLs that should be discovered.

Hands hold a paper labeled robots.txt over a wooden desk with laptop, glasses, magnifying glass, and gavel

Expert Note

When we review Robots.txt, we check its location, syntax, and rules against the website’s important sections. We compare blocked paths with crawl data and page-level indexing signals. Our team also checks that staging areas and functional resources are not exposed to unnecessary crawler requests.

Robots.txt in Practice: A Real-World Example

A Sydney law firm may use Robots.txt to limit crawler access to internal search results and account paths. The file can leave service pages, location pages, and the XML sitemap available for discovery.

Related Services

Technical SEO

Learn More

Related Terms

Noindex tag

A Noindex tag is an instruction that asks search engines not to include a webpage in…

Crawl budget

Crawl budget is the attention and access that a search engine assigns to discovering and revisiting…

LawyerSeoMarketing.com.au

Have Questions About Robots.txt?

Contact LawyerSeoMarketing.com.au for practical guidance on Robots.txt and related lawyer seo marketing work in Sydney.

Contact Our Experts