Declaring your sitemap in robots.txt

The robots.txt file is the first thing a crawler reads when it visits your site. One line in it is enough to point every search engine to your sitemap, even those where you never created an account.

WorldmapPDFFree world maps in PDF, PNG and JPG

What robots.txt is

robots.txt is a text file placed at the root of the domain, at an address such as https://www.example.com/robots.txt. It tells crawlers which parts of the site they may visit. The rules were used informally for decades before being written down as an internet standard, RFC 9309, in 2022. The Sitemap line comes from the sitemaps.org protocol, and Google, Bing, Yandex and the other major engines all read it.

robots.txt file with two Sitemap lines
A robots.txt file that declares two sitemaps

The syntax

User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml

Examples for common situations

A simple site with one sitemap:

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

A site with separate sitemaps for pages and images:

User-agent: *
Disallow: /cart.php
Sitemap: https://www.example.com/sitemap-pages.xml
Sitemap: https://www.example.com/sitemap-images.xml

A large site with an index file:

User-agent: *
Disallow: /search/
Sitemap: https://www.example.com/sitemap_index.xml

Mistakes that break the link

Is robots.txt enough, or should I also submit the sitemap?

The two methods complement each other. The robots.txt line works for every crawler, with no account and no verification. Submitting the sitemap in Google Search Console and Bing Webmaster Tools gives you something extra: a report showing whether the file was read, how many URLs were found and which ones caused errors. For a site you care about, do both. Our guide on submitting a sitemap to Google and Bing walks you through the second part.

Checking your robots.txt

Open the address of your robots.txt in a browser and read it line by line. Google Search Console also has a robots.txt report that shows the version Google last downloaded and any line it could not understand. After a change, it can take up to a day before Google uses the new version, because the file is cached.

Frequently asked questions

Where do I put the Sitemap line in robots.txt?

Anywhere in the file. It does not belong to a User-agent group, so placing it at the end is the clearest choice.

Can I list several sitemaps in robots.txt?

Yes, add one Sitemap line per file. For many files, point to a single sitemap index instead.

Can my sitemap be hosted on another domain?

It is possible when both sites are verified in Search Console or when robots.txt on the main domain points to it, but hosting it on the same domain avoids any doubt.

Does robots.txt stop a page from appearing in Google?

Not reliably. It stops crawling, but a blocked page can still be indexed from links. Use a noindex tag to keep a page out of the results.

Sitemap guides

What is an XML sitemap?The format, the tags and a complete example you can copy. Submit a sitemap to Google and BingStep by step in Search Console and Bing Webmaster Tools. Sitemap index files for large sitesHow to split a big site into several sitemaps. Sitemap best practicesWhat to include, what to leave out, and common mistakes. Sitemap FAQShort answers to the questions webmasters ask most. How sitemapauto.com worksWhat happens between your URL and your sitemap.xml file.

Another free tool we built

Need a strong password?

Createpass.net generates secure passwords online, free and instantly. Create up to 50 passwords at once, choose their length, and decide whether to include digits, letters and special characters - with the option to leave out look-alike characters that are easy to mistype.

Generate secure passwords on Createpass.net