How to Use Custom Robots.txt with Your Blog? – Blogspot/Blogger SEO

32

Robots.txt is a text file on the server in that you can write which directories, web pages or links should not be included for search results. It means you can restrict search engine bots to crawl some directories and web pages or links of your website or blog. Now custom robots.txt is available for Blogger. In Blogger search option is related to Labels. If you are not using labels wisely per post you should disallow crawl of search link. In Blogger by default, search link is disallowed to crawl. In this robots.txt you can also write the location of your sitemap file. A sitemap is a file located on a server which contains all posts’ permalinks of your website or blog. Mostly sitemap is found in XML format i.e. sitemap.xml.

Custom Robots.txt Blogger Tutorial

Presently Blogger is working on sitemap.xml. Now Blogger is reading sitemap entries through the feed. By this method, most recent 25 posts are submitted to search engines. If you want search engine bots only work on the most recent 25 posts then you should use robots.txt type 1 given below. If you set robots.txt like this, then Google Adsense bot is allowed to crawl the entire blog for best Adsense performance.

Robots.txt Type 1

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techprevue.blogspot.com/sitemap.xml

Note: You may alter Blogspot’s default robots.txt like above.’

Robots.txt Type 2

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techprevue.blogspot.com/feeds/posts/default?orderby=updated

Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain. If you want search engine bots to crawl most recent 500 posts then you should need to use following robots.txt type 2. If already you have more than 500 posts on your blog then you can add one more sitemap line highlighted in red. Robots.txt

Robots.txt Type 3

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=501&max-results=500

Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain.

Mathematical expression for Blogger robots.txt sitemap entries:

Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*0)+1&max-results=m
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*1)+1&max-results=m
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*2)+1&max-results=m
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*3)+1&max-results=m
.
.
.
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*n)+1&max-results=m

Where m=500 and n=1, 2, 3, 4,…, n. If you have organized post labels in a good format and good experience of search engine optimization (SEO) then you can remove the following line –

Disallow: /search

Most important: if you don’t want to submit any blogger post or page to search engines then you can add them like that: For Post add a line like that –

Disallow: /yyyy/mm/post-name.html

For Page add a line like that –

Disallow: /p/page-name.html

Manage Blogger custom robots.txt

For this please follow these steps carefully. Dashboard ›› Blog’s Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes To get a better understanding of this you can take reference of the image given below:

Manage Blogger Custom robots.txt

I hope you’ll get the benefit of this post and get better search engine presence and ranking.

32 COMMENTS

    • All Blogger blogs have sitemap.xml but when you use custom domain on Blogger all go vanish. Blogger will fix this bug soon. I have reported this problem to Blogger. If anything else please write me.

  1. i have a site
    but some post url is not showing in search engine

    and i saw report on WEBMASTER that Robot.txt files blocked 37 urls

    i want, that robot.txt work properly and never block any url.
    i don’t have any knowledge about robots.txt

    PLZ help me how to control robot.txt file

    itz to difficult to understand

    if possible please reply in hindi/hinglish

  2. HI, what if you have more than 50 posts? When I test sitemap with www_kome_cafe/feeds/posts/default?orderby=UPDATED, only 26 submitted status.
    What should I add please?

    How to remove blocked by robots txt in Blogger?

    Do you check help my blog www_kome_cafe please?

  3. Good post but I have one doubt
    I have already submitted my sitemap before 500 posts an now I have crossed 500 posts. My doubt is whether I have to submit the second sitemap or not???

    • Well, Blogspot now has own sitemap. It is an old post. If you need then can submit more than one sitemaps. If you more than 500 posts then can submit.

  4. Can you tell me which XML I should code when my blog is available on google, but it’s not showing the description below? When I clicked on tell me why then it says tha robot.txt is disallowing google to read the post. Or something like that. Can you suggest me which code i should copy?

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.