Robots.txt is a text file on the server that you can customize for search engine bots. It tells search engine bots which directories, web pages, or links should be indexed or not be indexed in search results. It means you can restrict search engine bots to crawl some directories and web pages or links of your website or blog. Now custom robots.txt is available for Blogspot. In Blogger search option is related to Labels. If you are not using labels wisely, you should disallow the crawl of the search result pages. In Blogger, by default, the search link is disallowed to crawl. In this robots.txt, you can also write the location of your sitemap file. A sitemap is a file located on a server that contains all posts’ permalinks of your website or blog. Mostly sitemap is found in XML format, i.e., sitemap.xml.
Presently Blogger has completed his work on sitemap.xml. Now Blogger is reading sitemap entries through the feed. By this method, the most recent 25 posts are submitted to search engines. If you want search engine, bots, only work on the most recent 25 posts, and then you should use robots.txt type 1 given below. If you set robots.txt like this, then the Google Adsense bot can crawl the entire blog for the best Adsense performance.
Robots.txt Type 1
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: /b Allow: / Sitemap: https://www.techprevue.blogspot.com/sitemap.xml
Note: You may alter Blogspot’s default robots.txt like above.’
Robots.txt Type 2
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: /b Allow: / Sitemap: https://www.techprevue.blogspot.com/feeds/posts/default?orderby=updated
Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain. If you want search engine bots to crawl the most recent 500 posts, you should need to use the following robots.txt type 2. If you already have more than 500 posts on your blog, you can add one more sitemap line highlighted in red. Robots.txt
Robots.txt Type 3
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: /b Allow: / Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500 Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=501&max-results=500
Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain.
Mathematical expression for Blogger robots.txt sitemap entries:
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*0)+1&max-results=m Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*1)+1&max-results=m Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*2)+1&max-results=m Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*3)+1&max-results=m . . . Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*n)+1&max-results=m
Where m=500 and n=1, 2, 3, 4,…, n. If you have organized post labels in a good format and good experience of search engine optimization (SEO), then you can remove the following line –
Most important: if you don’t want to submit any blogger post or page to search engines, then you can add them like that: For Post, add a line like that –
For Page add a line like that –
Manage Blogger custom robots.txt
For this, please follow these steps carefully. Dashboard ›› Blog’s Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes To get a better understanding of this you can take reference of the image given below:
I hope you’ll benefit from this post and get a better search engine presence and ranking.
65 thoughts on “How to Use Custom Robots.txt with Your Blog? – Blogspot/Blogger SEO”
And how to update the blog ?? or it will automatically update ?
It will update automatically…
my blog sitemap is not updating automatically… i will have to resubmit the sitemap to get them new posts indexed.. please help
Submit you sitemap via webmaster tools… take the reference of this video… http://www.youtube.com/watch?v=crSHBa5Xm-s
Most welcome Tharun
what is the basic time that will Google take to consider the new updated robots.txt file
Mohit Blogger @ The Geek Solutions
Very Soon it will be updated.
Hi, I’m a big fan of your blog. I checked your robots.txt – https://www.techprevue.com/robots.txt and it seems like you have a sitemap.xml under your domain. I’m just wondering how did you manage to do this?
To make myself clear, I’m referring to this – https://techprevue.blogspot.com/sitemap.xml
All Blogger blogs have sitemap.xml but when you use custom domain on Blogger all go vanish. Blogger will fix this bug soon. I have reported this problem to Blogger. If anything else please write me.
is this issue has been resolved now?
i am facing the same issue on my blog.
Tks so much.
Most welcome Gayatri!
What’s the use of Disallow: /b
didn’t get that..
Most welcome Sangram! To avoid your preview post crawling…
thanks to a lot …
for this article..
Most welcome MRizal! Please do not spam again!
i have a site
but some post url is not showing in search engine
and i saw report on WEBMASTER that Robot.txt files blocked 37 urls
i want, that robot.txt work properly and never block any url.
i don’t have any knowledge about robots.txt
PLZ help me how to control robot.txt file
itz to difficult to understand
if possible please reply in hindi/hinglish
HI, what if you have more than 50 posts? When I test sitemap with www_kome_cafe/feeds/posts/default?orderby=UPDATED, only 26 submitted status.
What should I add please?
How to remove blocked by robots txt in Blogger?
Do you check help my blog www_kome_cafe please?
submit sitemap like: domain name/sitemap.xml
very helpful article.it really helped me
THat was helpful
Good post but I have one doubt
I have already submitted my sitemap before 500 posts an now I have crossed 500 posts. My doubt is whether I have to submit the second sitemap or not???
Well, Blogspot now has own sitemap. It is an old post. If you need then can submit more than one sitemaps. If you more than 500 posts then can submit.
thanks to a lot …
for this article
Very nice article. Thanks for explaining about robots.txt in a simple manner.
CAN I ASK WHEN I CHECK MY SITE AT UBERSUGGEST, IT’S SAYS THAT I HAVE 1 pages have blocked meta tag robots or X-Robots-Tag HTTP header
Write us an email and send necessary information e.g. screenshot.
well nice post mister Vinay Prajapati for robot.txt
Can you tell me which XML I should code when my blog is available on google, but it’s not showing the description below? When I clicked on tell me why then it says tha robot.txt is disallowing google to read the post. Or something like that. Can you suggest me which code i should copy?
You may send an email with screenshot
I read your blog, this is so awesome, but I have a doubt if we Disallow “/blog/Xyz” page, does it disallow the whole /Blog folder?? I expect you to reply as soon as possible.
Perhaps not! But if you will disallow “/blog”, it will surely disallow “/blog/Xyz”.
Thank you Bhai. Mera Kam ho gaya Bhai thank you so much.
What is User-agent: * function please
Browsers, robots, spiders and any other program that retrieve information from the site can be considered as an User-agent.
Does the Disallow placed on Search not affecting the ranking of the labels article in the list?
Very well described, thank you for this entry. I didn’t know how to set it up on my blogspot to work as it should. You helped me, I set it up as you suggested, now it should be good.
Yeah, it is helpful, many thanks.
Sir. I am Ajit I am Blogger
My site is Blocked 😭 by robot.txt.
Your post is type 1 robot.txt file upload then
My website indexed in search console.
Thank you, sir
Thanks a lot… you really solved my problem.. thanks for such useful information. cheers
Sir my new posts are not indexing till more than 1 week. When i request for inexing in search console and then do live test it shows that url is in google but again when I check it is not in google. I have make baacklinks and ping but faild. Why this problem occurs?
I have gained from this post.
this article support me to add robots.txt
Very useful article!
Sir, What does this command do, or what does it mean? – Disallow: /b
Blogger’s a few dashboard pages come under “/b” thus you may hide them from search engines.
I have less than 500 posts but I use Robots.txt Type 3 method that you described. Is it ok or should I change it?
It is better if you don’t change from your side.
What type of robots.txt should I use if my blog is new and has no post? I’m also getting error message on the sitemap section under my google search console. Please help!
You don’t need to make changes. You should publish articles.
Sir, I have 30 posts in my blog which type robots.txt file I use… please help me
No need to make changes.
bhai……..maine 30 post likhe hai free domain pe blogspot wale…..aur abhi maine usi blog pe new domain name liya to kya muze custom robot . txt change krna padega??
You are not required to play with robots.txt. It only for advanced users.
Is these is correct custom robots.txt let me know
please help reply
Thanks, Vinay bro. It was a very helpful blog especially for new bloggers, I appreciate your hard work, well-done Bro.
You have made this tutorial very easy to understand for your readers as a blogger I am impressed by your writing skills and from your sound knowledge keep going and best of luck with your future posts.
Thank you for the tutorial! It is a big help!
The best article to manage robots.txt on blogspot, thank you!
Very vital information. Thank you for sharing such valuable content. Could you please make a write-up on header tags?
I have a doubt about this “Disallow: /p/page-name.html.” Should the page name of the site be written in place of “page-name”? Or need to be placed as it is?
“/p/page-name.html” is a permalink of a blog page. Please double-check what you are doing.
It is an excellent, informative, and knowledgeable article/post that you have shared with the audience/readers. You are doing a great job of providing helpful information on your website/blog, and I appreciate your efforts.