How to Edit Your Shopify Robots.txt File for Better SEO

James Johnson

Owner & Founder

SOAR Digital

With the recent update to Shopify in June 2021 it is now possible to update your very own custom Shopify robots.txt file!

So, why is this important for your Shopify store...?

It's going to make it easier for Google to navigate through your site, prioritise important pages and stop wasting precious crawl budget on thin pages that are created across your site.

This has been a long awaited feature which has been requested time and time again within the SEO community. For many SEO-savvy store owners this may have even swayed them to choose another CMS platform over Shopify such as Magento or Big Commerce.

So trust me when I say it's a pretty big deal for Shopify's technical SEO setup that it's now possible to update and edit it yourself.

If you're really looking to take your technical SEO to the next level, then you're going to want to read this....

Don't be intimidated if this is your first time editing a robots.txt file. I'm going to walk you through step-by-step exactly what to do with lot's of images along the way to keep it nice and visual. It's fairly straightforward... I promise!

So... let's get stuck in!

Firstly... What is a Robots.txt file?

A robots.txt file is a way for a website owner to communicate with search engine crawlers about what pages they would like the crawler to access on their site and which pages are not allowed.

It's essentially a set of instructions that shows Google the most helpful information on your site and prevents the crawler from discovering pages with sensitive information such as checkouts and login dashboards.

Here's what a Shopify robots.txt file looks like:

How to find your Shopify Robots.txt File

All Shopify stores come with a default robots.txt file so all you need to do to find yours is:

 Enter your store URL followed by:

 /robots.txt

Like this:

www.examplestore.co.uk/robots.txt

Why is the Robots.txt file important for SEO?

A robots.txt file is important for SEO because Google has a set 'crawl budget' for each website on the internet which is essentially how often Google will visit your website to check for new content and page updates. If you are allowing Google to access all of the thin pages on your site then you're wasting your crawl budget on pages that shouldn't show in the search results.

We need to make it as easy as possible for Google to understand which pages on your site are the most important and therefore deserve to be ranked in the SERPs (Search Engine Results Page). But more importantly, we need to make it very clear which pages we DON'T want to include in the search results.

With Shopify sites, there are often hundreds of 'thin' pages which are created as a results of filters and tags so it's our job as website owners to tell Google not to waste its time trawling through hundreds or thousands of these irrelevant, thin pages and instead divert the attention to our core collection pages, product pages and blog posts.

This is also achieved with a sitemap.xml feed which tells Google the important pages on your site that you DO want to rank, but Shopify does a pretty good job at generating this for you out of the box. Therefore, the focus in this article is to make proper use of the robots.txt file to tell Google which pages you DON'T want to be crawled, indexed and ranked.

Is Shopify's robots.txt Bad For SEO?

Out of the box Shopify does come with a default robots.txt file that covers most of the essentials. Although, it's not perfect.

With Shopify's default txt file, Google is still able to crawl:

1. Tagged collection pages
2. Tagged blog post categories
3. 'Shop all' page
4. 'Vendor' pages
5. 'Product type' pages

This can lead to issues with crawl budget across your site.

Previously, we could use canonical code to help Google understand which of these pages is the proper URL that deserves to be ranked. Although, canonicals don't always work as it's up to Google's discretion whether they use them or not. Plus, Google actually has to crawl the page before it 'reads' the code that tells it to no longer index this page.

Whereas the robots.txt file is absolute so there's no chance of Google misinterpreting which pages need to be crawled and which pages should be excluded from the search results. You do have to tread carefully because one mistype in the txt file can lead to entire sections of your site disappearing from the search results. However, in this guide I'll walk you through the right way to set it up.

Issue 1: Tagged Collection Pages

One of the most common problems we see time and time again with Shopify stores is an improper use of 'tagged' collection pages. If your collection pages use 'tags' to filter the search results (such as size, colour, price etc) then your site will create hundreds of duplicate pages for each of these tags.

For example:

Main collection page:

examplesite.com/collections/baking-equipment

'Tagged' versions:

examplesite.com/collections/baking-equipment/bread

examplesite.com/collections/baking-equipment/free-from

examplesite.com/collections/baking-equipment/gifts

examplesite.com/collections/baking-equipment/organic

examplesite.com/collections/baking-equipment/sourdogh

Issue 2: Tagged blog post categories

Ever tagged one of your blog posts before? This is great for user experience as visitors can find related topics but unfortunately this creates a blog category page for each new tag. These can all be crawled by Google and might be considered 'thin' or duplicate pages.

For example:

examplesite.com/blogs/news/tagged/funny

You can find your tagged blog posts by searching in Google:

site:yoursite.co.uk/blogs "tagged"

It should look something like this:

Issue 3: 'Shop all' collection page

This is often a useful page from a user experience standpoint, because your visitors can view your entire product catalogue in one seamless page. Although, it's often not properly optimised for any keyword since it's such a general range of products. You're much better off having specific collection pages targeting specific keywords and therefore you might choose to exclude this page from the search results.

Example shop all page:

examplesite.com/collections/all

Issue 4: 'Vendor' pages

Shopify auto-generates pages based on your 'Vendors' which you can't properly optimise. If you do want to create pages for your Vendors I would suggest creating a collection page manually instead and having the 'Vendor' pages excluded from the search results to keep things clean. This way you can properly optimise each page with relevant H1 headings, meta data & descriptive content.

Example vendor page:

examplesite.com/collections/vendors?q=Nike

Issue 5: 'Product type' pages

Shopify also auto-generates pages based on 'product types' that you assign to individual products. Again, it is preferable to create individual collection pages for each product type and use the robots.txt to stop Google from crawling these auto-generated URLs.

Example product type page:

examplesite.com/collections/types?q=Shoe

So without further ado... here's exactly how to add and update your very own custom robots.txt for Shopify that will prevent search engines from discovering these tagged collection and blog pages.

Steps To Update Your Custom Shopify Robots.txt File

Firstly, you'll need to navigate to your theme by clicking:

Online Store > Themes > Actions > Edit Code



Next, under templates select "Add new Template"

Select "Robots.txt" from the dropdown menu and create the template.

You will now have an editable Robots.txt that we can update an adjust to match your requirements:

Simply copy the code below and paste it in place of your current version:

# we use Shopify as our ecommerce platform

{%- comment -%}

# Caution! Please read https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt before proceeding to make changes to this file.

{% endcomment %}

{% for group in robots.default_groups %}

  {{- group.user_agent -}}

  {% for rule in group.rules %}

    {{- rule -}}

  {% endfor %}

  {%- if group.user_agent.value == '*' -%}

{{ 'Disallow: /collections/all*' }}

{{ 'Disallow: /*?q=*' }}

{{ 'Disallow: /blogs/*/tagged/*' }}

  {%- endif -%}

  {%- if group.sitemap != blank -%}

    {{ group.sitemap }}

  {%- endif -%}

{% endfor %}

This will exclude all of these 5 page types from being crawled and indexed by Google:

1. Tagged collection pages
2. Tagged blog post categories
3. 'Shop all' page
4. 'Vendor' pages
5. 'Product type' pages

Once saved you can visit yoursite.com/robots.txt to check that the new lines have been added.

Final considerations

Once live you can test your robots.txt to see if it's working by visiting the free Google tool here:

https://www.google.com/webmasters/tools/robots-testing-tool

Simply copy a URL into the field and it'll tell you whether or not Google has access to that URL.

Upon checking Google Search Console you might find some more URL's populating in the crawl stats that you don't want Google to crawl and index. If this is the case then you can simply find a common part of the URL to exclude in your robots.txt.

For example, a client of ours had numerous crawl requests for URL's with this structure:

/collections/all?page=2&_pjax=.product-listing-loadmore

So all we had to do was add a new line to the robots.txt which excluded any URL's with 'pjax'. This is how it looked:

{{ 'Disallow: /*&_pjax*' }}

You can also exclude specific pages too, such as:

{{ 'Disallow: /pages/search-results-page' }}

Anyway, I hope this helps anyone looking to clean up on the technical SEO for Shopify store.

As a Shopify SEO Agency, we're here if you'd like us to take a look at your Shopify store to increase your organic traffic and sales.

For more information on Shopify why not check out our recent Shopify SEO case study or our last blog post on how to optimise your Shopify collection pages with SEO.

Stay tuned for more guides, tips & info coming soon!