Fix guides · SEO

How to fix “robots.txt is not valid”

The robots.txt file tells search engines which parts of your site to crawl. A broken one can block your whole site or confuse crawlers. How to fix it on Shopify, WordPress, Webflow, Wix, Squarespace and Next.js.

Time to fix
10–30 minutes
Skill needed
Copy and paste
Who notices
Everyone searching for you

Why it matters

Search engines read yourdomain.com/robots.txt before crawling. It lists which paths they may or may not visit, and usually where your sitemap is.

Lighthouse flags lines it can't understand (a typo in a directive, a rule before any User-agent line, a sitemap that isn't a full URL) and files that return a server error, which can make search engines pause crawling altogether.

“These audits showed that 8.43% of desktop pages and 7.40% of mobile pages failed the tool’s check for valid robots.txt files.”

HTTP Archive Web Almanac 2024, SEO

Before you start

  • Open yourdomain.com/robots.txt and expand the issue in your report: it lists each invalid line.
  • Every group starts with a User-agent line, followed by Allow or Disallow lines that start with / or *. Sitemap lines need a full URL.
  • Be careful: "Disallow: /" blocks your entire site. robots.txt doesn't hide pages from search results; use noindex for that.

Fix it on your platform

Shopify

  1. Shopify generates a robots.txt that works for most stores. To change it, add a robots.txt.liquid template in your theme's Templates folder.
  2. If you already have one, compare it with Shopify's default (save a copy of the current /robots.txt first) and fix or remove custom rules. Use the robots Liquid objects rather than plain text where you can.

WordPress

  1. WordPress serves a virtual robots.txt unless a real robots.txt file exists in your site's root folder. Check whether your host or an SEO plugin has created one, and fix it there.
  2. SEO plugins usually offer a robots.txt editor; correct the flagged lines or reset to their default.

Webflow

  1. Go to Site settings → SEO → Indexing, correct the rules in the robots.txt field (each group starting with User-agent), then save and publish.

Wix

  1. Open the Robots.txt Editor in your dashboard's SEO tools, fix the flagged lines, or click Reset to Default to restore Wix's original file.

Squarespace Managed by the platform

  1. All Squarespace sites use the same robots.txt, and you can't edit it. If Lighthouse flags it, contact Squarespace support.
  2. Squarespace notes that Search Console warnings about its restricted internal pages (such as /config/) are normal and can be ignored.

Next.js

  1. Generate robots.txt from app/robots.ts so it's always valid, and remove any public/robots.txt that conflicts.
app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/", disallow: "/account/" },
    sitemap: "https://example.com/sitemap.xml",
  };
}

Any other site

  1. Put robots.txt at the root of your domain and keep it simple.
robots.txt
User-agent: *
Disallow: /cart
Disallow: /account

Sitemap: https://example.com/sitemap.xml

Check that it's fixed

  • Open yourdomain.com/robots.txt in a browser: it should load and match what you expect.
  • Search Console's robots.txt report shows how Google read the file. Re-run your StudioLNC audit too.

Find every issue like this on your site

Free audit of performance, accessibility, SEO and security, with fixes for your platform.