Fix guides · SEO
How to fix “robots.txt is not valid”
The robots.txt file tells search engines which parts of your site to crawl. A broken one can block your whole site or confuse crawlers. How to fix it on Shopify, WordPress, Webflow, Wix, Squarespace and Next.js.
- Time to fix
- 10–30 minutes
- Skill needed
- Copy and paste
- Who notices
- Everyone searching for you
Why it matters
Search engines read yourdomain.com/robots.txt before crawling. It lists which paths they may or may not visit, and usually where your sitemap is.
Lighthouse flags lines it can't understand (a typo in a directive, a rule before any User-agent line, a sitemap that isn't a full URL) and files that return a server error, which can make search engines pause crawling altogether.
“These audits showed that 8.43% of desktop pages and 7.40% of mobile pages failed the tool’s check for valid robots.txt files.”
Before you start
- Open yourdomain.com/robots.txt and expand the issue in your report: it lists each invalid line.
- Every group starts with a User-agent line, followed by Allow or Disallow lines that start with / or *. Sitemap lines need a full URL.
- Be careful: "Disallow: /" blocks your entire site. robots.txt doesn't hide pages from search results; use noindex for that.
Fix it on your platform
Shopify
- Shopify generates a robots.txt that works for most stores. To change it, add a robots.txt.liquid template in your theme's Templates folder.
- If you already have one, compare it with Shopify's default (save a copy of the current /robots.txt first) and fix or remove custom rules. Use the robots Liquid objects rather than plain text where you can.
WordPress
- WordPress serves a virtual robots.txt unless a real robots.txt file exists in your site's root folder. Check whether your host or an SEO plugin has created one, and fix it there.
- SEO plugins usually offer a robots.txt editor; correct the flagged lines or reset to their default.
Webflow
- Go to Site settings → SEO → Indexing, correct the rules in the robots.txt field (each group starting with User-agent), then save and publish.
Wix
- Open the Robots.txt Editor in your dashboard's SEO tools, fix the flagged lines, or click Reset to Default to restore Wix's original file.
Squarespace Managed by the platform
- All Squarespace sites use the same robots.txt, and you can't edit it. If Lighthouse flags it, contact Squarespace support.
- Squarespace notes that Search Console warnings about its restricted internal pages (such as /config/) are normal and can be ignored.
Next.js
- Generate robots.txt from app/robots.ts so it's always valid, and remove any public/robots.txt that conflicts.
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
return {
rules: { userAgent: "*", allow: "/", disallow: "/account/" },
sitemap: "https://example.com/sitemap.xml",
};
}Any other site
- Put robots.txt at the root of your domain and keep it simple.
User-agent: *
Disallow: /cart
Disallow: /account
Sitemap: https://example.com/sitemap.xmlCheck that it's fixed
- Open yourdomain.com/robots.txt in a browser: it should load and match what you expect.
- Search Console's robots.txt report shows how Google read the file. Re-run your StudioLNC audit too.
Find every issue like this on your site
Free audit of performance, accessibility, SEO and security, with fixes for your platform.