Robots.txt Generator & Validator

Build crawl rules and check which path Googlebot can access.

Crawl rules

Start with the crawler that should follow these rules.

Sitemaps

Generated robots.txt

0
Ready. The example blocks /admin for all crawlers.

Rule check

    Test a path

    Your crawl rules are created and checked in your browser. Nothing is uploaded to BroBroGo.

    FAQ

    What can this generator add to robots.txt?

    Add a user-agent group, Allow and Disallow paths, and one or more sitemap URLs. The preview is ready to copy into a robots.txt file.

    How does the path checker choose a rule?

    It follows Google’s matching order: the longest matching path wins. If matching Allow and Disallow rules are equally specific, Allow wins.

    Does Disallow remove a page from Google Search?

    No. A blocked page can still appear in search results when other pages link to it. Use noindex, access control, or remove the page when you need it kept out of search.

    XML Formatter / ValidatorFormat, minify and check XML structure in your browser.URL Encoder/Decoder + HTML EntitiesEncode URLs, decode percent escapes, and escape HTML entities.URL Slug GeneratorTurn page titles into clean, hyphenated URL slugs.