Robots.txt Generator

User Agents (Bots)

All Bots (*) Googlebot Bingbot Googlebot-Image Googlebot-News Yahoo Slurp DuckDuckBot Baiduspider YandexBot Facebot Alexa (ia_archiver) AhrefsBot

Sitemap URL

Crawl Delay (seconds)

Optional. Sets a delay between successive crawl requests.

Disallowed Paths

One path per line. These paths will be blocked from crawling.

Allowed Paths

One path per line. These paths will be explicitly allowed.