A robots.txt file is a special file that tells bots what content they should avoid downloading from a website.

Note that a robots.txt file is not enforced, i.e. it is up to the bot whether it should abide by what it says.

By default, Silktide purposely does not respect the robots.txt file.

This is because for most purposes we are testing whether a website functions correctly for end-users, and in our experience, many robots.txt files inhibit this. Many block user-facing content that would otherwise not be tested.

Note that even Google doesn’t fully abide by robots.txt, for similar reasons.

If you are sure you wish to enforce your robots.txt file, you can copy the rules into the settings for your website(s). Contact us and we’ll be happy to set this up for you.