What is a Robots.txt file?
A robots.txt file is a special file that tells bots what content they should avoid downloading from a website.
Note that a
robots.txt file is not enforced, i.e. it is up to the bot whether it should abide by what it says.
Does Silktide respect robots.txt?
By default, Silktide purposely does not respect the
This is because for most purposes we are testing whether a website functions correctly for end-users, and in our experience, many
robots.txt files inhibit this. Many block user-facing content that would otherwise not be tested.
Note that even Google doesn’t fully abide by
robots.txt, for similar reasons.
How to enforce robots.txt
If you are sure you wish to enforce your robots.txt file, you can copy the rules into the settings for your website(s). Contact us and we’ll be happy to set this up for you.