All Collections
The Silktide Platform
Does Silktide respect robots.txt?
Does Silktide respect robots.txt?
Julia Voortman avatar
Written by Julia Voortman
Updated over a week ago

What is a Robots.txt file?

A robots.txt file is a special file that tells bots what content they should avoid downloading from a website.

Note that a robots.txt file is not enforced, i.e. it is up to the bot whether it should abide by what it says.

Does Silktide respect robots.txt?

By default, Silktide purposely does not respect the robots.txt file.

This is because for most purposes we are testing whether a website functions correctly for end-users, and in our experience, many robots.txt files inhibit this. Many block user-facing content that would otherwise not be tested.

Note that even Google doesn’t fully abide by robots.txt, for similar reasons.

How to enforce robots.txt

If you are sure you wish to enforce your robots.txt file, you can copy the rules into the settings for your website(s). Contact us and we’ll be happy to set this up for you.

Did this answer your question?