MYWOT

Child safety

Excellent

Confidence

43%

Trustworthiness

Excellent

Confidence

48%

MALICIOUS CONTENT INDICATORS

Availability or unavailability of the flaggable/dangerous content on this website has not been fully explored by us, so you should rely on the following indicators with caution.

Robotstxt.org most likely does not offer any malicious content.

  • Siteadvisor

    Safety status

    N/A

  • SAFEBROWSING

    Safety status

    Safe

  • Phishtank

    Safety status

    N/A

Secure connection support

HTTPS

Robotstxt.org provides SSL-encrypted connection.

ADULT CONTENT INDICATORS

Availability or unavailability of the flaggable/dangerous content on this website has not been fully explored by us, so you should rely on the following indicators with caution.

Robotstxt.org most likely does not offer any adult content.

Audience

  • Bounce rate

    The accuracy of the provided data is based on the latest estimates available to us and can significantly differ from the real-life website stats, so should be taken with caution.

  • Time on site

    The accuracy of the provided data is based on the latest estimates available to us and can significantly differ from the real-life website stats, so should be taken with caution.

    01:39

    AVG

  • Gender

    The accuracy of the provided data is based on the latest estimates available to us and can significantly differ from the real-life website stats, so should be taken with caution.

    Men

    95%

    Women

    26.5%

Popular pages

  • The Web Robots Pages

    In a nutshell You can use a special HTML <META> tag to tell robots not to index the content of a page, and/or not scan it for links to follow. For example: <META NAME="ROBOTS" CONTENT="NOINDE...

  • The Web Robots Pages

  • The Web Robots Pages

    Martijn Koster, 1993 This document contains some suggestions for people who are thinking about developing Web Wanderers (Robots), programs that traverse the Web. Reconsider Are you sure you rea...

  • The Web Robots Pages

    Status of this document This document represents a consensus on 30 June 1994 on the robots mailing list (robots-request@nexor.co.uk), between the majority of robot authors and other people with an in...

  • The Web Robots Pages

    User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /~joe/ In this example, three directories are excluded. Note that you need a separate "Disallow" line for every URL prefix you want t...

  • The Web Robots Pages

    The Web Robots Pages Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content,...