TF-AI-Crawler Policy

About Our Crawler

TF-AI-Crawler is a web crawler operated by ToothFairyAI PTY LTD. Our crawler is used to index and scrape public website content for the purpose of building AI knowledge bases on behalf of our customers.

Purpose of Crawling

We crawl websites to:

• Build custom AI knowledge bases for our enterprise customers
• Index public content to enable AI-powered search and question-answering
• Help businesses create intelligent assistants trained on their own documentation

What Data We Collect

We only collect publicly available content from websites, including:

• Page text and content
• Page metadata (titles, descriptions)
• Hyperlinks within the crawled domain

We do not collect:

• Personal user data
• Login credentials or private information
• Content behind authentication

Robots.txt Compliance

We fully respect the Robots Exclusion Protocol. Our crawler will always check and comply with your robots.txt directives before crawling any website.

How to Allow Our Crawler

To explicitly allow TF-AI-Crawler to access your website, add the following to your robots.txt file:

User-agent: TF-AI-Crawler
Allow: /

How to Block Our Crawler

To prevent TF-AI-Crawler from accessing your website, add the following to your robots.txt file:

User-agent: TF-AI-Crawler
Disallow: /

Block Specific Paths

To block specific paths only:

User-agent: TF-AI-Crawler
Disallow: /private/
Disallow: /admin/

Crawler Identification

Our crawler identifies itself with the following User-Agent string:

TF-AI-Crawler/1.0 (+https://toothfairyai.com/crawler)

Crawl Rate and Behavior

• We crawl at a respectful rate to avoid overloading servers
• We honor crawl-delay directives in robots.txt
• We do not crawl content behind authentication
• We do not attempt to bypass access restrictions

Request Removal

If you believe our crawler has accessed content it should not have, or if you would like us to remove your content from our index, please contact us at:

Email: support@toothfairyai.com

Please include:

• The URL(s) in question
• Your relationship to the website (owner/administrator)
• The reason for the removal request

We will process removal requests within 5 business days.

Contact Us

For any questions about our crawling practices or to discuss allowing our crawler for your business:

ToothFairyAI PTY LTD
242 Exhibition St, Melbourne VIC 3000
Email: support@toothfairyai.com
Website: https://toothfairyai.com

Changes to This Policy

We may update this crawler policy from time to time. Any changes will be posted on this page with an updated revision date.

Last updated: March 2026