Automatically crawl your website so your agent can reference its content.Documentation Index
Fetch the complete documentation index at: https://docs.usecrow.ai/llms.txt
Use this file to discover all available pages before exploring further.
Setup
- Enter your website URL (e.g.,
https://docs.yourcompany.com) - Click Ingest
- Crow crawls linked pages on the same domain
What Gets Crawled
Included:- Publicly accessible pages
- Pages linked from starting URL
- Same domain only
- Password-protected pages
- Pages blocked by robots.txt
- External domains
- JavaScript-rendered content
Re-indexing
When content changes:- Go to Knowledge Base
- Enter URL again
- Click Ingest
Tips
- Use your docs subdomain for focused content (
docs.yourcompany.com) - Re-index after major content updates
- Test in Sandbox to verify content is accessible