Automatically crawl your website so your agent can reference its content.
Setup
- Enter your website URL (e.g.,
https://docs.yourcompany.com)
- Click Ingest
- Crow crawls linked pages on the same domain
What Gets Crawled
Included:
- Publicly accessible pages
- Pages linked from starting URL
- Same domain only
Not included:
- Password-protected pages
- Pages blocked by robots.txt
- External domains
- JavaScript-rendered content
Re-indexing
When content changes:
- Go to Knowledge Base
- Enter URL again
- Click Ingest
Re-indexing replaces previous content.
Tips
- Use your docs subdomain for focused content (
docs.yourcompany.com)
- Re-index after major content updates
- Test in Sandbox to verify content is accessible
