> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usecrow.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Website

> Crawl and index your website

Automatically crawl your website so your agent can reference its content.

## Setup

1. Enter your website URL (e.g., `https://docs.yourcompany.com`)
2. Click **Ingest**
3. Crow crawls linked pages on the same domain

## What Gets Crawled

**Included:**

* Publicly accessible pages
* Pages linked from starting URL
* Same domain only

**Not included:**

* Password-protected pages
* Pages blocked by robots.txt
* External domains
* JavaScript-rendered content

## Re-indexing

When content changes:

1. Go to [Knowledge Base](https://app.usecrow.ai/sources/files)
2. Enter URL again
3. Click **Ingest**

<Warning>
  Re-indexing replaces previous content.
</Warning>

## Tips

* Use your docs subdomain for focused content (`docs.yourcompany.com`)
* Re-index after major content updates
* Test in Sandbox to verify content is accessible