> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cosmo.humanizing.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Sources

> Import content from your website using URLs, sitemaps, or automated crawling

## Overview

Web sources allow you to import content directly from your website into your agent's knowledge base. This is the most common way to train your agent on existing website content like product pages, documentation, blog posts, and service descriptions.

<Tip>
  Start with your most important pages first. Quality matters more than quantity - a focused knowledge base with relevant content performs better than one filled with irrelevant pages.
</Tip>

## Discovery Methods

The platform offers four methods to discover and import web content:

<CardGroup cols={2}>
  <Card title="Quick Scan" icon="zap">
    Fast domain mapping that quickly discovers pages across your website.
  </Card>

  <Card title="Deep Scan" icon="layers">
    Thorough crawling with advanced options for precise control over what gets imported.
  </Card>

  <Card title="Sitemap Import" icon="file-lines">
    Import URLs directly from your website's sitemap.xml file.
  </Card>

  <Card title="Manual Entry" icon="clipboard">
    Paste specific URLs when you know exactly which pages to import.
  </Card>
</CardGroup>

***

## Quick Scan

Quick Scan is the fastest way to discover pages on your website. It uses intelligent domain mapping to find pages without fully crawling each one.

### How to Use

1. Select **Quick Scan** as your discovery method
2. Enter your website URL (e.g., `https://example.com`)
3. Click **Scan Domain**
4. Review the discovered URLs in the pending list
5. Save the pages you want to your agent's knowledge base

### Advanced Options

<Accordion title="URL Limit">
  By default, Quick Scan will discover unlimited pages. You can set a limit to cap the number of URLs discovered:

  * **Unlimited**: Discover all available pages
  * **Custom limit**: Set a specific number (e.g., 100 pages)

  This is useful when you have a large website but only need a subset of pages.
</Accordion>

***

## Deep Scan

Deep Scan provides thorough crawling with fine-grained control over the crawling process. Use this when you need precise control over which pages are discovered.

### How to Use

1. Select **Deep Scan** as your discovery method
2. Enter your starting URL (e.g., `https://example.com/docs`)
3. Configure advanced options (optional)
4. Click **Scan Domain**
5. Monitor the crawl progress in real-time
6. Review and save discovered URLs

### Advanced Options

Deep Scan offers several configuration options:

<AccordionGroup>
  <Accordion icon="arrow-down" title="Max Depth">
    Controls how many levels deep the crawler will follow links.

    | Depth | Behavior                                     |
    | ----- | -------------------------------------------- |
    | 0     | Only the starting URL                        |
    | 1     | Starting URL + pages linked from it          |
    | 2     | Starting URL + 2 levels of linked pages      |
    | 3+    | Continues following links to specified depth |

    **Default**: 2 levels

    <Note>Higher depth values result in more pages but longer crawl times.</Note>
  </Accordion>

  <Accordion icon="clock" title="Wait Time">
    Time in milliseconds to wait between requests. This helps avoid overwhelming your server and prevents rate limiting.

    **Default**: 200ms

    Increase this value if your server has rate limiting or if you're experiencing timeout errors.
  </Accordion>

  <Accordion icon="hashtag" title="URL Limit">
    Maximum number of URLs to discover during the crawl.

    * **Unlimited**: No cap on discovered URLs
    * **Custom limit**: Stop after discovering specified number of pages

    **Default**: 100 URLs
  </Accordion>

  <Accordion icon="globe" title="Domain Restriction">
    Controls whether the crawler stays on your domain or follows external links.

    | Option               | Behavior                                                |
    | -------------------- | ------------------------------------------------------- |
    | **Same Domain Only** | Only crawl pages on the same domain as the starting URL |
    | **All Domains**      | Follow links to external websites too                   |

    **Default**: Same Domain Only

    <Warning>Enabling "All Domains" can significantly increase crawl time and may include irrelevant content.</Warning>
  </Accordion>

  <Accordion icon="folder" title="Subpath Restriction">
    Limit crawling to specific paths on your website. Enter comma-separated paths to restrict the crawler.

    **Example**: `/docs, /blog, /products`

    This would only crawl URLs that contain `/docs`, `/blog`, or `/products` in their path.
  </Accordion>

  <Accordion icon="filter" title="Filtering Options">
    Additional filters to exclude unwanted URLs:

    | Filter                | What it excludes                           |
    | --------------------- | ------------------------------------------ |
    | **Skip Social Media** | Links to Facebook, Twitter, LinkedIn, etc. |
    | **Skip File URLs**    | Links to PDFs, images, downloads, etc.     |
    | **Skip Anchor Links** | URLs with `#` fragments                    |

    All filters are enabled by default.
  </Accordion>
</AccordionGroup>

### Canceling a Crawl

During a Deep Scan, you can click **Cancel** at any time to stop the crawl. Any URLs discovered up to that point will still be available in your pending list.

***

## Sitemap Import

If your website has a sitemap.xml file, you can import all URLs from it directly. This is often the most reliable method for well-maintained websites.

### How to Use

1. Select **Sitemap** as your discovery method
2. Enter your sitemap URL (e.g., `https://example.com/sitemap.xml`)
3. Click **Import Sitemap**
4. Review the parsed URLs
5. Save the pages you want

### Finding Your Sitemap

Common sitemap locations:

* `https://yoursite.com/sitemap.xml`
* `https://yoursite.com/sitemap_index.xml`
* `https://yoursite.com/sitemap/sitemap.xml`

<Tip>
  Check your website's `robots.txt` file - it often contains a link to your sitemap:

  ```
  Sitemap: https://yoursite.com/sitemap.xml
  ```
</Tip>

### Nested Sitemaps

The platform automatically handles **sitemap index files** - sitemaps that reference other sitemaps. When you import a sitemap index, it will:

1. Detect that it's an index file
2. Fetch each nested sitemap automatically
3. Combine all URLs into a single list
4. Support up to 3 levels of nesting

<Note>
  If your sitemap has more than 3 levels of nesting, some deeper sitemaps may be skipped. This limit helps prevent excessively long import times.
</Note>

***

## Manual URL Entry

When you know exactly which pages you want to import, manual entry is the fastest option.

### How to Use

1. Select **Manual** as your discovery method
2. Paste your URLs into the text area (one per line)
3. Click **Add URLs**
4. Review and save

### Supported Formats

The manual entry field accepts:

* Plain URLs (one per line)
* URLs with or without `https://` prefix
* Pasted HTML content (URLs will be automatically extracted)

**Example input:**

```
https://example.com/page-1
https://example.com/page-2
example.com/page-3
www.example.com/page-4
```

### Extracting URLs from HTML

If you copy HTML content (like from a webpage source), the platform will automatically extract all valid URLs from anchor tags and plain text.

<Tip>
  Use the **Parse from Clipboard** button to extract URLs from copied web content containing links.
</Tip>

***

## Managing Pending Sources

After discovering URLs using any method, they appear in the **Pending Sources** list where you can review and manage them before saving.

### Filtering Pending Sources

| Filter      | Purpose                                                             |
| ----------- | ------------------------------------------------------------------- |
| **Search**  | Find URLs containing specific text                                  |
| **Exclude** | Remove URLs matching patterns (e.g., `/admin`, `.pdf`)              |
| **Type**    | Filter by discovery method (Quick Scan, Deep Scan, Sitemap, Manual) |

### Duplicate Detection

The platform automatically detects duplicates:

| Status                     | Meaning                                    |
| -------------------------- | ------------------------------------------ |
| **NEW**                    | URL not in your knowledge base             |
| **Duplicate (in agent)**   | URL already exists in your agent's sources |
| **Duplicate (in pending)** | Same URL already in your pending list      |

Duplicates are shown in a separate section and can be cleared with one click.

### Saving Sources

Once you've reviewed your pending URLs:

1. Use filters to exclude unwanted pages
2. Click **Save to Agent** to add them to your knowledge base
3. Sources will begin processing automatically

***

## Best Practices

<AccordionGroup>
  <Accordion icon="bullseye" title="Start focused, then expand">
    Begin with your most important pages (product pages, key documentation, FAQs). Test your agent, then add more content as needed.
  </Accordion>

  <Accordion icon="sitemap" title="Use sitemaps when available">
    Sitemaps are maintained by your website and provide the most accurate list of pages. They're also faster than crawling.
  </Accordion>

  <Accordion icon="filter" title="Use exclusion filters liberally">
    Exclude admin pages, login pages, and irrelevant sections. Use patterns like `/admin`, `/login`, `/cart` in the exclude filter.
  </Accordion>

  <Accordion icon="clock" title="Be patient with large sites">
    Deep scans of large websites can take several minutes. The progress indicator shows real-time status.
  </Accordion>

  <Accordion icon="arrows-rotate" title="Re-import when content changes">
    When you update your website content, re-import the affected pages to keep your agent's knowledge current.
  </Accordion>
</AccordionGroup>

## Common Issues

### Crawl Times Out

If your crawl times out:

* Reduce the **Max Depth** setting
* Increase the **Wait Time** between requests
* Set a lower **URL Limit**
* Use **Subpath Restriction** to focus on specific sections

### Sitemap Won't Load

If sitemap import fails:

* Verify the sitemap URL is accessible in your browser
* Check that the sitemap is valid XML
* Ensure your server isn't blocking automated requests
* Try the direct sitemap URL (not the robots.txt reference)

### Missing Pages

If expected pages aren't discovered:

* Check if pages are linked from your starting URL
* Increase the **Max Depth** setting
* Verify pages aren't blocked by robots.txt
* Try using **Manual Entry** for specific pages

## Next Steps

<CardGroup cols={2}>
  <Card title="Text Sources" icon="file-lines" href="/sources/text-sources">
    Add custom text content not on your website.
  </Card>

  <Card title="Q&A Sources" icon="circle-question" href="/sources/qa-sources">
    Create targeted question-answer pairs.
  </Card>

  <Card title="Test Your Agent" icon="flask" href="/agents/playground">
    Verify your knowledge base in the Playground.
  </Card>

  <Card title="Sources Overview" icon="database" href="/sources/overview">
    Learn about all source types and best practices.
  </Card>
</CardGroup>