Websites

Websites are often the richest source of knowledge you already have. Ciele can crawl a site you own and turn its pages into knowledge the assistant answers from. Open an assistant, select Knowledge, then open the Websites tab.

Add a website

1. Select add a website.
2. Enter a name so you can recognize the source later.
3. Enter the address of the site or section to crawl.
4. Set any options that apply to your site, such as marking it as protected by a login or allowing files found on the pages to be imported.
5. Confirm the copyright statement, then select Add Website.

Once added, the site is crawled and its pages become available to the assistant. Sites can be recrawled on a regular basis so ongoing changes flow through.

Control what gets crawled

Large sites often contain pages you do not want in the assistant's knowledge. Two kinds of rules help you focus the crawl.

• Positive rules describe the pages you want to include.
• Negative rules describe the pages you want to exclude.

Use these to keep the assistant focused on the sections that matter, and to leave out pages such as archives or utility pages.

Advanced crawl settings

For sites that need gentler handling, additional settings let you slow the crawl down, extend how long each page is allowed to load, and wait a little longer before reading a page so content that loads late is captured. Reach for these only if a normal crawl misses content or the target site is sensitive to load.

Add a specific list of pages

If you only want a handful of pages rather than a whole crawl, you can provide a list of exact addresses instead. Give the list a name, paste one address per line, set the same options as above, and continue. This is ideal when the useful content lives on a few known pages.