Websites

Add a website Source

1. Open the Assistant.
2. Select Knowledge.
3. Add a website Source.
4. Enter the page URL.
5. Start ingestion.
6. Wait until the Source status is Ready.
7. Ask a question that the page answers.
8. Confirm the returned Source citation.

The crawler retrieves the configured website content. A JavaScript-rendering worker is required for pages that need browser rendering.

Monitor the crawl

Ciele shows an activity card in the bottom-right corner of the console when a crawl starts. It stays visible on every console page until the work ends.

The card counts the pages that the crawl adds to Ciele, for example 162/291. The total is available when the crawler gives its last group of pages to Ciele. Before that moment the card shows only the number of pages that are indexed.

While Apify crawls the website, the row for the Source shows the progress that Apify reports, for example 12 of 197 pages crawled. The second number increases while Apify finds more links.

The Library counts Documents for the same Source. Ciele makes one Document from one page. The card counts what the crawler retrieves. The Library counts what Ciele keeps. The two numbers agree when the crawl ends.

1. Select the arrow on the card to show the Sources.
2. Read the status of each Source. The card shows a maximum of four Sources at one time.
3. Select the close control to hide the card. The crawl continues.

A Source that fails stays on the card until you close it. A Source that is complete disappears after some seconds.

Recrawl a Source

Use Recrawl when the source website content changes. Ciele restarts the crawl and replaces the indexed content after successful ingestion.

The API and CLI support recrawl only for website Sources.

Source access

The crawler must reach the URL from the deployment network. Authentication-protected content is not available without a supported access path.

The built-in crawler follows a redirect to the same site. For example, https://example.com can go to https://www.example.com, and http can go to https. A redirect to a different domain or subdomain stops the crawl with an error.

Use your own Apify account

Apify is the managed crawler for file downloads, login-protected sites and large crawls. You can connect the Apify account of your Organization. Then Apify crawls use your token and your Apify account pays for them.

1. In Apify Console, open Settings and then API & Integrations.
2. Copy the API token.
3. In Ciele, open Settings and select Crawling.
4. Select Connect on the Apify card.
5. Paste the token in API token.
6. Optional: enter your Apify user ID in User ID. Ciele makes sure that the token belongs to this account.
7. Select Save.

When the account is connected, the Automatic crawler uses Apify for all crawls. New website Sources also use No page limit by default. Apify then continues until it finds no more pages on the website. To stop at a specific number of pages, turn off No page limit and enter a value in Max pages to crawl.

Other crawlers always stop at their own limit. The built-in crawler stops at 30 pages and Crawl4AI at 5,000. Apify on the platform account stops at 100,000.

Ciele encrypts the token and shows only its last four characters. Only owners and admins can connect or remove the account. If you remove the token while an Apify crawl is in progress, that crawl stops with an error.