Background workers

The default stack does not start the workers. Website ingestion continues with reduced capabilities.

Without Crawl4AI, Ciele uses its built-in fetch crawler. The built-in crawler reads server-rendered pages well. It is weaker on pages that need JavaScript.

The workers are a separate Compose file, deploy/docker-compose.workers.yml. The stack reads that file only when COMPOSE_FILE lists it.

Enable the workers

Run the bootstrap script with the --workers option:

./deploy/bootstrap.sh --workers

The script adds the file to COMPOSE_FILE in deploy/.env. It also generates the two crawler credentials:

COMPOSE_FILE=docker-compose.yml:docker-compose.workers.yml
CRAWL4AI_API_TOKEN=<generated>
CRAWL4AI_SECRET_KEY=<generated>

The --workers option and the --images option are independent. Neither option disables the other.

The crawler can require approximately 4 GiB of memory. Measure the actual use on your selected host.

Check worker status

Include the workers file in each command:

docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.workers.yml ps
docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.workers.yml logs -f crawl4ai

Do not expose worker ports or tokens to the public network.

Disable the workers

Remove docker-compose.workers.yml from COMPOSE_FILE in deploy/.env. Then remove the container:

docker compose -f deploy/docker-compose.yml up -d --remove-orphans

Upgrade from a release with the graph worker

Earlier releases also started a graph worker from this file. The graph engine is removed. Knowledge search now uses the database index and a rerank step.

After you upgrade, run the --remove-orphans command above one time. It stops the old graph-worker container. You can also delete the old GRAPH_* variables from deploy/.env.

Long-term memory jobs

Long-term memory uses the built-in durable job ledger. It does not require the optional workers.

Each signed-in turn schedules a promote_memories job after a 15-minute quiet window. The cron profile drains these jobs through the existing scheduled route.

Keep cron in COMPOSE_PROFILES when long-term memory is enabled. The default self-host configuration already includes it.

The job skips extraction when memory is disabled or the daily token budget is exhausted. It also skips when no suitable model connection exists.