Adaptive & Deep Crawling

You will send a focused crawler into a site, let it decide which links matter, and stop it before it spirals forever.

AdaptiveCrawler BFSDeepCrawlStrategy stop conditions state persistence

01/concept

Concept

AdaptiveCrawler combines a seed URL, a query, and an LLM judge to decide which discovered links are worth following. It is useful when the target site has no predictable list page but the content you want follows a theme.

BFSDeepCrawlStrategy gives you breadth-first exploration with configurable depth and stop conditions. Use it when you know the structure and want to enumerate pages systematically.

Both strategies need guardrails: a hard max_pages limit, rate limiting, and saved state. Save state to disk so a long crawl can resume after an interruption.

02/use-cases

Use cases

Discover every documentation page related to a specific feature.
Enumerate a blog archive to a fixed depth without hitting every tag page.
Resume a long crawl from a saved state file after a restart.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Adaptive crawler configuration

from crawl4ai import AdaptiveCrawler, AdaptiveConfig

config = AdaptiveConfig(
    confidence_threshold=0.85,
    max_pages=30,
    top_k_links=3,
    min_gain_threshold=0.05,
    save_state=True,
    state_path="crawl_state.json",
)

async def research(url: str, query: str):
    async with AsyncWebCrawler() as crawler:
        adaptive = AdaptiveCrawler(crawler, config=config)
        result = await adaptive.digest(url, query)
        print(result)

python Custom crawl strategy

from crawl4ai.adaptive_crawler import CrawlStrategy, CrawlState
from typing import List

class DomainSpecificStrategy(CrawlStrategy):
    def calculate_coverage(self, state: CrawlState) -> float:
        # Return coverage score between 0 and 1
        return 0.0

    def calculate_consistency(self, state: CrawlState) -> float:
        return 1.0

    def rank_links(self, links: List, state: CrawlState) -> List:
        # Sort links by domain relevance
        return sorted(links, key=lambda l: l.url.count("/docs/"), reverse=True)

05/build

Practice lab

Lab 4.3

Practice lab

0/3

Objective: Configure an adaptive crawl with a hard page limit and a save-state file.

Steps

Import AdaptiveCrawler and AdaptiveConfig.
Set max_pages=10, top_k_links=2, and save_state=True with a state_path.
Start the crawl from one seed URL and a focused query.
Check that the state file is created.
Print how many pages were visited.

Success criteria

The crawl stops at or before max_pages.
A state file is written to disk.
No infinite crawl occurs.

Expected output

A state file path and a page count up to the configured limit.

Hints

Start with a small max_pages value to keep the lab fast.
Use a focused query; broad queries produce larger crawl graphs.

Solution

python Complete solution

import asyncio
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig

async def main():
    config = AdaptiveConfig(
        max_pages=5,
        top_k_links=2,
        save_state=True,
        state_path="adaptive_state.json",
    )
    async with AsyncWebCrawler() as crawler:
        adaptive = AdaptiveCrawler(crawler, config=config)
        result = await adaptive.digest(
            url="https://example.com/docs",
            query="async crawler configuration",
        )
        print(f"Visited {len(result)} pages")

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Setting a high confidence_threshold and a low min_gain_threshold, causing very long crawls.

TRAP 02

Not saving state for crawls that may run for hours.

TRAP 03

Crawling domains without respecting robots.txt or rate limits.