Browser vs. Run Configuration

Where do you put headless mode, and where do you put cache mode? Put them in the wrong object and the API will silently ignore you.

Where global browser settings live vs. per-crawl settings; CacheMode

01/concept

Concept

Think of BrowserConfig as the browser environment you set once per crawler instance: browser type, viewport, user agent mode, and persistent profile. CrawlerRunConfig is the per-crawl instruction: cache behavior, word-count threshold, extraction strategy, and JavaScript to run. Mixing the two is the most common beginner mistake because older examples accepted flat keyword arguments.

Caching lives only on CrawlerRunConfig through the CacheMode enum: ENABLED, DISABLED, READ_ONLY, WRITE_ONLY, and BYPASS. Boolean cache flags were removed; if you pass them, the call will reject the setting.

02/use-cases

Use cases

Run the same browser with different cache modes during development and production.
Switch from headless to headed only when debugging a stubborn page.
Reuse a logged-in browser profile across multiple per-crawl configurations.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Browser settings vs. run settings

import asyncio
import os
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    # Global browser environment
    browser_cfg = BrowserConfig(
        headless=True,
        viewport_width=1280,
        viewport_height=720,
        user_agent_mode="random",
    )

    # Per-crawl configurations
    fresh_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        word_count_threshold=20,
    )
    cached_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.ENABLED,
        word_count_threshold=20,
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        fresh = await crawler.arun(url="https://example.com", config=fresh_cfg)
        cached = await crawler.arun(url="https://example.com", config=cached_cfg)

        for r in (fresh, cached):
            print(r.url, r.success, r.status_code)

asyncio.run(main())

05/build

Practice lab

Lab 1.2

Practice lab

0/3

Objective: Run the same URL twice with different cache modes and compare timing.

Steps

Create a BrowserConfig with headless=True.
Create two CrawlerRunConfig objects: one with CacheMode.BYPASS and one with CacheMode.ENABLED.
Crawl the same URL with each config and record the elapsed time.
Print the status code and a 200-character markdown snippet from each run.

Success criteria

Both runs complete successfully.
The cached run is noticeably faster than the bypass run.
No deprecated boolean cache flags appear in the code.

Expected output

Two lines showing URL, success, status code, and elapsed time; the cached run is faster.

Hints

Use time.perf_counter() around each await crawler.arun() call.
Make sure the URL returns 200 before comparing timing.

Solution

python Complete solution

import asyncio
import time
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_cfg = BrowserConfig(headless=True)
    url = "https://example.com"

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        for mode in (CacheMode.BYPASS, CacheMode.ENABLED):
            run_cfg = CrawlerRunConfig(cache_mode=mode)
            start = time.perf_counter()
            result = await crawler.arun(url=url, config=run_cfg)
            elapsed = time.perf_counter() - start
            print(f"{mode.name}: success={result.success} status={result.status_code} time={elapsed:.3f}s")

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Putting cache_mode on BrowserConfig instead of CrawlerRunConfig.

TRAP 02

Using removed boolean cache flags instead of the CacheMode enum.

TRAP 03

Treating BrowserConfig as a per-run tuning knob.