Anti-Bot, Stealth & Fallback Escalation

What do you do when a site returns a captcha or blank HTML? Escalate from stealth to an external fetch, one step at a time.

enable_stealth UndetectedAdapter proxy lists fallback_fetch_function

01/concept

Concept

The cheapest anti-bot tool is also the first one you should try: enable_stealth=True in BrowserConfig. It removes common automation leaks such as the navigator.webdriver flag and patches WebDriver-derived properties. If that is not enough, use UndetectedAdapter with AsyncPlaywrightCrawlerStrategy for a heavier evasion layer.

When the site still blocks headless traffic, route through a proxy escalation chain and, as a last resort, provide a fallback_fetch_function. The fallback can call a third-party scraping API or a browser farm; Crawl4AI will use it only after the built-in attempts fail. Keep escalation measured: each layer adds cost and complexity.

02/use-cases

Use cases

Crawl a site that flags headless Chrome with enable_stealth=True.
Use UndetectedAdapter when stealth mode alone is detected.
Fall back to a premium proxy or external API only after cheaper options fail.
Rotate user agents and geolocation together so each session looks consistent.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Stealth mode

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    browser_cfg = BrowserConfig(
        headless=True,
        enable_stealth=True,
    )
    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(),
        )
        print(result.success, result.status_code)

asyncio.run(main())

python Undetected adapter

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, UndetectedAdapter
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

async def main():
    browser_cfg = BrowserConfig(headless=False, verbose=True)
    strategy = AsyncPlaywrightCrawlerStrategy(
        browser_config=browser_cfg,
        browser_adapter=UndetectedAdapter(),
    )
    async with AsyncWebCrawler(
        crawler_strategy=strategy,
        config=browser_cfg,
    ) as crawler:
        result = await crawler.arun(
            url="https://protected-site.com",
            config=CrawlerRunConfig(),
        )
        print(len(result.html))

asyncio.run(main())

python Proxy escalation with fallback

import os
import aiohttp
from crawl4ai.async_configs import ProxyConfig
from crawl4ai import CrawlerRunConfig

async def external_fetch(url: str) -> str:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.my-service.com/scrape",
            json={"url": url, "render_js": True},
            headers={"Authorization": f"Bearer {os.getenv('SCRAPE_API_TOKEN')}"},
        ) as resp:
            return await resp.text()

config = CrawlerRunConfig(
    max_retries=2,
    proxy_config=[
        ProxyConfig.DIRECT,
        ProxyConfig(server="http://datacenter.example.com:8080"),
        ProxyConfig(server="http://residential.example.com:9090"),
    ],
    fallback_fetch_function=external_fetch,
)

05/build

Practice lab

Lab 3.3

Practice lab

0/3

Objective: Enable stealth mode and compare the result to a default crawl.

Steps

Crawl a public page with default settings and record the HTML length.
Crawl the same page with BrowserConfig(enable_stealth=True).
Optionally try headless=False and record the HTML length again.
Compare lengths and status codes.

Success criteria

Default and stealth runs both complete.
Differences in HTML length or status code are noted.
No proxy credentials are hardcoded.

Expected output

Two or three lines showing headless/stealth flags, status code, and HTML length.

Hints

Use a page you are allowed to crawl; do not test anti-bot bypasses against sites that prohibit it.
If you do not have proxies, focus on the stealth comparison.

Solution

python Complete solution

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def try_config(name: str, browser_cfg: BrowserConfig):
    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(),
        )
        print(f"{name}: status={result.status_code} html_len={len(result.html)}")

async def main():
    await try_config("default", BrowserConfig(headless=True))
    await try_config("stealth", BrowserConfig(headless=True, enable_stealth=True))

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Turning on stealth for every site by default instead of escalating only when needed.

TRAP 02

Calling a paid fallback API on the first attempt, before trying proxies.

TRAP 03

Ignoring robots.txt or terms of service because you can bypass detection.