Errors, Status Codes & Cache Discipline

A failed crawl is still a result. Never touch result.markdown until result.success is true.

Failure handling redirects CacheMode.BYPASS vs. ENABLED

01/concept

Concept

Network timeouts, 403/429/500 responses, robots.txt blocks, and browser launch failures all surface through result.success and result.error_message. The status_code field tells you the HTTP status, while redirected_status_code captures the final status after redirects. result.url is always the final URL.

Cache discipline matters during development. CacheMode.BYPASS fetches a fresh page, which is essential when iterating on selectors or verifying fixes. CacheMode.ENABLED reads from cache when available and writes fresh responses when not. Use READ_ONLY when you want to avoid polluting the cache, and WRITE_ONLY when you want to refresh cache entries without reading them.

Respect robots.txt by setting check_robots_txt=True on allowed targets, and stay within rate limits and terms of service. A respectful crawler is a repeatable crawler.

02/use-cases

Use cases

Detect and log 404s in a URL inventory before sending them to extraction.
Retry a failed crawl with CacheMode.BYPASS after fixing a selector.
Honor robots.txt when crawling a site you do not own.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Handle errors and redirects

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    urls = [
        "https://example.com",
        "https://httpbin.org/status/404",
        "https://httpbin.org/redirect/1",
    ]

    async with AsyncWebCrawler() as crawler:
        for url in urls:
            result = await crawler.arun(
                url=url,
                config=CrawlerRunConfig(
                    cache_mode=CacheMode.BYPASS,
                    check_robots_txt=True,
                ),
            )

            print(f"URL: {url}")
            print(f"  success: {result.success}")
            print(f"  final_url: {result.url}")
            print(f"  status: {result.status_code}")
            if not result.success:
                print(f"  error: {result.error_message}")
            print()

asyncio.run(main())

05/build

Practice lab

Lab 1.4

Practice lab

0/4

Objective: Crawl a mix of good, missing, and redirecting URLs and report the outcomes.

Steps

Create a list of three URLs: one known good page, one 404 page, and one redirecting page.
Loop over the list with CacheMode.BYPASS and check_robots_txt=True.
For each result, print the original URL, final URL, success flag, status code, and error message if it failed.
Count how many URLs succeeded and how many failed.

Success criteria

The good URL succeeds with status 200.
The 404 URL reports failure or status 404.
The redirect URL reports the final URL and final status code.
No deprecated cache flags are used.

Expected output

Three rows showing final URL, status, and error messages for failures.

Hints

https://httpbin.org/status/404 reliably returns 404.
https://httpbin.org/redirect/2 follows two redirects.

Solution

python Complete solution

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    urls = [
        "https://example.com",
        "https://httpbin.org/status/404",
        "https://httpbin.org/redirect/2",
    ]
    async with AsyncWebCrawler() as crawler:
        for url in urls:
            result = await crawler.arun(
                url=url,
                config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS),
            )
            print(f"{url} -> {result.url}")
            print(f"  success={result.success} status={result.status_code}")
            if not result.success:
                print(f"  error={result.error_message}")

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Blaming extraction logic when the real problem is a 403 or timeout.

TRAP 02

Leaving CacheMode.BYPASS on in production, which disables useful caching.

TRAP 03

Ignoring check_robots_txt=True results and crawling disallowed paths anyway.