Installation & First Crawl

You have the package installed and the browser binary ready; now run one crawl and prove it succeeded before you do anything else.

AsyncWebCrawler BrowserConfig CrawlerRunConfig context manager result.success

01/concept

Concept

Run this mental check before every script: open AsyncWebCrawler with BrowserConfig, pass a CrawlerRunConfig to arun(), and guard every payload read with if result.success. It is the simplest complete loop in Crawl4AI, and every later technique is just more knobs on the same loop.

BrowserConfig owns headless mode, viewport, proxy, and persistent profiles. CrawlerRunConfig owns cache mode, waiting, extraction, and session IDs. The context manager handles browser start and stop, so a failed crawl still leaves the process clean.

02/use-cases

Use cases

Snapshot a competitor pricing page once a day.
Fetch a documentation page and print the markdown for an LLM context window.
Verify that a site is reachable from your deployment environment before adding extraction logic.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Install and run a first crawl

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_cfg = BrowserConfig(headless=True, verbose=True)
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        check_robots_txt=True,
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_cfg,
        )

        if result.success:
            print(f"OK: {result.url}")
            print(result.markdown.raw_markdown[:500])
        else:
            print(f"FAILED: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(main())

05/build

Practice lab

Lab 1.1

Practice lab

0/3

Objective: Install Crawl4AI and crawl one public page that you are allowed to scrape.

Steps

Create a virtual environment and run pip install crawl4ai.
Run crawl4ai-setup to install the browser binaries.
Write a script that creates BrowserConfig(headless=True) and CrawlerRunConfig(cache_mode=CacheMode.BYPASS).
Crawl https://example.com (or another page you own / have permission to scrape).
Print the status code and the first 300 characters of raw markdown.

Success criteria

Script runs without import errors.
result.success is True.
Markdown output is non-empty.

Expected output

A status code of 200 and a block of plain-text markdown starting with the page heading.

Hints

If the browser fails to launch, run crawl4ai-setup again.
Use CacheMode.BYPASS while testing so you do not get stale cached pages.

Solution

python Complete solution

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS),
        )
        print("success:", result.success)
        print("status:", result.status_code)
        if result.success:
            print(result.markdown.raw_markdown[:300])

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Calling await crawler.arun(url, keyword=value) instead of passing a CrawlerRunConfig object.

TRAP 02

Reading result.markdown before checking result.success.

TRAP 03

Forgetting asyncio.run(main()) or running async code in a synchronous context.