Network Capture, Hooks & C4A-Script

Seeing the rendered DOM is not always enough. Sometimes you need the network traffic that built it.

lifecycle hooks network/console capture declarative browser automation

01/concept

Concept

Lifecycle hooks let you run code before a request, after a response, or after the DOM is ready. Combine them with capture_console_messages=True and capture_network_requests=True to record XHR, fetch, and websocket frames. The captured data appears in result.console_messages and result.network_requests.

C4A-Script is a declarative automation layer that expresses clicks, waits, fills, and scrolls as JSON. It is useful for repeatable workflows that would otherwise require raw JavaScript strings.

02/use-cases

Use cases

Capture API responses that hydrate a single-page application.
Log JavaScript errors that explain why a page is blank.
Encode a multi-step checkout flow as reusable C4A-Script.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Network and console capture

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    run_cfg = CrawlerRunConfig(
        capture_network_requests=True,
        capture_console_messages=True,
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_cfg,
        )
        if result.success:
            reqs = result.network_requests or []
            responses = [r for r in reqs if r.get("event_type") == "response"]
            errors = [m for m in (result.console_messages or []) if m.get("type") == "error"]
            print(f"Responses: {len(responses)}, Console errors: {len(errors)}")
            with open("network.json", "w") as f:
                json.dump({"requests": reqs, "console": result.console_messages}, f, indent=2)

asyncio.run(main())

c4a C4A-Script workflow

# Accept cookies, load more items, and wait
IF (EXISTS `.cookie-banner`) THEN CLICK `.accept-all`
CLICK `.load-more-button`
WAIT `.new-item` 5
SCROLL DOWN 500

05/build

Practice lab

Lab 3.5

Practice lab

0/3

Objective: Capture network requests for a page and count response events.

Steps

Enable capture_network_requests=True in CrawlerRunConfig.
Crawl a page that loads several resources.
Filter the captured requests by event_type into request, response, and failed counts.
Print the counts and save the full capture to a JSON file.

Success criteria

At least one response event is captured.
Request, response, and failed counts are printed.
The full capture is written to network.json.

Expected output

Counts for each event type and a confirmation that network.json was written.

Hints

Heavy pages produce large captures; cap the output file size if needed.
Console capture is optional but useful for debugging JavaScript errors.

Solution

python Complete solution

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(
                capture_network_requests=True,
                capture_console_messages=True,
            ),
        )
        if not result.success:
            print("Failed:", result.error_message)
            return
        reqs = result.network_requests or []
        counts = {
            "request": len([r for r in reqs if r.get("event_type") == "request"]),
            "response": len([r for r in reqs if r.get("event_type") == "response"]),
            "failed": len([r for r in reqs if r.get("event_type") == "request_failed"]),
        }
        print(counts)
        with open("network.json", "w") as f:
            json.dump({"network": reqs, "console": result.console_messages}, f, indent=2)

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Assuming network capture is on by default; you must enable the flags.

TRAP 02

Recording traffic without filtering, then drowning in request noise.

TRAP 03

Writing long JavaScript strings when C4A-Script would be clearer and reusable.