Batch Crawling & Rate Limiting

Crawling 1,000 URLs at full concurrency will get you blocked and exhaust your memory. Polite batching is the fix.

arun_many MemoryAdaptiveDispatcher RateLimiter streaming

01/concept

Concept

arun_many() crawls multiple URLs concurrently. Pair it with MemoryAdaptiveDispatcher so concurrency drops if RAM climbs, and with RateLimiter to add random delays and exponential backoff on 429/503 responses. Set stream=True in CrawlerRunConfig to process results as they finish rather than holding the entire list in memory.

When URLs need different configs, pass a list of CrawlerRunConfig objects with url_matcher. Order matters: specific matchers first, fallback config last. Always include a final config with no matcher.

02/use-cases

Use cases

Process a product catalog with thousands of detail pages.
Stream article URLs into a database as they complete.
Route PDFs, blog posts, and JSON endpoints through different extraction strategies in one batch.
Back off automatically when the target starts returning 429 responses.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Polite batch crawl with adaptive dispatcher

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.dispatcher import MemoryAdaptiveDispatcher, RateLimiter
from crawl4ai.monitor import CrawlerMonitor, DisplayMode

async def main():
    urls = [
        "https://example.com/page1",
        "https://example.com/page2",
        "https://example.com/page3",
    ]

    browser_cfg = BrowserConfig(headless=True, verbose=False)
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        stream=False,
    )

    dispatcher = MemoryAdaptiveDispatcher(
        memory_threshold_percent=70.0,
        max_session_permit=5,
        rate_limiter=RateLimiter(
            base_delay=(1.0, 3.0),
            max_delay=60.0,
            max_retries=3,
            rate_limit_codes=[429, 503],
        ),
        monitor=CrawlerMonitor(display_mode=DisplayMode.DETAILED),
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        results = await crawler.arun_many(
            urls=urls,
            config=run_cfg,
            dispatcher=dispatcher,
        )

        for r in results:
            print(r.url, r.success, r.status_code)

asyncio.run(main())

python Streaming mode

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    stream=True,
)

async with AsyncWebCrawler(config=browser_cfg) as crawler:
    async for result in await crawler.arun_many(
        urls=urls,
        config=config,
        dispatcher=dispatcher,
    ):
        if result.success:
            print(f"Done: {result.url} ({len(result.markdown.raw_markdown)} chars)")

05/build

Practice lab

Lab 2.4

Practice lab

0/3

Objective: Crawl five URLs in one batch and report success rates.

Steps

Create a list of five allowed URLs.
Configure a MemoryAdaptiveDispatcher with a RateLimiter and a CrawlerMonitor.
Run crawler.arun_many() with stream=False.
Count how many URLs succeeded and how many failed.
Print the average markdown length for successful results.

Success criteria

All five URLs are attempted.
A success count and failure count are printed.
A rate limiter is configured to avoid hammering the target site.

Expected output

Two counts and an average character length for successful crawls.

Hints

Use example.com sub-pages or httpbin.org endpoints if you do not have your own list.
Keep max_session_permit low for testing.

Solution

python Complete solution

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.dispatcher import MemoryAdaptiveDispatcher, RateLimiter
from crawl4ai.monitor import CrawlerMonitor, DisplayMode

async def main():
    urls = [f"https://example.com" for _ in range(5)]
    dispatcher = MemoryAdaptiveDispatcher(
        memory_threshold_percent=70.0,
        max_session_permit=3,
        rate_limiter=RateLimiter(base_delay=(1.0, 2.0)),
        monitor=CrawlerMonitor(display_mode=DisplayMode.SIMPLE),
    )
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        results = await crawler.arun_many(
            urls=urls,
            config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS),
            dispatcher=dispatcher,
        )
    ok = [r for r in results if r.success]
    fail = [r for r in results if not r.success]
    avg = sum(len(r.markdown.raw_markdown) for r in ok) / len(ok) if ok else 0
    print(f"Success: {len(ok)}, Fail: {len(fail)}, Avg chars: {avg:.0f}")

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Running large batches without a dispatcher or rate limiter.

TRAP 02

Forgetting that the fallback config must come last in a URL-specific config list.

TRAP 03

Holding huge result lists in memory instead of streaming.

TRAP 04

Setting max_session_permit so high that the target blocks you before the rate limiter reacts.