Regex Extraction

When is a regex faster than an LLM? Whenever the pattern is regular, repeatable, and already on your hard drive.

RegexExtractionStrategy built-in patterns custom patterns one-shot LLM pattern generation

01/concept

Concept

RegexExtractionStrategy is the right tool for common entities and domain-specific patterns. It accepts a bitmask of built-in patterns (Email, Url, Currency, PhoneUS, DateIso, and many others) or a dictionary of custom {label: regex} patterns. Set input_format to html, markdown, text, or fit_html depending on what you want to scan.

For complex patterns, generate them once with an LLM via RegexExtractionStrategy.generate_pattern(), cache the pattern to disk, and reuse it for zero-LLM extraction at scale. This keeps costs low and throughput high.

02/use-cases

Use cases

Pull all email addresses and URLs from a contact page.
Extract prices in a custom currency format from a product catalog.
Cache an LLM-generated regex pattern, then run it over thousands of pages.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Built-in and custom regex patterns

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import RegexExtractionStrategy

async def main():
    strategy = RegexExtractionStrategy(
        pattern=RegexExtractionStrategy.Email | RegexExtractionStrategy.Url,
        custom={
            "usd_price": r"\$\s?\d{1,3}(?:,\d{3})*(?:\.\d{2})?",
        },
        input_format="markdown",
    )

    run_cfg = CrawlerRunConfig(extraction_strategy=strategy)

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com/contact",
            config=run_cfg,
        )

        if result.success and result.extracted_content:
            matches = json.loads(result.extracted_content)
            for m in matches[:10]:
                print(f"{m['label']}: {m['value']}")

asyncio.run(main())

python One-shot LLM pattern generation

import json
import asyncio
from pathlib import Path
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, RegexExtractionStrategy, LLMConfig

async def main():
    cache_file = Path("price_pattern.json")
    if cache_file.exists():
        pattern = json.loads(cache_file.read_text())
    else:
        llm_config = LLMConfig(
            provider="openai/gpt-4o-mini",
            api_token="env:OPENAI_API_KEY",
        )
        async with AsyncWebCrawler() as crawler:
            sample = await crawler.arun("https://example.com/products")
            html = sample.markdown.fit_html

        pattern = RegexExtractionStrategy.generate_pattern(
            label="usd_price",
            html=html,
            query="Product prices in USD format",
            llm_config=llm_config,
        )
        cache_file.write_text(json.dumps(pattern))

    strategy = RegexExtractionStrategy(custom=pattern)
    run_cfg = CrawlerRunConfig(extraction_strategy=strategy)

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com/products", config=run_cfg)
        if result.success:
            print(json.loads(result.extracted_content))

asyncio.run(main())

05/build

Practice lab

Lab 2.2

Practice lab

0/3

Objective: Extract every email address and URL from a page using built-in regex patterns.

Steps

Import RegexExtractionStrategy and compose a bitmask of Email and Url.
Create a CrawlerRunConfig with the strategy.
Crawl a page that contains emails and links.
Parse result.extracted_content and group matches by label.
Print a count for each label and the first three matches.

Success criteria

At least one email or URL is extracted.
Matches are grouped by label.
No LLM is used for simple pattern extraction.

Expected output

Lines like email: user@example.com and url: https://example.com/about.

Hints

Use input_format='markdown' to scan cleaned text.
Combine multiple built-in patterns with the | bitwise OR operator.

Solution

python Complete solution

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import RegexExtractionStrategy

async def main():
    strategy = RegexExtractionStrategy(
        pattern=RegexExtractionStrategy.Email | RegexExtractionStrategy.Url,
        input_format="markdown",
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(extraction_strategy=strategy),
        )
        if result.success and result.extracted_content:
            matches = json.loads(result.extracted_content)
            by_label = {}
            for m in matches:
                by_label.setdefault(m["label"], []).append(m["value"])
            for label, values in by_label.items():
                print(f"{label}: {len(values)} found")
                for v in values[:3]:
                    print(f"  {v}")

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Using an LLM for patterns that RegexExtractionStrategy can handle natively.

TRAP 02

Forgetting to escape backslashes in Python regex strings.

TRAP 03

Scanning raw HTML when fit_html or markdown would produce cleaner matches.