LLM-Powered Extraction & Chunking

You will use an LLM to extract structured data from messy HTML and chunk long pages before sending them to the model.

LLMExtractionStrategy LLMConfig Pydantic schemas chunking token tracking

01/concept

Concept

LLMExtractionStrategy takes a Pydantic schema or JSON model and asks an LLM to map HTML or markdown to structured JSON. It is the right choice when CSS selectors are brittle because the page changes often or the data is inherently unstructured. Configure the provider through LLMConfig.

Long pages can exceed token limits, so pair LLM extraction with chunking. Pass chunk size and overlap parameters, and aggregate the per-chunk results afterward. Track token usage and cost per call so the pipeline stays economical.

02/use-cases

Use cases

Extract job postings from sites that use different HTML for every employer.
Turn a long research paper into chunked summaries.
Normalize inconsistent product descriptions into a fixed schema.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Pydantic schema extraction with chunking

import asyncio
import json
import os
from pydantic import BaseModel
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy

class Product(BaseModel):
    name: str
    price: str

async def main():
    llm_strategy = LLMExtractionStrategy(
        llm_config=LLMConfig(
            provider="openai/gpt-4o-mini",
            api_token=os.getenv("OPENAI_API_KEY"),
        ),
        schema=Product.model_json_schema(),
        extraction_type="schema",
        instruction="Extract all products with name and price.",
        chunk_token_threshold=1000,
        overlap_rate=0.0,
        apply_chunking=True,
        input_format="markdown",
        extra_args={"temperature": 0.0, "max_tokens": 800},
    )

    run_cfg = CrawlerRunConfig(
        extraction_strategy=llm_strategy,
        cache_mode=CacheMode.BYPASS,
    )

    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(
            url="https://example.com/products",
            config=run_cfg,
        )
        if result.success:
            data = json.loads(result.extracted_content)
            print(json.dumps(data, indent=2))
            llm_strategy.show_usage()

asyncio.run(main())

05/build

Practice lab

Lab 4.1

Practice lab

0/3

Objective: Extract a structured list from a page using an LLM and report token usage.

Steps

Define a Pydantic model with at least two fields.
Create an LLMConfig that loads the API token from an environment variable.
Configure LLMExtractionStrategy with schema, instruction, and chunking enabled.
Crawl a page and parse the extracted JSON.
Call show_usage() and print the token counts.

Success criteria

The API token is loaded from OPENAI_API_KEY or similar env var.
Valid JSON is extracted and parsed.
Token usage is printed.

Expected output

A JSON array of objects and a usage report with prompt/completion/total tokens.

Hints

Start with gpt-4o-mini to keep costs low while learning.
If extraction is empty, check that the instruction matches the page content.

Solution

python Complete solution

import asyncio
import json
import os
from pydantic import BaseModel
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy

class Feature(BaseModel):
    name: str
    description: str

async def main():
    strategy = LLMExtractionStrategy(
        llm_config=LLMConfig(
            provider="openai/gpt-4o-mini",
            api_token=os.getenv("OPENAI_API_KEY"),
        ),
        schema=Feature.model_json_schema(),
        extraction_type="schema",
        instruction="Extract product features with name and description.",
        chunk_token_threshold=1200,
        apply_chunking=True,
        input_format="markdown",
    )
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(
            url="https://example.com/product",
            config=CrawlerRunConfig(extraction_strategy=strategy, cache_mode=CacheMode.BYPASS),
        )
        if result.success:
            print(json.dumps(json.loads(result.extracted_content), indent=2))
            strategy.show_usage()

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Using an LLM for structured data that a CSS or regex schema could extract.

TRAP 02

Sending an entire long document in one prompt instead of chunking.

TRAP 03

Forgetting to track token usage and cost per extraction.