Custom Strategies & Deployment

You will write a custom extraction strategy and package the crawler as a repeatable, schedulable service.

custom ExtractionStrategy Docker/self-hosting CLI scheduled jobs

01/concept

Concept

When off-the-shelf strategies fall short, subclass ExtractionStrategy and implement extract() and run(). Your strategy receives the page content and can return any JSON-serializable structure. This is the cleanest way to add domain-specific parsing or post-processing.

For deployment, Crawl4AI can run inside Docker, be self-hosted on platforms like Coolify, or execute as a scheduled job via cron or a workflow orchestrator. Keep configuration external through environment variables, mount caches and profiles as volumes, and run the browser setup step in the image build.

02/use-cases

Use cases

Parse a proprietary data format that no built-in strategy supports.
Run a nightly price-monitoring job in a Docker container.
Self-host Crawl4AI behind an internal API for a team.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Custom ExtractionStrategy

import json
from typing import Any
from crawl4ai.extraction_strategy import ExtractionStrategy

class LineCountStrategy(ExtractionStrategy):
    def extract(self, url: str, html: str, **kwargs) -> list[dict[str, Any]]:
        lines = [line.strip() for line in html.splitlines() if line.strip()]
        return [{"url": url, "line_count": len(lines)}]

    async def run(self, url: str, html: str, **kwargs) -> list[dict[str, Any]]:
        return self.extract(url, html, **kwargs)

# Usage
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    strategy = LineCountStrategy()
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(extraction_strategy=strategy),
        )
        if result.success:
            print(json.loads(result.extracted_content))

asyncio.run(main())

dockerfile Docker deployment outline

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN crawl4ai-setup

COPY . /app
CMD ["python", "crawler_service.py"]

05/build

Practice lab

Lab 4.5

Practice lab

0/3

Objective: Implement and run a custom extraction strategy.

Steps

Create a class that inherits from ExtractionStrategy.
Implement extract() and run() to return a list of dictionaries.
Pass the custom strategy to CrawlerRunConfig.
Crawl a page and parse the JSON output.
Verify the output matches your custom logic.

Success criteria

The custom strategy returns data in the expected shape.
The result is parsed with json.loads().
No built-in strategy is used for the custom logic.

Expected output

A JSON array containing the custom extraction result.

Hints

Start with a simple strategy, such as counting headings or links.
Make the strategy async-safe if it will be used with arun_many.

Solution

python Complete solution

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import ExtractionStrategy

class HeadingStrategy(ExtractionStrategy):
    def extract(self, url: str, html: str, **kwargs):
        import re
        headings = re.findall(r"<h([1-6])[^>]*>(.*?)</h\1>", html, re.S)
        return [{"level": int(h[0]), "text": re.sub(r"<[^>]+>", "", h[1]).strip()} for h in headings]

    async def run(self, url: str, html: str, **kwargs):
        return self.extract(url, html, **kwargs)

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(extraction_strategy=HeadingStrategy()),
        )
        if result.success:
            print(json.dumps(json.loads(result.extracted_content), indent=2))

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Putting blocking I/O inside the custom strategy without making it async-friendly.

TRAP 02

Baking secrets into Docker images.

TRAP 03

Forgetting to run crawl4ai-setup in the container build.