Markdown Generation & Content Filtering

Feeding an entire page into an LLM wastes tokens and context. Prune the boilerplate first.

DefaultMarkdownGenerator PruningContentFilter raw vs. fit markdown

01/concept

Concept

Crawl4AI converts HTML to markdown through a DefaultMarkdownGenerator. You can attach a content filter such as PruningContentFilter, which scores DOM nodes by content density and removes low-value chrome. The output appears as fit_markdown and fit_html inside the MarkdownGenerationResult. Use raw_markdown when you want the full conversion and fit_markdown when you want only the article body. Start with a threshold of 0.48 and tune per site.

02/use-cases

Use cases

Convert documentation pages into LLM-ready context windows.
Remove ads and navigation from news articles before summarization.
Compare raw vs. fit markdown to tune a content filter threshold.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Pruning content filter

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        markdown_generator=DefaultMarkdownGenerator(
            content_filter=PruningContentFilter(threshold=0.48),
        ),
    )

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com/article",
            config=run_cfg,
        )

        if result.success and result.markdown:
            raw = result.markdown.raw_markdown
            fit = result.markdown.fit_markdown or ""
            print(f"Raw markdown: {len(raw)} chars")
            print(f"Fit markdown: {len(fit)} chars")
            print(fit[:500])

asyncio.run(main())

05/build

Practice lab

Lab 2.3

Practice lab

0/3

Objective: Crawl a content-heavy page and compare raw markdown with pruned fit markdown.

Steps

Crawl a page with navigation, footer, and main content.
Run it once with default markdown generation and record raw_markdown length.
Run it again with PruningContentFilter(threshold=0.48) and record fit_markdown length.
Print the length reduction percentage.
Inspect the first 500 characters of fit markdown to confirm boilerplate is gone.

Success criteria

Fit markdown is shorter than raw markdown.
The first 500 characters of fit markdown are mostly body content.
Markdown and fit output are accessed through result.markdown, not removed top-level fields.

Expected output

Two length numbers and a percentage reduction, followed by a clean text snippet.

Hints

If too much content is removed, lower the threshold; if too little, raise it.
Use result.markdown.fit_markdown or '' to avoid None errors.

Solution

python Complete solution

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    url = "https://example.com/article"
    async with AsyncWebCrawler() as crawler:
        raw_result = await crawler.arun(url=url, config=CrawlerRunConfig())
        fit_result = await crawler.arun(
            url=url,
            config=CrawlerRunConfig(
                cache_mode=CacheMode.BYPASS,
                markdown_generator=DefaultMarkdownGenerator(
                    content_filter=PruningContentFilter(threshold=0.48)
                ),
            ),
        )

    raw_len = len(raw_result.markdown.raw_markdown)
    fit_len = len(fit_result.markdown.fit_markdown or "")
    reduction = (1 - fit_len / raw_len) * 100 if raw_len else 0
    print(f"Raw: {raw_len} chars")
    print(f"Fit: {fit_len} chars")
    print(f"Reduction: {reduction:.1f}%")
    print(fit_result.markdown.fit_markdown[:500])

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Passing removed markdown parameters instead of configuring DefaultMarkdownGenerator.

TRAP 02

Expecting fit_markdown to exist when no content filter was configured.

TRAP 03

Tuning the threshold on one page and assuming it works for every site.