Semantic Clustering & Hybrid Pipelines

Why pay LLM tokens for every paragraph when cosine clustering can group them first?

CosineStrategy combining clustering with downstream LLM extraction

01/concept

Concept

CosineStrategy embeds chunks of content and groups semantically similar text before downstream processing. It is ideal for discovery tasks: finding FAQ entries, support articles, or reviews that belong to the same topic without writing a schema.

The real power comes from hybrid pipelines: cluster first, then run a smaller LLM extraction only on the representative chunks. This cuts token spend and reduces noise from unrelated sections of a page.

02/use-cases

Use cases

Group support articles by topic before summarizing each cluster.
Find duplicate or near-duplicate reviews across a product catalog.
Feed only the most relevant chunks to an LLM extraction strategy.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Cosine clustering

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import CosineStrategy

async def main():
    strategy = CosineStrategy(
        semantic_filter="customer reviews",
        word_count_threshold=20,
        sim_threshold=0.4,
        top_k=5,
    )
    run_cfg = CrawlerRunConfig(extraction_strategy=strategy)

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com/reviews",
            config=run_cfg,
        )
        if result.success and result.extracted_content:
            clusters = json.loads(result.extracted_content)
            print(f"Clusters: {len(clusters)}")
            for c in clusters[:2]:
                print(c["text"][:300])

asyncio.run(main())

python Hybrid pipeline: cosine then LLM

import json
import os
from pydantic import BaseModel
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig
from crawl4ai import CosineStrategy, LLMExtractionStrategy

class Review(BaseModel):
    author: str
    rating: int
    body: str

async def extract_reviews(url: str):
    cosine = CosineStrategy(
        semantic_filter="customer reviews",
        word_count_threshold=20,
        top_k=3,
    )
    async with AsyncWebCrawler() as crawler:
        first = await crawler.arun(url=url, config=CrawlerRunConfig(extraction_strategy=cosine))
        if not first.success or not first.extracted_content:
            return []
        clusters = json.loads(first.extracted_content)
        combined = "\n\n".join(c["text"] for c in clusters)

        llm = LLMExtractionStrategy(
            llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY")),
            schema=Review.model_json_schema(),
            extraction_type="schema",
            instruction="Extract structured reviews from the text.",
        )
        second = await crawler.arun(
            url=f"raw://{combined}",
            config=CrawlerRunConfig(extraction_strategy=llm),
        )
        return json.loads(second.extracted_content) if second.success else []

05/build

Practice lab

Lab 4.2

Practice lab

0/3

Objective: Use CosineStrategy to isolate the main article content from a noisy page.

Steps

Choose a content-heavy page with navigation and sidebars.
Create a CosineStrategy with a semantic filter like main article content.
Crawl the page and parse the clusters.
Print the number of clusters and the first 400 characters of the top cluster.

Success criteria

The top cluster contains mostly body text, not navigation.
The number of clusters matches the configured top_k or fewer.
Output is parsed with json.loads().

Expected output

Cluster count and a clean article snippet.

Hints

Raise word_count_threshold to filter out short navigation snippets.
If the result is too broad, raise sim_threshold.

Solution

python Complete solution

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import CosineStrategy

async def main():
    strategy = CosineStrategy(
        semantic_filter="main article content",
        word_count_threshold=100,
        top_k=1,
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com/article",
            config=CrawlerRunConfig(extraction_strategy=strategy),
        )
        if result.success and result.extracted_content:
            clusters = json.loads(result.extracted_content)
            print(f"Clusters: {len(clusters)}")
            print(clusters[0]["text"][:400])

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Running LLM extraction on every chunk instead of only representative ones.

TRAP 02

Tuning clustering without inspecting the resulting groups.

TRAP 03

Forgetting that cosine grouping needs enough text per chunk to produce meaningful embeddings.