Virtual Scroll, PDFs & Downloads

You will capture content that only appears after scrolling, parse PDFs, and download files the crawler retrieves.

VirtualScrollConfig PDFContentScrapingStrategy file downloads

01/concept

Concept

Virtual-scroll feeds load new items as the user scrolls. Use VirtualScrollConfig with scroll_timeout, max_iterations, and optional scroll_selector to keep scrolling until a stop condition is reached. Then extract the full rendered DOM.

For PDFs, PDFContentScrapingStrategy extracts text from documents without an external parser. For file downloads, configure the crawler to capture binary responses and save them with the original filename or a derived name.

Both features benefit from CacheMode.BYPASS during testing, because a cached partial scroll or stale PDF can hide real behavior.

02/use-cases

Use cases

Scrape a feed that appends items as the user scrolls.
Extract text from PDFs linked inside a documentation site.
Download images or documents discovered during a crawl.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Virtual scroll feed

import asyncio
import re
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, VirtualScrollConfig

async def main():
    virtual_cfg = VirtualScrollConfig(
        container_selector="[data-testid='primaryColumn']",
        scroll_count=10,
        scroll_by="container_height",
        wait_after_scroll=1.0,
    )
    run_cfg = CrawlerRunConfig(virtual_scroll_config=virtual_cfg)
    browser_cfg = BrowserConfig(headless=True)

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://twitter.com/search?q=AI",
            config=run_cfg,
        )
        tweets = re.findall(r'data-testid="tweet"', result.html)
        print(f"Tweets captured: {len(tweets)}")

asyncio.run(main())

python PDF text and image extraction

import asyncio
import os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy

async def main():
    os.makedirs("pdf_images", exist_ok=True)
    pdf_scraping = PDFContentScrapingStrategy(
        extract_images=True,
        save_images_locally=True,
        image_save_dir="pdf_images",
        batch_size=2,
    )
    pdf_crawler = PDFCrawlerStrategy()
    run_cfg = CrawlerRunConfig(scraping_strategy=pdf_scraping)

    async with AsyncWebCrawler(crawler_strategy=pdf_crawler) as crawler:
        result = await crawler.arun(
            url="https://arxiv.org/pdf/2310.06825.pdf",
            config=run_cfg,
        )
        if result.success:
            print("Metadata:", result.metadata.get("title"))
            if result.markdown:
                print(result.markdown.raw_markdown[:500])
            images = result.media.get("images", [])
            print(f"Images extracted: {len(images)}")

asyncio.run(main())

05/build

Practice lab

Lab 3.4

Practice lab

0/3

Objective: Extract text from a remote PDF and report its metadata.

Steps

Create an output directory for any extracted images.
Configure PDFContentScrapingStrategy with extract_images=False for a first pass.
Pass PDFCrawlerStrategy() as the crawler strategy.
Crawl a public PDF URL.
Print the title metadata and the first 400 characters of text.

Success criteria

The PDF is fetched successfully.
Title metadata is printed.
At least 200 characters of text are extracted.

Expected output

PDF title, a snippet of extracted text, and a note about image count.

Hints

Use a small PDF for faster testing; arxiv.org PDFs are a stable choice.
If text extraction is poor, the PDF may be image-based and needs OCR.

Solution

python Complete solution

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy

async def main():
    pdf_scraping = PDFContentScrapingStrategy(extract_images=False)
    pdf_crawler = PDFCrawlerStrategy()
    async with AsyncWebCrawler(crawler_strategy=pdf_crawler) as crawler:
        result = await crawler.arun(
            url="https://arxiv.org/pdf/2310.06825.pdf",
            config=CrawlerRunConfig(scraping_strategy=pdf_scraping),
        )
        if result.success:
            print("Title:", result.metadata.get("title"))
            text = result.markdown.raw_markdown if result.markdown else ""
            print(text[:400])

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Setting max_iterations too high and crawling forever.

TRAP 02

Forgetting CacheMode.BYPASS while debugging virtual scroll behavior.

TRAP 03

Trying to extract PDFs with a CSS schema instead of PDFContentScrapingStrategy.