Sessions, Pagination & Stateful Crawling

Some pages only make sense in sequence: log in, click Next, extract. Sessions keep the browser context alive across those calls.

session_id js_only sequential workflows kill_session

01/concept

Concept

Logins, multi-step forms, and pagination need the same browser tab or context across calls. Assign a session_id in CrawlerRunConfig and pass it to sequential arun() calls. After the first navigation, use js_only=True with js_code to click Next or Load More without reloading the page.

Always call await crawler.crawler_strategy.kill_session(session_id) when the workflow ends. Sessions are sequential only; never reuse the same session_id in parallel, or you will corrupt the browser context.

02/use-cases

Use cases

Paginate through a search result set that updates via AJAX.
Log in once, then crawl several authenticated pages.
Submit a form and scrape the resulting report pages.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Paginate through commits with a session

import asyncio
import json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def main():
    url = "https://github.com/microsoft/TypeScript/commits/main"
    session_id = "ts_commits"

    schema = {
        "name": "Commit",
        "baseSelector": "li[data-testid='commit-row-item']",
        "fields": [{"name": "title", "selector": "h4 a", "type": "text"}],
    }

    js_next = """
    const commits = document.querySelectorAll('li[data-testid="commit-row-item"] h4');
    if (commits.length > 0) window.lastCommit = commits[0].textContent.trim();
    const btn = document.querySelector('a[data-testid="pagination-next-button"]');
    if (btn) { btn.click(); console.log('next clicked'); }
    """

    wait_for = """() => {
        const commits = document.querySelectorAll('li[data-testid="commit-row-item"] h4');
        if (commits.length === 0) return false;
        return commits[0].textContent.trim() !== window.lastCommit;
    }"""

    browser_cfg = BrowserConfig(headless=False, verbose=True)
    extraction = JsonCssExtractionStrategy(schema, verbose=True)
    all_commits = []

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        for page in range(3):
            run_cfg = CrawlerRunConfig(
                session_id=session_id,
                extraction_strategy=extraction,
                js_code=js_next if page > 0 else None,
                wait_for=wait_for if page > 0 else None,
                js_only=page > 0,
                cache_mode=CacheMode.BYPASS,
                capture_console_messages=True,
            )
            result = await crawler.arun(url=url, config=run_cfg)
            if result.success and result.extracted_content:
                commits = json.loads(result.extracted_content)
                all_commits.extend(commits)
                print(f"Page {page + 1}: {len(commits)} commits")

        await crawler.crawler_strategy.kill_session(session_id)
        print(f"Total commits: {len(all_commits)}")

asyncio.run(main())

05/build

Practice lab

Lab 3.1

Practice lab

0/3

Objective: Crawl two pages of a paginated site using a shared session_id.

Steps

Choose a paginated site you are allowed to crawl.
Create a session_id and run the first page with js_only=False.
For the second page, set js_only=True and provide js_code that clicks the next button.
Use wait_for to wait for new content before extracting.
Kill the session when finished and print the combined results.

Success criteria

Two distinct pages of data are extracted.
The same session_id is reused across calls.
kill_session is called at the end.

Expected output

Counts for page 1 and page 2, plus a combined total of extracted items.

Hints

Test the JavaScript click snippet in the browser console first.
Use headless=False while debugging pagination.

Solution

python Complete solution

import asyncio
import json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def main():
    url = "https://example.com/items"
    session_id = "pagination_lab"
    schema = {
        "name": "Item",
        "baseSelector": "div.item",
        "fields": [
            {"name": "name", "selector": "h3", "type": "text"},
        ],
    }
    extraction = JsonCssExtractionStrategy(schema)
    items = []

    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        for page in range(2):
            cfg = CrawlerRunConfig(
                session_id=session_id,
                extraction_strategy=extraction,
                js_only=page > 0,
                js_code="document.querySelector('.next')?.click();" if page > 0 else None,
                wait_for="css:div.item" if page > 0 else None,
                cache_mode=CacheMode.BYPASS,
            )
            result = await crawler.arun(url=url if page == 0 else None, config=cfg)
            if result.success and result.extracted_content:
                page_items = json.loads(result.extracted_content)
                items.extend(page_items)
                print(f"Page {page + 1}: {len(page_items)} items")
        await crawler.crawler_strategy.kill_session(session_id)
    print(f"Total: {len(items)}")

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Reusing the same session_id in parallel calls.

TRAP 02

Forgetting kill_session, which leaks browser contexts.

TRAP 03

Passing a URL on every page when js_only=True; the page is already loaded.