Proxies, SSL & Identity-Based Crawling

You will route traffic through proxies, inspect SSL certificates, and rotate identity signals so requests do not all look the same.

ProxyConfig rotation SSLCertificate BrowserProfiler geolocation

01/concept

Concept

ProxyConfig lets you route requests through HTTP, SOCKS, or authenticated proxies. Pass a list of ProxyConfig objects to CrawlerRunConfig and Crawl4AI will try them in order. Combine proxies with user_agent_mode, geolocation, and browser profiles so each session presents a consistent identity.

SSLCertificate support and BrowserProfiler let you verify TLS details and compare how a site responds to different browser fingerprints. Use these when you need to prove identity, bypass geo-restrictions, or separate accounts.

02/use-cases

Use cases

Route traffic through a specific country to test geo-targeted content.
Rotate residential proxies for a high-friction target.
Inspect SSL certificate details for compliance monitoring.

03/watch

Video walkthrough

Official Crawl4AI tutorialyoutube · loads on click

04/run

Code example

python Proxy rotation from environment

import asyncio
import os
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy

async def main():
    proxies = ProxyConfig.from_env("PROXIES")
    if not proxies:
        print("Set the PROXIES environment variable first.")
        return

    strategy = RoundRobinProxyStrategy(proxies)
    browser_cfg = BrowserConfig(headless=True)
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        proxy_rotation_strategy=strategy,
    )

    urls = ["https://httpbin.org/ip"] * (len(proxies) * 2)
    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        results = await crawler.arun_many(urls=urls, config=run_cfg)
        for i, result in enumerate(results):
            print(f"Request {i + 1}: {result.success} - {result.status_code}")

asyncio.run(main())

python SSL certificate export

import asyncio
import os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    os.makedirs("certs", exist_ok=True)
    run_cfg = CrawlerRunConfig(
        fetch_ssl_certificate=True,
        cache_mode=CacheMode.BYPASS,
    )

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_cfg,
        )
        if result.success and result.ssl_certificate:
            cert = result.ssl_certificate
            print("Issuer CN:", cert.issuer.get("CN", ""))
            print("Valid until:", cert.valid_until)
            cert.to_json("certs/cert.json")
            cert.to_pem("certs/cert.pem")

asyncio.run(main())

05/build

Practice lab

Lab 3.2

Practice lab

0/3

Objective: Fetch and export the SSL certificate for a public site.

Steps

Create a directory named certs.
Configure CrawlerRunConfig(fetch_ssl_certificate=True, cache_mode=CacheMode.BYPASS).
Crawl https://example.com and check result.success.
If the certificate is present, print the issuer CN and valid-until date.
Export the certificate to JSON and PEM files in the certs directory.

Success criteria

The issuer CN and valid-until date are printed.
Two files (cert.json and cert.pem) are created.
No proxy credentials are hardcoded in the script.

Expected output

Issuer information, expiration date, and confirmation that cert files were written.

Hints

SSL fetching uses a separate socket connection; it does not validate the trust chain.
If you do not have proxies, skip the proxy rotation part of the lab.

Solution

python Complete solution

import asyncio
import os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    os.makedirs("certs", exist_ok=True)
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig(
                fetch_ssl_certificate=True,
                cache_mode=CacheMode.BYPASS,
            ),
        )
        if result.success and result.ssl_certificate:
            cert = result.ssl_certificate
            print(f"Issuer: {cert.issuer.get('CN', '')}")
            print(f"Valid until: {cert.valid_until}")
            cert.to_json("certs/cert.json")
            cert.to_pem("certs/cert.pem")
            print("Exported to certs/")

asyncio.run(main())

06/check

Common mistakes

TRAP 01

Using the same proxy for every request, which defeats the purpose of rotation.

TRAP 02

Baking proxy credentials into committed code instead of environment variables.