Intermediate

Structured Extraction & Batch Crawling

01/modules

Modules

2.1Schema-Based Extraction — JsonCssExtractionStrategy, JsonXPathExtractionStrategy, schemas, sibling source
2.2Regex Extraction — RegexExtractionStrategy, built-in patterns, custom patterns, one-shot LLM pattern generation
2.3Markdown Generation & Content Filtering — DefaultMarkdownGenerator, PruningContentFilter, raw vs. fit markdown
2.4Batch Crawling & Rate Limiting — arun_many, MemoryAdaptiveDispatcher, RateLimiter, streaming
2.5Lazy Loading & Basic Interaction — scan_full_page, wait_for_images, js_code, wait_for

02/outcomes

Extract structured JSON with CSS/XPath and regex, filter markdown, chunk content, and run polite concurrent batch crawls.