TamimLab
← All projects
DATA INGESTION · ADVENSIS

Tiered web scraping for RAG

Getting clean web content in, without paying for a browser on every page.

The scraping layer that feeds the RAG pipeline. Requests go through tiers: a lightweight HTTP client first, and a headless browser only when a page needs one.

Key decisions

  • Tiered routing. curl_cffi handles most pages cheaply; Browserless takes the JavaScript-heavy ones.
Python · curl_cffi · Browserless