🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Scrapy, a fast high-level web crawling & scraping framework for Python.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

A list of AI agents and robots to block.

Web Scraping Framework

📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告

🤖 Scrape data from HTML websites automatically by just providing examples

<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

Scalable Python web scraping scripts for +40 popular domains

The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

Scrapy middleware to handle javascript pages using selenium

HTTP API for Scrapy spiders

[Unmaintained] A simple and clean video/music/image downloader 👾

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

ISP Data Pollution to Protect Private Browsing History with Obfuscation

Python 3 script to dump/scrape/extract company employees from LinkedIn API

Scrapy Extension for monitoring spiders execution.

WarcDB: Web crawl data as SQLite databases.

🕷 Automatically detect changes made to the official Telegram sites, clients and servers.