Spiders Contracts Custom Contracts Detecting check runs Spiders Contracts New in version 0.15. Testing spiders can get particularly annoying and while nothing prevents you f...

Spiders Contracts Custom Contracts Detecting check runs Spiders Contracts Testing spiders can get particularly annoying and while nothing prevents you from writing unit tests...

Architecture overview Overview Data flow Components Scrapy Engine Scheduler Downloader Spiders Item Pipeline Downloader middlewares Spider middlewares Event-driven networ...

Architecture overview Overview Data flow Components Scrapy Engine Scheduler Downloader Spiders Item Pipeline Downloader middlewares Spider middlewares Event-driven networ...

Broad Crawls Use the right SCHEDULER_PRIORITY_QUEUE Increase concurrency Increase Twisted IO thread pool maximum size Setup your own DNS Reduce log level Disable cookies Di...

架构概览 概述 组件 Scrapy Engine 调度器(Scheduler) 下载器(Downloader) Spiders Item Pipeline 下载器中间件(Downloader middlewares) Spider中间件(Spider middlewares) 数据流(Data flow) 事件驱动网络(Event-dri...

Spiders scrapy.Spider 爬取规则(Crawling rules) CrawlSpider样例 XMLFeedSpider XMLFeedSpider例子 CSVFeedSpider CSVFeedSpider例子 SitemapSpider SitemapSpider样例 讨论 Spiders Spider类定...

Items Item Types Dictionaries Item objects Dataclass objects attr.s objects Working with Item objects Declaring Item subclasses Declaring fields Working with Item objects Cr...

例子 例子 学习的最好方法就是参考例子，Scrapy也不例外。Scrapy提供了一个叫做 dirbot 的样例项目供您把玩学习。其包含了在教程中介绍的dmoz spider。 您可以通过 https://github.com/scrapy/dirbot 找到 dirbot 。项目中包含了README文件，对项目内容进行了详细的介绍。 如果您...

Coroutines Supported callables Usage Coroutines New in version 2.0. Scrapy has partial support for the coroutine syntax . Supported callables The following callables may...