Architecture overview Overview Data flow Components Scrapy Engine Scheduler Downloader Spiders Item Pipeline Downloader middlewares Spider middlewares Event-driven networ...

AutoThrottle extension Design goals How it works Throttling algorithm Settings AUTOTHROTTLE_ENABLED AUTOTHROTTLE_START_DELAY AUTOTHROTTLE_MAX_DELAY AUTOTHROTTLE_TARGET_CONCUR...

AutoThrottle extension Design goals How it works Throttling algorithm Settings AUTOTHROTTLE_ENABLED AUTOTHROTTLE_START_DELAY AUTOTHROTTLE_MAX_DELAY AUTOTHROTTLE_TARGET_CONCUR...

Items Item Types Dictionaries Item objects Dataclass objects attr.s objects Working with Item objects Declaring Item subclasses Declaring fields Working with Item objects Cr...

Broad Crawls Use the right SCHEDULER_PRIORITY_QUEUE Increase concurrency Increase Twisted IO thread pool maximum size Setup your own DNS Reduce log level Disable cookies Di...

Items Item Types Dictionaries Item objects Dataclass objects attr.s objects Working with Item objects Declaring Item subclasses Declaring fields Working with Item objects Cr...

Downloading and processing files and images Using the Files Pipeline Using the Images Pipeline Enabling your Media Pipeline Supported Storage File system storage FTP server sto...

Items Item Types Dictionaries Item objects Dataclass objects attr.s objects Working with Item objects Declaring Item subclasses Declaring fields Working with Item objects Cr...

Items Item Types Dictionaries Item objects Dataclass objects attr.s objects Working with Item objects Declaring Item subclasses Declaring fields Working with Item objects Cr...

Spiders Spider class scrapy.spider.Spider Spider样例 案例 CrawlSpider scrapy.spiders.CrawlSpider 爬取规则(Crawling rules) CrawlSpider案例 process_links参数:动态网页爬取，动态url的处理 process_req...