Telnet Console

Scrapy comes with a built-in telnet console for inspecting and controlling aScrapy running process. The telnet console is just a regular python shellrunning inside the Scrapy process, so you can do literally anything from it.

The telnet console is a built-in Scrapy extension which comes enabled by default, but you can alsodisable it if you want. For more information about the extension itself seeTelnet console extension.

Warning

It is not secure to use telnet console via public networks, as telnetdoesn’t provide any transport-layer security. Having username/passwordauthentication doesn’t change that.

Intended usage is connecting to a running Scrapy spider locally(spider process and telnet client are on the same machine)or over a secure connection (VPN, SSH tunnel).Please avoid using telnet console over insecure connections,or disable it completely using TELNETCONSOLE_ENABLED option.

How to access the telnet console

The telnet console listens in the TCP port defined in theTELNETCONSOLE_PORT setting, which defaults to 6023. To accessthe console you need to type:

telnet localhost 6023
Trying localhost...
Connected to localhost.
Escape character is '^]'.
Username:
Password:
>>>

By default Username is scrapy and Password is autogenerated. Theautogenerated Password can be seen on Scrapy logs like the example below:

2018-10-16 14:35:21 [scrapy.extensions.telnet] INFO: Telnet Password: 16f92501e8a59326

Default Username and Password can be overridden by the settingsTELNETCONSOLE_USERNAME and TELNETCONSOLE_PASSWORD.

Warning

Username and password provide only a limited protection, as telnetis not using secure transport - by default traffic is not encryptedeven if username and password are set.

You need the telnet program which comes installed by default in Windows, andmost Linux distros.

Available variables in the telnet console

The telnet console is like a regular Python shell running inside the Scrapyprocess, so you can do anything from it including importing new modules, etc.

However, the telnet console comes with some default variables defined forconvenience:

Telnet console usage examples

Here are some example tasks you can do with the telnet console:

View engine status

You can use the est() method of the Scrapy engine to quickly show its stateusing the telnet console:

telnet localhost 6023
>>> est()
Execution engine status
 
time()-engine.start_time                        : 8.62972998619
engine.has_capacity()                           : False
len(engine.downloader.active)                   : 16
engine.scraper.is_idle()                        : False
engine.spider.name                              : followall
engine.spider_is_idle(engine.spider)            : False
engine.slot.closing                             : False
len(engine.slot.inprogress)                     : 16
len(engine.slot.scheduler.dqs or [])            : 0
len(engine.slot.scheduler.mqs)                  : 92
len(engine.scraper.slot.queue)                  : 0
len(engine.scraper.slot.active)                 : 0
engine.scraper.slot.active_size                 : 0
engine.scraper.slot.itemproc_size               : 0
engine.scraper.slot.needs_backout()             : False

Pause, resume and stop the Scrapy engine

To pause:

telnet localhost 6023
>>> engine.pause()
>>>

To resume:

telnet localhost 6023
>>> engine.unpause()
>>>

To stop:

telnet localhost 6023
>>> engine.stop()
Connection closed by foreign host.

Telnet Console signals

• scrapy.extensions.telnet.updatetelnet_vars(_telnet_vars)
• Sent just before the telnet console is opened. You can hook up to thissignal to add, remove or update the variables that will be available in thetelnet local namespace. In order to do that, you need to update thetelnet_vars dict in your handler.

Parameters:telnet_vars (dict) – the dict of telnet variables

Telnet settings

These are the settings that control the telnet console’s behaviour:

TELNETCONSOLE_PORT

Default: [6023, 6073]

The port range to use for the telnet console. If set to None or 0, adynamically assigned port is used.

TELNETCONSOLE_HOST

Default: '127.0.0.1'

The interface the telnet console should listen on

TELNETCONSOLE_USERNAME

Default: 'scrapy'

The username used for the telnet console

TELNETCONSOLE_PASSWORD

Default: None

The password used for the telnet console, default behaviour is to have itautogenerated