Python3 web crawler combat -14 deployment-related library is installed: Scrapyrt, Gerapy

Scrapyrt installation

Scrapyrt provides for the dispatch of a Scrapy HTTP interface, with which we do not need to execute the command Scrapy Scrapy but you can schedule tasks by requesting a HTTP interface, Scrapyrt than Scrapyd lightweight, distributed multi-task if you do not, then you can simple to use remote scheduling Scrapyrt Scrapy task.

1. Links

• GitHub：https://github.com/scrapinghu...
• The official document:http://scrapyrt.readthedocs.io

2. Pip installation

Pip recommended installation order is as follows:

pip3 install scrapyrt

After the command to complete the installation is finished.

Next run on any project in a Scrapy the following command to start the HTTP service:

scrapyrt
Python资源分享qun 784758214 ,内有安装包，PDF，学习视频，这里是Python学习者的聚集地，零基础，进阶，都欢迎

After running the default start service on port 9080, similar to the output results are as follows:

scrapyrt
2017-07-12 22:31:03+0800 [-] Log opened.
2017-07-12 22:31:03+0800 [-] Site starting on 9080
2017-07-12 22:31:03+0800 [-] Starting factory <twisted.web.server.Site object at 0x10294b160>

If you want to run the port can be used to replace the -p parameter, such as:

scrapyrt -p 9081

This will run on port 9081.

3. Docker installation

In addition Scrapyrt also supports Docker, as you want to run on port 9080, and the project's path to the local Scrapy / home / quotesbot, you can run the following command:

docker run -p 9080:9080 -tid -v /home/user/quotesbot:/scrapyrt/project scrapinghub/scrapyrt

This can also listen on the specified port Scrapy project in 9080.

Gerapy installation

Gerapy Scrapy is a distributed management module, this section to tell us about the installation of Gerapy.

1. Links

GitHub：https://github.com/Gerapy

2. Pip installation

Pip recommended installation order is as follows:

pip3 install gerapy

3. Test the installation

After the installation is complete, you can test at the Python command line.

$ python3
>>> import gerapy
Python资源分享qun 784758214 ,内有安装包，PDF，学习视频，这里是Python学习者的聚集地，零基础，进阶，都欢迎

If no error is reported, then it proves that libraries have been installed.