Learn python the ninth day

Original link: http://www.cnblogs.com/feiyufei/p/11066377.html




Today content: 1 Scrapy crawler frame 
          2 micro channel robot

'' ' 
A, Scrapy crawler frame transmission request ---> fetch response data ---> --- parsed data> # 1 Scarpy frame data stored Introduction: 

        1) Engine (EGINE) 
        among all components of the system responsible for controlling the engine data stream, and trigger events when certain actions occur. For more information, see the section above the data stream. 

        2) scheduler (SCHEDULER) 
        for receiving a request sent over the engine, is pressed into the queue, and returns the engine again when the request may be thought of as a URL of the priority queue, it is determined by a lower to crawl What is the URL, while removing duplicate URLs 

        3) Downloader (DOWLOADER) for downloading web content and web content back to EGINE, downloader is built on this twisted efficient asynchronous model 
        4) reptiles (sPIDERS) 
        sPIDERS a developer from the class definition for parsing Responses, and extracts items, or send a new request 

        5) project pipe (iTEM PIPLINES) 
        after they are responsible for the extracted items, including cleaning, validation, persistence (memory such as to the database) and other operations 
        to download middleware (Downloader middlewares) positioned between the engine and the downloader Scrapy, mainly used to process the request from the request transmitted DOWLOADER EGINE, has spread from DOWNLOADER EGINE response response, 
        you can use this middleware do the following things:
          　　(1) process a request just before it is sent to the Downloader (i.e. right before Scrapy sends the request to the website);
          　　(2) change received response before passing it to a spider;
          　　(3) send a new Request instead of passing received response to a spider;
          　　(4) pass response to a spider without fetching a web page;
          　　(5) silently drop some requests.

        6、爬虫中间件(Spider Middlewares)
        位于EGINE和SPIDERS之间，主要工作是处理SPIDERS的输入（即responses）和输出（即requests）

# 2 Scarpy mounted 

     . 1) PIP3 the install Wheel 
     2) the install lxml PIP3 
     . 3) the install pyopenssl PIP3 
     . 4) the install pypiwin32 PIP3 
     . 5) is mounted twisted frame 
         download twisted: http://www.lfd.uci.edu/~gohlke/pythonlibs/# twisted 

         install the downloaded twisted: 
         PIP3 install the download directory \ Twisted-17.9.0-CP36-cp36m-win_amd64.whl 

     6) PIP3 install scrapy

# Scarpy use 
        1) enter the terminal cmd 
            - scrapy 
                C: \ the Users \ administortra> scrapy 
                Scrapy 1.6.0 - the Active Project NO 

        2) create scrapy Project 
            1. Create a folder dedicated to storing scrapy project 
                - D: \ Scrapy_prject 
            2 .cmd terminal enter the command 
                scrapy startproject Spider_Project (project name) 
                - will be in D: \ folder under Scrapy_prject will generate a file 
                       Spider_Project: Scrapy project file 

            3. create crawler 
                cd Spider_Project # to switch to scrapy project directory 
                              # crawlers name of the destination domain 
                scrapy genspider baidu www.baidu.com # create crawlers 

   # 3, start scrapy project execution crawlers

        # 1) find the reptile program file execution 
        scrapy runspider can only execute a crawler .py 
            # crawlers switch to the executable file directory 
            - cd D: \ Scrapy_prject \ Spider_Project \ Spider_Project \ Spiders 
            - scrapy runspider baidu.py 

        # 2) find the reptiles reptile name based on the implementation of 
        scrapy crawl crawlers name 
            # change to the project directory 
            - cd D: \ Scrapy_prject \ Spider_Project 
            - scrapy crawl baidu 
'' '

# Second, the micro-channel robot

# Installation wxpy support Python 3.4-3.6, and version 2.7

pip3 install -U wxpy

pip3 install pillow

pip3 install pyecharts

1. Log scan code

# 1

# from wxpy import Bot
# print(222)
# bot=Bot(cache_path=True)
# print(111)

2. The male to female ratio micro letter friends

. 1  # 2 
2  
. 3  from wxpy Import Bot.
 . 4  from pyecharts Import Pie
 . 5  Import WebBrowser
 . 6  
. 7  # example of a micro channel robot object 
. 8 BOT = Bot. ()
 . 9  
10  # acquired micro message all friends 
. 11 Friends = bot.friends ()
 12 is  
13  # set male \ female \ location gender friend name 
14 attr = [ ' male friends ' , ' female friend ' , ' unknown attribute ' ]
15  
16  # of initializing the corresponding number of contacts 
. 17 value = [0, 0, 0]
 18 is  
. 19  # traverse all friends, friends of this determination is male or female 
20 is  for Friend in Friends:
 21 is      IF friend.sex ==. 1 :
 22 is          value [ 0] +. 1 =
 23 is      elif friend.sex == 2 :
 24          value [. 1] +. 1 =
 25      the else :
 26 is          value [2] =. 1 +
 27  
28  # instantiate a pie chart objects 
29 PIE = Pie ( ' friends they! ' )
 30  
31  #Chart name str, attribute name list, the attribute value corresponding list, is_label_show label whether it is now 
32 pie.add ( '' , attr, value, is_label_show = True)
 33 is  
34 is  # generates an html file 
35 pie.render ( ' Friends. html ' )
 36  
37 [  # open html file 
38 is webbrowser.open ( ' friends.html ' )

View Code

3. The micro-channel friend area where distribution

installation:

pip3 install echarts-countries-pypkg
pip3 install echarts-china-provinces-pypkg
pip3 install echarts-china-cities-pypkg
pip3 install echarts-china-counties-pypkg
pip3 install echarts-china-misc-pypkg

. 1  # . 3 
2  from wxpy Import *
 . 3  from pyecharts Import the Map
 . 4  Import WebBrowser
 . 5 BOT = Bot. (Cache_path = True)
 . 6  
. 7 Friends = bot.friends ()
 . 8  
. 9  
10 area_dic} = { # define a dictionary for storing province City and the number of provinces and 
11  for Friend in Friends:
 12      IF friend.province not  in area_dic:
 13          area_dic [friend.province] = 1
 14      the else:
 15          area_dic [friend.province] +. 1 =
 16  
. 17 attr = area_dic.keys ()
 18 is value = area_dic.values ()
 . 19  
20 is  
21 is  
22 is Map the Map = ( " micro-channel where the geographical distribution friend " , width = 1200, height = 600 )
 23 is  map.add (
 24      " friends geographical distribution " ,
 25      attr,
 26 is      value,
 27      MapType = ' China ' ,
 28      is_visualmap = True, # combination VisualMap 
29  
30  )
 31 is # Is_visualmap -> BOOL whether visual mapping assembly 
32  #
 33 is map.render ( ' area.html ' )
 34 is  
35  
36 webbrowser.open ( " area.html " )

View Code

4 chatbot

http://www.tuling123.com/

. 1  Import Jason
 2  Import Requests
 . 3  from wxpy Import *
 . 4 BOT = Bot. (Cache_path = True)
 . 5  
. 6  # call Turing robot API, sending a reply message and obtains the robot 
. 7  DEF auto_reply (text):
 . 8      URL = " HTTP: / /www.tuling123.com/openapi/api " 
. 9      api_key = " 9df516a74fc443769b233b01e8536a42 " 
10      payload = {
 . 11          " Key " : api_key,
 12 is          " info ": text,
13     }
14     r = requests.post(url, data=json.dumps(payload))
15     result = json.loads(r.content)
16     return "[来自智能机器人] " + result["text"]
17 
18 
19 @bot.register()
20 def forward_message(msg):
21     return auto_reply(msg.text)
22 
23 embed()

View Code

Python has ended a two-week, very grateful to the teacher's guidance tank.

Reproduced in: https: //www.cnblogs.com/feiyufei/p/11066377.html

Learn python the ninth day

Guess you like