python reptile tutorial: "With Python crawling expression package."

python reptile tutorial: "With Python crawling expression package", without expression, micro-channel package? Do not worry! The teacher led you to use multi-threaded crawling reptiles a key expression 20w pack ~

Project Source:

1   
2  '' ' 
3  What I do not know how you can add in the learning process
 4  Python learning exchanges buttoned Qun, 934 109 170
 5  group, there are good tutorials, development tools and e-books.
6  Share python current business needs and your talent and how good python learning from zero base, and learn what content.
. 7  '' ' 
. 8  from pyquery Import pyquery AS PQ
 . 9  Import Requests
 10  from Redis Import StrictRedis
 . 11 O = 0
 12 is headers = {
 13 is      ' User-Agent ' : 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0'
14  
15 }
16  
17  
18 def xpan(canshu):
19     try:
20         url = 'https://www.doutula.com/photo/list/?page=' + str(canshu)
21         sp = requests.get(url, headers=headers).text
22  
23         doc=pq(sp)
24         item=doc('.col-xs-6 img').items()
25  
26         for i in Item:
 27              Cu = i.attr ( ' Data-Original ' )
 28   
29   
30   
31 is              # R & lt (Cu, 'F.: / bucket of FIG /%s.jpg'%u) 
32             # U +. 1 = 
33 is              # Print ( 'Double currently stored% d'% U) 
34 is              # Print (Cu) 
35              Redis = StrictRedis (= Host ' 172.18.200.5 ' , Port = 6379, DB =. 1, password = '' )
 36              redis.sadd ( ' bucket FIG web URL download pictures ' , Cu)
 37 [   
38 is      the except Exception AS E:
 39   
40        Print ( ' error ' , e.args)
 41 A = int (the INPUT ( ' I ask you need crawled how many pages it: ' ))
 42  Print ( ' 're crawling link and save it to redis database Please wait .. ...... ' )
 43 is  for I in Range (A):
 44 is   
45     XPan (I +. 1 )
 46 is   
47 Redis = StrictRedis (= Host ' 172.18.200.5 ' , Port = 6379, DB =. 1, password = ' ' )
 48 redis.srem ( ' fighting network diagram download pictures the URL of ' , ' None ' )
49  for I in redis.smembers ( ' bucket of FIG net Photo the URL ' ):
 50      T = STR (I, encoding = ' UTF8 ' )
 51 is      # Print (T) 
52 is      R & lt t.split = ( ' / ' ) [ -1 ]
 53 is      dizhi = ' F.: / bucket FIG / ' + R & lt
 54 is      REQ = requests.get (T)
 55      with Open (dizhi, ' WB ' ) P AS:
 56 is          p.write (req.content)
 57 is          Print ( 'Has been saved saved of% d ' % O)
 58          O +. 1 =

python reptile tutorial: "With Python crawling expression package."

Guess you like