爬虫爬取电影天堂电影链接

比较热爱python，最近在用eclipse写java web，那就使用eclipse+PyDv配置环境，小试一次爬虫吧~

看电影还要到处找资源，索性自己直接爬取电影链接，只要在迅雷上crtl+c/v就可以边播边下了~

仅以用来学习娱乐呦~~

进入正题：

　　网页打开电影天堂，发现是一个非常好解析的网页，它网页元素构成简单，容易理解，爬取就不用太多验证控制，首先下载网页源码，然后保存在txt文件中，然后用python读取，进行获取。（完整精简代码在文后）

　　

# -*- coding: utf-8 -*-
import pandas as pd
import numpy as np
import urllib
from urllib import request
from bs4 import BeautifulSoup
from pandas.tests.frame.test_validate import dataframe
import requests
import sys
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf-8')


url =r'e:\tiantang.txt'
resp=open(url,encoding='gb18030',errors='ignore')
s = BeautifulSoup(resp,"lxml")

mingzitag=s.findAll('a')
#print(mingzitag)
href=[]
mingzis=[]
for tag in mingzitag:
    mingzis.append(str(tag.text))
    href.append(tag['href'])
#print(href)
view=pd.DataFrame(mingzis,columns=['NAME'])
view2=pd.DataFrame(href,columns=['URL'])
view3=view.join(view2)
print(view3)

结果：

                              NAME                                              URL

0                                                                       /index.html

1                             最新影片   http://www.ygdy8.net/html/gndy/dyzz/index.html

2                             经典影片        http://www.ygdy8.net/html/gndy/index.html

3                             国内电影  http://www.ygdy8.net/html/gndy/china/index.html

4                             欧美电影  

16                            本站首页                            http://www.dytt8.net/

17                              电影                            /html/gndy/index.html

18                            最新电影                       /html/gndy/dyzz/index.html

19                            日韩电影                      /html/gndy/rihan/index.html

20                            欧美电影                      /html/gndy/oumei/index.html

21                            国内电影                      /html/gndy/china/index.html

22                            综合电影                       /html/gndy/jddy/index.html

23                  2017年科幻动作《全球风暴              /html/gndy/dyzz/20180103/55959.html

24                  2017年高分恐怖《小丑回魂              /html/gndy/dyzz/20171230/55920.html

25                  2017年惊悚悬疑《雪人/雪              /html/gndy/jddy/20171224/55874.html

26                 2017年喜剧《性别之战》BD              /html/gndy/jddy/20171222/55861.html

73      2018年剧情动作《反贪风暴3/L风暴》HD国语中字              /html/gndy/dyzz/20181026/57680.html

74    2018年8.0分动画喜剧《超人总动员2》HD中英双字幕              /html/gndy/dyzz/20181025/57675.html

75                             [2]                                   list_23_2.html

76                             [3]                                   list_23_3.html

77                             [4]                                   list_23_4.html

85                           电影APP                   https://www.dytt8.net/app.html

86                            下载声明                                      /index.html

87                            网站地图                               /plus/sitemap.html

.....

删除无关信息

#把无关信息删除
view3=view3.drop([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22])
view3=view3.drop([75,76,77,78,79,80,81,82,83,84,85,86,87])
view3.reset_index(drop=True,inplace=True)
#print(view3)

                              NAME                                      URL

0                   2017年科幻动作《全球风暴      /html/gndy/dyzz/20180103/55959.html

1                   2017年高分恐怖《小丑回魂      /html/gndy/dyzz/20171230/55920.html

2                   2017年惊悚悬疑《雪人/雪      /html/gndy/jddy/20171224/55874.html

3                  2017年喜剧《性别之战》BD      /html/gndy/jddy/20171222/55861.html

4                   2017年甄子丹刘德华动作《      /html/gndy/dyzz/20171218/55832.html

5                   2017年詹妮弗·劳伦斯悬疑      /html/gndy/dyzz/20171208/55749.html

6                   2017年动作喜剧《宝贝特攻      /html/gndy/jddy/20171206/55738.html

7                  2017年8.3高分战争《敦刻      /html/gndy/dyzz/20171205/55728.html

8                   2017年高分获奖剧情《相爱      /html/gndy/dyzz/20171205/55726.html

9                   2017年高分剧情《天才枪手      /html/gndy/dyzz/20171202/55684.html

10                  2017年爱情喜剧《胡杨的夏      /html/gndy/jddy/20171202/55682.html

11                  2017年动作喜剧《王牌特工      /html/gndy/dyzz/20171130/55665.html

12                  2017年动作《英伦对决》国      /html/gndy/dyzz/20171130/55664.html

13                  2017年动作《美国刺客/美      /html/gndy/dyzz/20171123/55603.html

14                  2017年悬疑动作《极寒之城      /html/gndy/dyzz/20171105/55468.html

15                  2011台湾最新偶像剧《旋风        /html/tv/hytv/20110620/32833.html

16                  2010最新台湾偶像剧《爱似  /html/tv/gangtai/tw/20101227/30016.html

17                  2010潘玮柏最新偶像剧《爱  /html/tv/gangtai/tw/20100823/27737.html

18                  2010台湾热播偶像剧《钟无     /html/tv/gangtai/20100726/27236.html

19                  2010热播偶像剧《呼叫大明  /html/tv/gangtai/tw/20100517/26035.html

20                  2010台湾偶像剧《就想赖着  /html/tv/gangtai/tw/20100118/24051.html

21                  2009热播偶像剧《下一站幸  /html/tv/gangtai/tw/20091005/22040.html

22                  2009台湾八大剧《桃花小妹  /html/tv/gangtai/tw/20091015/22259.html

23         2018年剧情历史《7月22日》BD中英双字幕      /html/gndy/dyzz/20181108/57761.html

24         2018年动作喜剧《欧洲攻略》BD国粤双语中字      /html/gndy/dyzz/20181108/57760.html

25     2018年高分动作《碟中谍6：全面瓦解》HD中英双字幕      /html/gndy/dyzz/20181107/57755.html

26          2018年剧情战争《飓风行动》BD中英双字幕      /html/gndy/dyzz/20181106/57753.html

27         2018年剧情犯罪《你给的仇恨》HD中英双字幕      /html/gndy/dyzz/20181106/57748.html

28    2018年高分动画喜剧《超人总动员2》BD英国粤三语双字      /html/gndy/dyzz/20181104/57728.html

29           2018年奇幻动画《朝花夕誓》BD日语中字      /html/gndy/dyzz/20181104/57726.html

30     2018年科幻动作《巨齿鲨/极悍巨鲨》HD国英双语双字      /html/gndy/dyzz/20181104/57722.html

31  2018年动画喜剧《精灵旅社3：疯狂假期》BD英国粤三语双字      /html/gndy/dyzz/20181103/57720.html

32     2018年高分奇幻《与神同行：罪与罚》BD韩粤双语中字      /html/gndy/dyzz/20181102/57714.html

33    2018年高分奇幻《与神同行2：因与缘》BD韩语中英双字      /html/gndy/dyzz/20181102/57713.html

34       2018年悬疑恐怖《修女/招魂外传》BD中英双字幕      /html/gndy/dyzz/20181102/57712.html

35          2018年喜剧《西虹市首富》HD国语中英双字      /html/gndy/dyzz/20181101/57703.html

36       2018年剧情《巴比龙/逃离恶魔岛》BD中英双字幕      /html/gndy/dyzz/20181101/57702.html

37    2018年动作惊悚《伸冤人2/私刑教育2》BD中英双字幕      /html/gndy/dyzz/20181101/57700.html

38            2018年爱情喜剧《牵线》BD中英双字幕      /html/gndy/dyzz/20181031/57699.html

39           2018年剧情《三角草的春天》BD日语中字      /html/gndy/dyzz/20181031/57698.html

40            2018年惊悚恐怖《幼儿怨》BD粤语中字      /html/gndy/dyzz/20181029/57694.html

41      2018年高分喜剧《克里斯托弗·罗宾》BD中英双字幕      /html/gndy/dyzz/20181028/57688.html

42         2018年动作历史战争《大轰炸》HD中英双字幕      /html/gndy/dyzz/20181028/57686.html

43  2018年高分剧情爱情《冷战/没有烟硝的爱情》BD中英双字幕      /html/gndy/dyzz/20181027/57685.html

44      2018年冒险剧情《阿尔法：狼伴归途》BD中英双字幕      /html/gndy/dyzz/20181026/57682.html

45        2018年高分历史战争《冒牌上尉》BD中英双字幕      /html/gndy/dyzz/20181026/57681.html

46      2018年剧情动作《反贪风暴3/L风暴》HD国语中字      /html/gndy/dyzz/20181026/57680.html

47    2018年8.0分动画喜剧《超人总动员2》HD中英双字幕      /html/gndy/dyzz/20181025/57675.html

发现这些链接只有后半段，缺少前半段，需要加上前半段url

wanzheng=[]
t=view3['URL']
for row in t:
    src='https://www.dytt8.net'+row
    wanzheng.append(src)
#print(wanzheng)

['https://www.dytt8.net/html/gndy/dyzz/20180103/55959.html', 'https://www.dytt8.net/html/gndy/dyzz/20171230/55920.html', 'https://www.dytt8.net/html/gndy/jddy/20171224/55874.html', 'https://www.dytt8.net/html/gndy/jddy/20171222/55861.html', 'https://www.dytt8.net/html/gndy/dyzz/20171218/55832.html', 'https://www.dytt8.net/html/gndy/dyzz/20171208/55749.html', 'https://www.dytt8.net/html/gndy/jddy/20171206/55738.html', 'https://www.dytt8.net/html/gndy/dyzz/20171205/55728.html', 'https://www.dytt8.net/html/gndy/dyzz/20171205/55726.html', 'https://www.dytt8.net/html/gndy/dyzz/20171202/55684.html', 'https://www.dytt8.net/html/gndy/jddy/20171202/55682.html', 'https://www.dytt8.net/html/gndy/dyzz/20171130/55665.html', 'https://www.dytt8.net/html/gndy/dyzz/20171130/55664.html', 'https://www.dytt8.net/html/gndy/dyzz/20171123/55603.html', 'https://www.dytt8.net/html/gndy/dyzz/20171105/55468.html', 'https://www.dytt8.net/html/tv/hytv/20110620/32833.html', 'https://www.dytt8.net/html/tv/gangtai/tw/20101227/30016.html', 'https://www.dytt8.net/html/tv/gangtai/tw/20100823/27737.html', 'https://www.dytt8.net/html/tv/gangtai/20100726/27236.html', 'https://www.dytt8.net/html/tv/gangtai/tw/20100517/26035.html', 'https://www.dytt8.net/html/tv/gangtai/tw/20100118/24051.html', 'https://www.dytt8.net/html/tv/gangtai/tw/20091005/22040.html', 'https://www.dytt8.net/html/tv/gangtai/tw/20091015/22259.html', 'https://www.dytt8.net/html/gndy/dyzz/20181108/57761.html', 'https://www.dytt8.net/html/gndy/dyzz/20181108/57760.html', 'https://www.dytt8.net/html/gndy/dyzz/20181107/57755.html', 'https://www.dytt8.net/html/gndy/dyzz/20181106/57753.html', 'https://www.dytt8.net/html/gndy/dyzz/20181106/57748.html', 'https://www.dytt8.net/html/gndy/dyzz/20181104/57728.html', 'https://www.dytt8.net/html/gndy/dyzz/20181104/57726.html', 'https://www.dytt8.net/html/gndy/dyzz/20181104/57722.html', 'https://www.dytt8.net/html/gndy/dyzz/20181103/57720.html', 'https://www.dytt8.net/html/gndy/dyzz/20181102/57714.html', 'https://www.dytt8.net/html/gndy/dyzz/20181102/57713.html', 'https://www.dytt8.net/html/gndy/dyzz/20181102/57712.html', 'https://www.dytt8.net/html/gndy/dyzz/20181101/57703.html', 'https://www.dytt8.net/html/gndy/dyzz/20181101/57702.html', 'https://www.dytt8.net/html/gndy/dyzz/20181101/57700.html', 'https://www.dytt8.net/html/gndy/dyzz/20181031/57699.html', 'https://www.dytt8.net/html/gndy/dyzz/20181031/57698.html', 'https://www.dytt8.net/html/gndy/dyzz/20181029/57694.html', 'https://www.dytt8.net/html/gndy/dyzz/20181028/57688.html', 'https://www.dytt8.net/html/gndy/dyzz/20181028/57686.html', 'https://www.dytt8.net/html/gndy/dyzz/20181027/57685.html', 'https://www.dytt8.net/html/gndy/dyzz/20181026/57682.html', 'https://www.dytt8.net/html/gndy/dyzz/20181026/57681.html', 'https://www.dytt8.net/html/gndy/dyzz/20181026/57680.html', 'https://www.dytt8.net/html/gndy/dyzz/20181025/57675.html']

这就是完整的链接了，但是得到链接，却没有电影名字，所以需要给每个链接上加上电影名字

view4=pd.DataFrame(wanzheng,columns=['URL'])
view5=view3[['NAME']]
view6=view5.join(view4)
#这是完整的电影名字和电影简介页面链接（进入这个简介页面才能找到下载电影资源的链接）

示例输出如下：

                             NAME                                                URL

0                   2017年科幻动作《全球风暴  https://www.dytt8.net/html/gndy/dyzz/20180103/...

1                   2017年高分恐怖《小丑回魂  https://www.dytt8.net/html/gndy/dyzz/20171230/...

2                   2017年惊悚悬疑《雪人/雪  https://www.dytt8.net/html/gndy/jddy/20171224/...

3                  2017年喜剧《性别之战》BD  https://www.dytt8.net/html/gndy/jddy/20171222/...

所以，我们需要进入简介页面内，获取到完整的电影资源。（好像上一步有点做无用功，，）

ZJURL=[]
for row in view6['URL']: 
    resp=urllib.request.urlopen(row)
s = BeautifulSoup(resp,"lxml")
#按标签层数进入实现精确查找
    ftptag=s.findAll('table')
    for tag in ftptag:
        for tag2 in tag.findAll('tbody'):
            for tag3 in tag2.findAll('tr'):
                for tag4 in tag3.findAll('a'):
                    ZJURL.append(tag4['href'])
try:
    file = open(r'E:\ziyuan22.txt', 'w+',encoding='utf-8')
    for title in ZJURL:
        file.write(title+'                ')
#加些空格将上一条与下一条链接分开
finally:
    if file:
        file.close()
print('over')

保存后的txt文件大概长这样：

好了，到这里你就获得了一份简单的电影资源了！

现在附上完整精简代码：

 1 # -*- coding: utf-8 -*-
 2 import pandas as pd
 3 import numpy as np
 4 import urllib
 5 from urllib import request
 6 from bs4 import BeautifulSoup
 7 from pandas.tests.frame.test_validate import dataframe
 8 import requests
 9 import sys
10 import io
11 sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf-8')
12 
13 url=r'https://xa.58.com/chuzu/?PGTID=0d100000-001e-365c-b1e8-2142aa50dcb5&ClickID=3'
14 page = request.Request(url)
15 page_info = request.urlopen(page).read().decode('utf-8')
16 
17 s = BeautifulSoup(page_info,"lxml")
18 #在a标签里获取链接
19 mingzitag=s.findAll('a')
20 href=[]
21 mingzi=[]
22 for tag in mingzitag:
23     mingzi.append(str(tag.text))
24     href.append(tag['href'])
25 #由于a标签里会有许多无用的标签，所以先建立dataframe,以方便查看无效信息的索引，便于删除
26 view=pd.DataFrame(mingzis,columns=['NAME'])
27 view2=pd.DataFrame(href,columns=['URL'])
28 view3=view.join(view2)
29 view3=view3.drop([1～22])
30 view3=view3.drop([75～87])
31 view3.reset_index(drop=True,inplace=True)
32 #将完整的电影简介页面的url找到，所以必须加入前缀
33 wanzheng=[]
34 t=view3['URL']
35 for row in t:
36     src='https://www.dytt8.net'+row
37     wanzheng.append(src)
38 
39 #将完整url打开找资源链接（ZJURL）
40 view4=pd.DataFrame(wanzheng,columns=['URL'])
41 ZJURL=[]
42 for row in view4['URL']: 
43     resp=urllib.request.urlopen(row)
44     s = BeautifulSoup(resp,"lxml")
45     ftptag=s.findAll('table')
46     for tag in ftptag:
47         for tag2 in tag.findAll('tbody'):
48             for tag3 in tag2.findAll('tr'):
49                 for tag4 in tag3.findAll('a'):
50                     ZJURL.append(tag4['href'])
51 try:
52     file = open(r'E:\最终电影资源.txt', 'w+',encoding='utf-8')
53     for title in ZJURL:
54         file.write(title+'                ')
55 finally:
56     if file:
57         file.close()
58 print('job Done!')