Python-Requests库网络爬取实战

其他 2020-06-20 10:24:38 阅读次数: 0

目录

1.京东商品页面的爬取

2.亚马逊商品页面的爬取

3.百度/360搜索关键词提交

4.网络图片的爬取和存储

5. IP地址归属地的自动查询

1.京东商品页面的爬取

>>> import requests
>>> url = "https://item.jd.com/57150995672.html"
>>> try:
	r = requests.get(url);
	r.raise_for_status();
	r.encoding = r.apparent_encoding;
	print(r.text[:1000]);
except:
	print("爬取失败");

#运行结果：
<!DOCTYPE HTML>
<html lang="zh-CN">
<head>
    <!-- shouji -->
    <meta http-equiv="Content-Type" content="text/html; charset=gbk" />
    <title>【现货速发-国行正品】苹果 iPhone 11 Pro 全网通手机 金色 256GB【图片 价格 品牌 报价】-京东</title>
    <meta name="keywords" content="【现货速发-国行正品】苹果 iPhone 11 Pro 全网通手机 金色 256GB,Apple,,京东,网上购物"/>
    <meta name="description" content="【现货速发-国行正品】苹果 iPhone 11 Pro 全网通手机 金色 256GB图片、价格、品牌样样齐全！【京东正品行货，全国配送，心动不如行动，立即购买享受更多优惠哦！】" />
    <meta name="format-detection" content="telephone=no">
    <meta http-equiv="mobile-agent" content="format=xhtml; url=//item.m.jd.com/product/57150995672.html">
    <meta http-equiv="mobile-agent" content="format=html5; url=//item.m.jd.com/product/57150995672.html">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <link rel="canonical" href="//item.jd.com/57150995672.html"/>

2.亚马逊商品页面的爬取

添加了反爬取测试，requests无法成功爬取

>>> import requests
>>> url = "https://www.amazon.cn/gp/product/B01M8L5Z3Y";
>>> try:
	r = requests.get(url);
	r.raise_for_status();
	r.encoding = r.apparent_encoding;
	print(r.text[1000:2000]);
except Exception as e:
	print("爬取失败:{}".format(e));
#运行结构
#爬取失败

>>> import requests
>>> url = "https://www.amazon.cn/gp/product/B01M8L5Z3Y";
>>> try:
	kv = {'user-agent':'Mozilla/5.0'};
	r = requests.get(url,headers=kv);
	r.raise_for_status();
	r.encoding = r.apparent_encoding;
	print(r.text[1000:2000]);
except Exception as e:
	print("爬取失败:{}".format(e));
#运行结果
<div class="a-row a-spacing-medium a-text-center"><i class="a-icon a-logo"></i></div>\n\n        <div class="a-box a-alert a-alert-info a-spacing-base">\n            <div class="a-box-inner">\n                <i class="a-icon a-icon-alert"></i>\n                <h4>请输入您在下方看到的字符</h4>\n                <p class="a-last">抱歉，我们只是想确认一下当前访问者并非自动程序。为了达到最佳效果，请确保您浏览器上的 Cookie 已启用。</p>\n                </div>\n            </div>\n\n            <div class="a-section">\n\n                <div class="a-box a-color-offset-background">\n                    <div class="a-box-inner a-padding-extra-large">\n\n                        <form method="get" action="/errors/validateCaptcha" name="">\n                            <input type=hidden name="amzn" value="aXLaqRuwTkNyJTVtx50Eng==" /><input type=hidden name="amzn-r" value="&#047;gp&#047;product&#047;B01M8L5Z3Y" />\n                            <div class="a-row a-spacing-large">\n                                <div class="a-box">\n                                    <div class="a-box-inner">\n                                        <h4>请输入您在这个图片中看到的字符：</h4>\n

3.百度/360搜索关键词提交

  • 百度关键词接口 http://www.baidu.com/s?wd=keyword
  • 360关键词接口 http://www.so.com/s?q=keyword
import requests

url = "http://www.baidu.com/s";
kv1 = {'wd':'Python'};
kv2 = {'user-agent':'Mozilla/5.0'};

try:
    r = requests.get(url,params=kv1,headers=kv2);
    r.encoding = r.apparent_encoding;
    r.raise_for_status();
    print(r.status_code);
    print(r.text[:2000]);
except:
    print("爬取失败");
#运行结构
200
<!DOCTYPE html>
<html lang="zh-CN">
<head>
    <meta charset="utf-8">
    <title>百度安全验证</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <meta name="apple-mobile-web-app-capable" content="yes">
    <meta name="apple-mobile-web-app-status-bar-style" content="black">
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0">
    <meta name="format-detection" content="telephone=no, email=no">
    <link rel="shortcut icon" href="https://www.baidu.com/favicon.ico" type="image/x-icon">
    <link rel="icon" sizes="any" mask href="https://www.baidu.com/img/baidu.svg">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta http-equiv="Content-Security-Policy" content="upgrade-insecure-requests">
    <link rel="stylesheet" href="https://wappass.bdimg.com/static/touch/css/api/mkdjump_8befa48.css" />
</head>
<body>
    <div class="timeout hide">
        <div class="timeout-img"></div>
        <div class="timeout-title">网络不给力，请稍后重试</div>
        <button type="button" class="timeout-button">返回首页</button>
    </div>
    <div class="timeout-feedback hide">
        <div class="timeout-feedback-icon"></div>
        <p class="timeout-feedback-title">问题反馈</p>
    </div>

<script src="https://wappass.baidu.com/static/machine/js/api/mkd.js"></script>
<script src="https://wappass.bdimg.com/static/touch/js/mkdjump_1b4f317.js"></script>
</body>
</html>
import requests

url = "http://www.so.com/s";
kv1 = {'wd':'Python'};
kv2 = {'user-agent':'Mozilla/5.0'};

try:
    r = requests.get(url,params=kv1,headers=kv2);
    r.encoding = r.apparent_encoding;
    r.raise_for_status();
    print(r.status_code);
    print(r.text[:2000]);
except:
    print("爬取失败");
#运行结果
200
<!DOCTYPE html>
<!--[if lt IE 7 ]><html class="ie6"><![endif]-->
<!--[if IE 7 ]><html class="ie7"><![endif]-->
<!--[if IE 8 ]><html class="ie8"><![endif]-->
<!--[if IE 9 ]><html class="ie9"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!--><html class="w3c"><!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>360搜索，SO靠谱</title>
<link rel="dns-prefetch" href="//p.ssl.qhimg.com"><link rel="dns-prefetch" href="//s.ssl.qhimg.com"><link rel="dns-prefetch" href="//s.ssl.qhres.com">
<link rel="dns-prefetch" href="//p418.ssl.qhimgs4.com"><link rel="dns-prefetch" href="//p419.ssl.qhimgs4.com"><link rel="dns-prefetch" href="//p420.ssl.qhimgs4.com">
<link rel="search" type="application/opensearchdescription+xml" href="https://www.so.com/soopensearch.xml" title="360搜索">
<meta name="keywords" content="360搜索,360搜索,网页搜索,视频搜索,图片搜索,音乐搜索,新闻搜索,软件搜索,学术搜索">
<meta name="description" content="360搜索是安全、精准、可信赖的新一代搜索引擎，依托于360母品牌的安全优势，全面拦截各类钓鱼欺诈等恶意网站，提供更放心的搜索服务。 360搜索 so靠谱。">
<meta content="always" name="referrer">
<noscript>
<img src="//s.qhupdate.com/so/click.gif?pro=so&pid=home&mod=noscript&t=1574387001" style="display:none">
<meta http-equiv="refresh" content="0; url=http://www.so.com/haosou.html?src=home">
</noscript>
<link rel="shortcut icon" href="https://s2.ssl.qhres.com/static/121a1737750aa53d.ico" type="image/x-icon">
<script>var TIME = {_: +new Date}</script><script>(function(e,t){function n(e){return t.getElementById(e)}function r(){u("stc_nls",1,1)}function i(n,r){var i="";try{i=p[n]||"",i.length<99&&(u(r,0),t.documentElement.style.display="none",l(),e.onbeforeunload=null,location.reload(!0))}catch(s){l()}return i}function s(e,t){try{p[e]=t,t!==p[e]&&l()}catch(n){l()}}function o(e){var n=t.cookie.split("; ");for(var r=0,i=n.length,s;r<i;r++){s=n[r].split("=");if(s[0]===e)return s[1]}return""}function u(e,n,r){r=r||60,n||(r=-1),r=(new Date(+(new Date)+r*864e5)).toGMTString();var i=e+"="+n+"; path=/; expires="+r;location.prot

4.网络图片的爬取和存储

import requests
import os

root = "D://pics//";
url = "https://timgsa.baidu.com/timg?image&quality=80&size=b9999_10000&sec=1574397390863&di=8ee06aae5c5357905d07e94ca10368dd&imgtype=0&src=http%3A%2F%2Fb-ssl.duitang.com%2Fuploads%2Fitem%2F201805%2F14%2F20180514003735_dtxkl.jpg";
path = root + url.split('%')[-1];
try:
    if not os.path.exists(root):
        os.mkdir(root);
    if not os.path.exists(path):
        r = requests.get(url);
        print(r.status_code);
        print(path);
        with open(path, 'wb') as f:
            f.write(r.content);
            f.close();
            print('文件保存成功');
    else:
        print("文件已存在");
except Exception as e:
    print("爬取失败{0}".format(e));

#运行结构
200
D://pics//2F20180514003735_dtxkl.jpg
文件保存成功

5. IP地址归属地的自动查询

import requests

url = "http://m.ip138.com/ip.asp?ip=";
try:
    r = requests.get(url+'202.204.80.112');
    r.raise_for_status();
    r.encoding = r.apparent_encoding;
    print(r.text[:3000]);
except:
    print("爬取失败");

#运行结果
<!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.0//EN" "http://www.wapforum.org/DTD/xhtml-mobile10.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8"/>
<meta name="viewport" content="width=device-width,user-scalable=no" />
<link href="/apple-touch-icon.png" rel="apple-touch-icon"  />
<link href="/style/common.css" rel="stylesheet" type="text/css" />
<link href="/style/ip.css" rel="stylesheet" type="text/css" />
<title>IP地址在线查询 m.ip138.com 查询网</title>
</head>
<body>
	<div class="wraper">
		<div class="header">
			<div class="header-box">
				<a href="/" class="logo"><img src="/image/logo.png" width="47" height="35"/>iP138</a>
				<span class="naver" id="naver"><span class="naver-text">导航</span><span class="naver-icon"></span></span>
				<div class="naver-sub" id="naver-sub">
					<ul>
						<li><a href="/ip.html">iP查询</a></li>
						<li><a href="/mobile.html">手机号码归属地查询</a></li>
						<li><a href="/huilv/">汇率查询</a></li>
						<li><a href="/youbian/">邮编区号查询</a></li>
						<li><a href="/kuaidi/">快递查询</a></li>
						<li><a href="http://qq.ip138.com/tianqi/">天气预报查询</a></li>
						<li><a href="http://qq.ip138.com/shenfenzheng/">身份证号码查询</a></li>
						<li><a href="http://qq.ip138.com/huoche/">列车时刻表查询</a></li>
						<li><a href="http://qq.ip138.com/daishoudian/mobile.htm">火车票代售点查询</a></li>
					</ul>
				</div>
			</div>
		</div>
		<div class="container">
			<div class="module">
				<p class="form-hd">IP地址或者域名：</p>
				<div class="form">
					<form action="" method="get">
						<input type="text" name="ip" size="30" class="form-text" value="202.204.80.112" />
						<input type="submit" value="查询" class="form-btn" />
					</form>
				</div>
				<div class="query-hd">ip138.com IP查询(搜索IP地址的地理位置)</div>
				<h1 class="query">您查询的IP：202.204.80.112</h1><p class="result">本站主数据：北京市海淀区 北京理工大学 教育网</p><p class="result">参考数据一：北京市 北京理工大学</p>
			</div>
		</div>
		<div class="footer">
			<a href="http://www.miitbeian.gov.cn/" rel="nofollow" target="_blank">沪ICP备10013467号-1</a>
		</div>
	</div>
	<script type="text/javascript" src="/script/common.js"></script></body>
</html>

本文仅为学习Python记录，资料来源于中国大学MOOC《Python网络爬虫与信息提取》—嵩天

猜你喜欢

转载自blog.csdn.net/cheng6023/article/details/103194226
今日推荐
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
周排行
Metasploit文件目录与入侵基本概念
跨域(CORS)请求问题[No 'Access-Control-Allow-Origin' header is present on the requested resource]常见解决方案
CodeIgniter 源码解读之 CodeIgniter.php（二）
SAS入门之（四）改变数据类型
初识元组
[数学建模]数学建模算法和模型（B站视频）（二）
Nginx 服务器源码安装配置流程
C#实现语音视频录制 【基于MCapture + MFile】
开发进度4
下载安装vue的方法网址
每日归档
更多
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022