做爬虫，出师未捷身先死，体会过吗？！！！
最近在做一个国外的网站爬虫中文名叫蝙蝠，有网友这样介绍的：“贸易中介类的网站，PR值是6，网站比较可靠”；上面记录了很多公司的信息，如电话、地址、业务等等，目标就是采集特定公司的信息。
本文不是讲如何突破淘宝和美团，而是通过一个案列讲解他们所使用的共同技术和思想，因为不久之后即使采用selenium抓取网站也会遇上强大的反扒机制，因为在这场较量中对手已经开始对selenium反击。
那就开始把，首先是翻墙打开VPN访问手动网站，结果返回下面的页面：
第一次手动访问这个网站居然出现这种情况，即使反扒机制再强也没遇到过连网页都打不开的情况！！！除了惊喜还是惊喜
本来页面中还有个Google图形验证码，我已经手动打了，然后让你填写信息，填写之后就得到下面结果：
大致意思是他们要审核一下。这究竟是一个怎样的网站，我第一次访问居然要Google图形验证+访问申请+访问审核，如果我是善良的客户那么对网站的体验是及其差的。
好吧，该网站可能提前知道我是爬虫所以拒绝也有可能，然后接下来我又试了更换代理，包括使用国外chrome集群直接访问、使用亚马逊ec2动态IP访问，结果如下：
感觉被侮辱了，真的是出师未捷身先死，从来没遇到过一个网站如此厉害的。
然后换了多个姿势来手动访问：
打开VPN，手动打开一个浏览器访问，结果：Access To Website Blocked
打开VPN，使用driver，结果：Access To Website Blocked
使用美国的chrome集群访问，结果：Access To Website Blocked
使用亚马逊的ec2动态IP，结果：Access To Website Blocked
在多次尝试是失败后大胆猜测，该网站应该有访问IP白名单，接下来就验证这一想法，使用了美国的一台本地主机（该主机不是云服务器提供的），居然一下就打开了：
就是这个不熟悉的页面，终于访问成功了，于是大胆猜测：这个网站是有ip白名单的，IP出口是国外的、是国内云服务器供应的那那就拒绝访问。
本以为这样的就完了，然后是我想的太简单了，接下来使用美国本地IP加上driver依然得到下面的结果：
顿时心中把把这个网站问候了一遍，顿时感觉这网站的反扒技术可能和淘宝、美团一样了，通过环境监测，识别浏览器是否是driver控制的，下面就来讲如何分析的。
接下来只有分析数据请求了：
一下就看见了这个409错误，然后依次分析在409错误之前的几个请求，功夫不负有心人，找到了一个成功的post请求：
然后看post提交的数据：
p=%7B%22proof%22%3A%22b3%3A1545721180848%3AHcCAd39S7g05mx3PpKsI%22%2C%22fp2%22%3A%7B%22userAgent%22%3A%22Mozilla%2F5.0(WindowsNT10.0%3BWOW64)AppleWebKit%2F537.36(KHTML%2ClikeGecko)Chrome%2F71.0.3578.98Safari%2F537.36%22%2C%22language%22%3A%22zh-CN%22%2C%22screen%22%3A%7B%22width%22%3A1920%2C%22height%22%3A1080%2C%22availHeight%22%3A1040%2C%22availWidth%22%3A1920%2C%22pixelDepth%22%3A24%2C%22innerWidth%22%3A968%2C%22innerHeight%22%3A889%2C%22outerWidth%22%3A1539%2C%22outerHeight%22%3A1020%2C%22devicePixelRatio%22%3A1%7D%2C%22timezone%22%3A8%2C%22indexedDb%22%3Atrue%2C%22addBehavior%22%3Afalse%2C%22openDatabase%22%3Atrue%2C%22cpuClass%22%3A%22unknown%22%2C%22platform%22%3A%22Win32%22%2C%22doNotTrack%22%3A%22unknown%22%2C%22plugins%22%3A%22ChromePDFPlugin%3A%3APortableDocumentFormat%3A%3Aapplication%2Fx-google-ug_shaders%3BWEBGL_depth_texture%3BWEBKIT_WEBGL_depth_texture%3BWEBGL_draw_buffers%3BWEBGL_lose_context%3BWEBKIT_WEBGL_lose_context%22%2C%22aliasedlinewidthrange%22%3A%22%5B1%2C1%5D%22%2C%22aliasedpointsizerange%22%3A%22%5B1%2C1024%5D%22%2C%22alphabits%22%3A8%2C%22antialiasing%22%3A%22yes%22%2C%22bluebits%22%3A8%2C%22depthbits
·········
很明显是经过URL编码后的数据，然后用URL解码：
from urllib import parse
parse.unquote("%7B%22proof%22%3A%22b3%3A1545721180848%3AHcCAd39S7g05mx3PpKsI%22%2C%22fp2%22%3A%7B%22userAgent%22%3A%22Mozilla%2F5.0(WindowsNT10.0%3BWOW64)AppleWebKit%2F537.36(KHTML%2ClikeGecko)Chrome%2F71.0.3578.98Safari%2F537.36%22%2C%22language%22%3A%22zh-CN%22%2C%22screen%22%3A%7B%22width%22%3A1920%2C%22height%22%3A1080%2C%22availHeight%22%3A1040%2C%22availWidth%22%3A1920%2C%22pixelDepth%22%3A24%2C%22innerWidth%22%3A968%2C%22innerHeight%22%3A889%2C%22outerWidth%22%3A1539%2C%22outerHeight%22%3A1020%2C%22devicePixelRatio%22%3A1%7D%2C%22timezone%22%3A8%2C%22indexedDb%22%3Atrue%2C%22addBehavior%22%3Afalse%2C%22openDatabase%22%3Atrue%2C%22cpuClass%22%3A%22unknown%22%2C%22platform%22%3A%22Win32%22%2C%22doNotTrack%22%3A%22unknown%22%2C%22plugins%22%3A%22ChromePDFPlugin%3A%3APortableDocumentFormat%3A%3Aapplication%2Fx-google-chrome-pdfpdf%3BChromePDFViewer%3A%3A%3A%3Aapplication%2Fpdfpdf%3BNativeClient%3A%3A%3A%3Aapplication%2Fx-nacl%2Capplication%2Fx-pnacl%22%2C%22canvas%22%3A%7B%22winding%22%3A%22yes%22%2C%22towebp%22%3Atrue%2C%22blending%22%3Atrue%2C%22img%22%3A%22f41a4128d68ea76d3784bd619744f7b83fe826eb%22%7D%2C%22webGL%22%3A%7B%22img%22%3A%22bd6549c125f67b18985a8c509803f4b883ff810c%22%2C%22extensions%22%3A%22ANGLE_instanced_arrays%3BEXT_blend_minmax%3BEXT_color_buffer_half_float%3BEXT_disjoint_timer_query%3BEXT_frag_depth%3BEXT_shader_texture_lod%3BEXT_texture_filter_anisotropic%3BWEBKIT_EXT_texture_filter_anisotropic%3BEXT_sRGB%3BOES_element_index_uint%3BOES_standard_derivatives%3BOES_texture_float%3BOES_texture_float_linear%3BOES_texture_half_float%3BOES_texture_half_float_linear%3BOES_vertex_array_object%3BWEBGL_color_buffer_float%3BWEBGL_compressed_texture_s3tc%3BWEBKIT_WEBGL_compressed_texture_s3tc%3BWEBGL_compressed_texture_s3tc_srgb%3BWEBGL_debug_renderer_info%3BWEBGL_debug_shaders%3BWEBGL_depth_texture%3BWEBKIT_WEBGL_depth_texture%3BWEBGL_draw_buffers%3BWEBGL_lose_context%3BWEBKIT_WEBGL_lose_context%22%2C%22aliasedlinewidthrange%22%3A%22%5B1%2C1%5D%22%2C%22aliasedpointsizerange%22%3A%22%5B1%2C1024%5D%22%2C%22alphabits%22%3A8%2C%22antialiasing%22%3A%22yes%22%2C%22bluebits%22%3A8%2C%22depthbits%22%3A24%2C%22greenbits%22%3A8%2C%22maxanisotropy%22%3A16%2C%22maxcombinedtextureimageunits%22%3A32%2C%22maxcubemaptexturesize%22%3A16384%2C%22maxfragmentuniformvectors%22%3A1024%2C%22maxrenderbuffersize%22%3A16384%2C%22maxtextureimageunits%22%3A16%2C%22maxtexturesize%22%3A16384%2C%22maxvaryingvectors%22%3A30%2C%22maxvertexattribs%22%3A16%2C%22maxvertextextureimageunits%22%3A16%2C%22maxvertexuniformvectors%22%3A4096%2C%22maxviewportdims%22%3A%22%5B16384%2C16384%5D%22%2C%22redbits%22%3A8%2C%22renderer%22%3A%22WebKitWebGL%22%2C%22shadinglanguageversion%22%3A%22WebGLGLSLES1.0(OpenGLESGLSLES1.0Chromium)%22%2C%22stencilbits%22%3A0%2C%22vendor%22%3A%22WebKit%22%2C%22version%22%3A%22WebGL1.0(OpenGLES2.0Chromium)%22%2C%22vertexshaderhighfloatprecision%22%3A23%2C%22vertexshaderhighfloatprecisionrangeMin%22%3A127%2C%22vertexshaderhighfloatprecisionrangeMax%22%3A127%2C%22vertexshadermediumfloatprecision%22%3A23%2C%22vertexshadermediumfloatprecisionrangeMin%22%3A127%2C%22vertexshadermediumfloatprecisionrangeMax%22%3A127%2C%22vertexshaderlowfloatprecision%22%3A23%2C%22vertexshaderlowfloatprecisionrangeMin%22%3A127%2C%22vertexshaderlowfloatprecisionrangeMax%22%3A127%2C%22fragmentshaderhighfloatprecision%22%3A23%2C%22fragmentshaderhighfloatprecisionrangeMin%22%3A127%2C%22fragmentshaderhighfloatprecisionrangeMax%22%3A127%2C%22fragmentshadermediumfloatprecision%22%3A23%2C%22fragmentshadermediumfloatprecisionrangeMin%22%3A127%2C%22fragmentshadermediumfloatprecisionrangeMax%22%3A127%2C%22fragmentshaderlowfloatprecision%22%3A23%2C%22fragmentshaderlowfloatprecisionrangeMin%22%3A127%2C%22fragmentshaderlowfloatprecisionrangeMax%22%3A127%2C%22vertexshaderhighintprecision%22%3A0%2C%22vertexshaderhighintprecisionrangeMin%22%3A31%2C%22vertexshaderhighintprecisionrangeMax%22%3A30%2C%22vertexshadermediumintprecision%22%3A0%2C%22vertexshadermediumintprecisionrangeMin%22%3A31%2C%22vertexshadermediumintprecisionrangeMax%22%3A30%2C%22vertexshaderlowintprecision%22%3A0%2C%22vertexshaderlowintprecisionrangeMin%22%3A31%2C%22vertexshaderlowintprecisionrangeMax%22%3A30%2C%22fragmentshaderhighintprecision%22%3A0%2C%22fragmentshaderhighintprecisionrangeMin%22%3A31%2C%22fragmentshaderhighintprecisionrangeMax%22%3A30%2C%22fragmentshadermediumintprecision%22%3A0%2C%22fragmentshadermediumintprecisionrangeMin%22%3A31%2C%22fragmentshadermediumintprecisionrangeMax%22%3A30%2C%22fragmentshaderlowintprecision%22%3A0%2C%22fragmentshaderlowintprecisionrangeMin%22%3A31%2C%22fragmentshaderlowintprecisionrangeMax%22%3A30%7D%2C%22touch%22%3A%7B%22maxTouchPoints%22%3A0%2C%22touchEvent%22%3Afalse%2C%22touchStart%22%3Afalse%7D%2C%22video%22%3A%7B%22ogg%22%3A%22probably%22%2C%22h264%22%3A%22probably%22%2C%22webm%22%3A%22probably%22%7D%2C%22audio%22%3A%7B%22ogg%22%3A%22probably%22%2C%22mp3%22%3A%22probably%22%2C%22wav%22%3A%22probably%22%2C%22m4a%22%3A%22maybe%22%7D%2C%22vendor%22%3A%22GoogleInc.%22%2C%22product%22%3A%22Gecko%22%2C%22productSub%22%3A%2220030107%22%2C%22browser%22%3A%7B%22ie%22%3Afalse%2C%22chrome%22%3Atrue%2C%22webdriver%22%3Atrue%7D%2C%22window%22%3A%7B%22historyLength%22%3A4%2C%22hardwareConcurrency%22%3A4%2C%22iframe%22%3Afalse%7D%2C%22fonts%22%3A%22Calibri%3BCentury%3BHaettenschweiler%3BMarlett%3BPristina%3BSimHei%3BZWAdobeF%22%7D%2C%22cookies%22%3A1%2C%22setTimeout%22%3A0%2C%22setInterval%22%3A0%2C%22appName%22%3A%22Netscape%22%2C%22platform%22%3A%22Win32%22%2C%22syslang%22%3A%22zh-CN%22%2C%22userlang%22%3A%22zh-CN%22%2C%22cpu%22%3A%22%22%2C%22productSub%22%3A%2220030107%22%2C%22plugins%22%3A%7B%220%22%3A%22ChromePDFPlugin%22%2C%221%22%3A%22ChromePDFViewer%22%2C%222%22%3A%22NativeClient%22%7D%2C%22mimeTypes%22%3A%7B%220%22%3A%22application%2Fpdf%22%2C%221%22%3A%22PortableDocumentFormatapplication%2Fx-google-chrome-pdf%22%2C%222%22%3A%22NativeClientExecutableapplication%2Fx-nacl%22%2C%223%22%3A%22PortableNativeClientExecutableapplication%2Fx-pnacl%22%7D%2C%22screen%22%3A%7B%22width%22%3A1920%2C%22height%22%3A1080%2C%22colorDepth%22%3A24%7D%2C%22fonts%22%3A%7B%220%22%3A%22Calibri%22%2C%221%22%3A%22Cambria%22%2C%222%22%3A%22Times%22%2C%223%22%3A%22Constantia%22%2C%224%22%3A%22LucidaBright%22%2C%225%22%3A%22Georgia%22%2C%226%22%3A%22SegoeUI%22%2C%227%22%3A%22Candara%22%2C%228%22%3A%22TrebuchetMS%22%2C%229%22%3A%22Verdana%22%2C%2210%22%3A%22Consolas%22%2C%2211%22%3A%22LucidaConsole%22%2C%2212%22%3A%22LucidaSansTypewriter%22%2C%2213%22%3A%22DejaVuSansMono%22%2C%2214%22%3A%22CourierNew%22%2C%2215%22%3A%22Courier%22%7D%7D")
Out[75]: ‘{“proof”:“b3:1545721180848:HcCAd39S7g05mx3PpKsI”,“fp2”:{“userAgent”:“Mozilla/5.0(WindowsNT10.0;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/71.0.3578.98Safari/537.36”,“language”:“zh-CN”,“screen”:{“width”:1920,“height”:1080,“availHeight”:1040,“availWidth”:1920,“pixelDepth”:24,“innerWidth”:968,“innerHeight”:889,“outerWidth”:1539,“outerHeight”:1020,“devicePixelRatio”:1},“timezone”:8,“indexedDb”:true,“addBehavior”:false,“openDatabase”:true,“cpuClass”:“unknown”,“platform”:“Win32”,“doNotTrack”:“unknown”,“plugins”:“ChromePDFPlugin::PortableDocumentFormat::application/x-google-chrome-pdfpdf;ChromePDFViewer::::application/pdfpdf;NativeClient::::application/x-nacl,application/x-pnacl”,“canvas”:{“winding”:“yes”,“towebp”:true,“blending”:true,“img”:“f41a4128d68ea76d3784bd619744f7b83fe826eb”},“webGL”:{“img”:“bd6549c125f67b18985a8c509803f4b883ff810c”,“extensions”:“ANGLE_instanced_arrays;EXT_blend_minmax;EXT_color_buffer_half_float;EXT_disjoint_timer_query;EXT_frag_depth;EXT_shader_texture_lod;EXT_texture_filter_anisotropic;WEBKIT_EXT_texture_filter_anisotropic;EXT_sRGB;OES_element_index_uint;OES_standard_derivatives;OES_texture_float;OES_texture_float_linear;OES_texture_half_float;OES_texture_half_float_linear;OES_vertex_array_object;WEBGL_color_buffer_float;WEBGL_compressed_texture_s3tc;WEBKIT_WEBGL_compressed_texture_s3tc;WEBGL_compressed_texture_s3tc_srgb;WEBGL_debug_renderer_info;WEBGL_debug_shaders;WEBGL_depth_texture;WEBKIT_WEBGL_depth_texture;WEBGL_draw_buffers;WEBGL_lose_context;WEBKIT_WEBGL_lose_context”,“aliasedlinewidthrange”:"[1,1]",“aliasedpointsizerange”:"[1,1024]",“alphabits”:8,“antialiasing”:“yes”,“bluebits”:8,“depthbits”:24,“greenbits”:8,“maxanisotropy”:16,“maxcombinedtextureimageunits”:32,“maxcubemaptexturesize”:16384,“maxfragmentuniformvectors”:1024,“maxrenderbuffersize”:16384,“maxtextureimageunits”:16,“maxtexturesize”:16384,“maxvaryingvectors”:30,“maxvertexattribs”:16,“maxvertextextureimageunits”:16,“maxvertexuniformvectors”:4096,“maxviewportdims”:"[16384,16384]",“redbits”:8,“renderer”:“WebKitWebGL”,“shadinglanguageversion”:“WebGLGLSLES1.0(OpenGLESGLSLES1.0Chromium)”,“stencilbits”:0,“vendor”:“WebKit”,“version”:“WebGL1.0(OpenGLES2.0Chromium)”,“vertexshaderhighfloatprecision”:23,“vertexshaderhighfloatprecisionrangeMin”:127,“vertexshaderhighfloatprecisionrangeMax”:127,“vertexshadermediumfloatprecision”:23,“vertexshadermediumfloatprecisionrangeMin”:127,“vertexshadermediumfloatprecisionrangeMax”:127,“vertexshaderlowfloatprecision”:23,“vertexshaderlowfloatprecisionrangeMin”:127,“vertexshaderlowfloatprecisionrangeMax”:127,“fragmentshaderhighfloatprecision”:23,“fragmentshaderhighfloatprecisionrangeMin”:127,“fragmentshaderhighfloatprecisionrangeMax”:127,“fragmentshadermediumfloatprecision”:23,“fragmentshadermediumfloatprecisionrangeMin”:127,“fragmentshadermediumfloatprecisionrangeMax”:127,“fragmentshaderlowfloatprecision”:23,“fragmentshaderlowfloatprecisionrangeMin”:127,“fragmentshaderlowfloatprecisionrangeMax”:127,“vertexshaderhighintprecision”:0,“vertexshaderhighintprecisionrangeMin”:31,“vertexshaderhighintprecisionrangeMax”:30,“vertexshadermediumintprecision”:0,“vertexshadermediumintprecisionrangeMin”:31,“vertexshadermediumintprecisionrangeMax”:30,“vertexshaderlowintprecision”:0,“vertexshaderlowintprecisionrangeMin”:31,“vertexshaderlowintprecisionrangeMax”:30,“fragmentshaderhighintprecision”:0,“fragmentshaderhighintprecisionrangeMin”:31,“fragmentshaderhighintprecisionrangeMax”:30,“fragmentshadermediumintprecision”:0,“fragmentshadermediumintprecisionrangeMin”:31,“fragmentshadermediumintprecisionrangeMax”:30,“fragmentshaderlowintprecision”:0,“fragmentshaderlowintprecisionrangeMin”:31,“fragmentshaderlowintprecisionrangeMax”:30},“touch”:{“maxTouchPoints”:0,“touchEvent”:false,“touchStart”:false},“video”:{“ogg”:“probably”,“h264”:“probably”,“webm”:“probably”},“audio”:{“ogg”:“probably”,“mp3”:“probably”,“wav”:“probably”,“m4a”:“maybe”},“vendor”:“GoogleInc.”,“product”:“Gecko”,“productSub”:“20030107”,“browser”:{“ie”:false,“chrome”:true,“webdriver”:true},“window”:{“historyLength”:4,“hardwareConcurrency”:4,“iframe”:false},“fonts”:“Calibri;Century;Haettenschweiler;Marlett;Pristina;SimHei;ZWAdobeF”},“cookies”:1,“setTimeout”:0,“setInterval”:0,“appName”:“Netscape”,“platform”:“Win32”,“syslang”:“zh-CN”,“userlang”:“zh-CN”,“cpu”:"",“productSub”:“20030107”,“plugins”:{“0”:“ChromePDFPlugin”,“1”:“ChromePDFViewer”,“2”:“NativeClient”},“mimeTypes”:{“0”:“application/pdf”,“1”:“PortableDocumentFormatapplication/x-google-chrome-pdf”,“2”:“NativeClientExecutableapplication/x-nacl”,“3”:“PortableNativeClientExecutableapplication/x-pnacl”},“screen”:{“width”:1920,“height”:1080,“colorDepth”:24},“fonts”:{“0”:“Calibri”,“1”:“Cambria”,“2”:“Times”,“3”:“Constantia”,“4”:“LucidaBright”,“5”:“Georgia”,“6”:“SegoeUI”,“7”:“Candara”,“8”:“TrebuchetMS”,“9”:“Verdana”,“10”:“Consolas”,“11”:“LucidaConsole”,“12”:“LucidaSansTypewriter”,“13”:“DejaVuSansMono”,“14”:“CourierNew”,“15”:“Courier”}}’
a = parse.unquote("%7B%22proof%22%3A%22b3%3A1545721180848%3AHcCAd39S7g05mx3PpKsI%22%2C%22fp2%22%3A%7B%22userAgent%22%3A%22Mozilla%2F5.0(WindowsNT10.0%3BWOW64)AppleWebKit%2F537.36(KHTML%2ClikeGecko)Chrome%2F71.0.3578.98Safari%2F537.36%22%2C%22language%22%3A%22zh-CN%22%2C%22screen%22%3A%7B%22width%22%3A1920%2C%22height%22%3A1080%2C%22availHeight%22%3A1040%2C%22availWidth%22%3A1920%2C%22pixelDepth%22%3A24%2C%22innerWidth%22%3A968%2C%22innerHeight%22%3A889%2C%22outerWidth%22%3A1539%2C%22outerHeight%22%3A1020%2C%22devicePixelRatio%22%3A1%7D%2C%22timezone%22%3A8%2C%22indexedDb%22%3Atrue%2C%22addBehavior%22%3Afalse%2C%22openDatabase%22%3Atrue%2C%22cpuClass%22%3A%22unknown%22%2C%22platform%22%3A%22Win32%22%2C%22doNotTrack%22%3A%22unknown%22%2C%22plugins%22%3A%22ChromePDFPlugin%3A%3APortableDocumentFormat%3A%3Aapplication%2Fx-google-chrome-pdfpdf%3BChromePDFViewer%3A%3A%3A%3Aapplication%2Fpdfpdf%3BNativeClient%3A%3A%3A%3Aapplication%2Fx-nacl%2Capplication%2Fx-pnacl%22%2C%22canvas%22%3A%7B%22winding%22%3A%22yes%22%2C%22towebp%22%3Atrue%2C%22blending%22%3Atrue%2C%22img%22%3A%22f41a4128d68ea76d3784bd619744f7b83fe826eb%22%7D%2C%22webGL%22%3A%7B%22img%22%3A%22bd6549c125f67b18985a8c509803f4b883ff810c%22%2C%22extensions%22%3A%22ANGLE_instanced_arrays%3BEXT_blend_minmax%3BEXT_color_buffer_half_float%3BEXT_disjoint_timer_query%3BEXT_frag_depth%3BEXT_shader_texture_lod%3BEXT_texture_filter_anisotropic%3BWEBKIT_EXT_texture_filter_anisotropic%3BEXT_sRGB%3BOES_element_index_uint%3BOES_standard_derivatives%3BOES_texture_float%3BOES_texture_float_linear%3BOES_texture_half_float%3BOES_texture_half_float_linear%3BOES_vertex_array_object%3BWEBGL_color_buffer_float%3BWEBGL_compressed_texture_s3tc%3BWEBKIT_WEBGL_compressed_texture_s3tc%3BWEBGL_compressed_texture_s3tc_srgb%3BWEBGL_debug_renderer_info%3BWEBGL_debug_shaders%3BWEBGL_depth_texture%3BWEBKIT_WEBGL_depth_texture%3BWEBGL_draw_buffers%3BWEBGL_lose_context%3BWEBKIT_WEBGL_lose_context%22%2C%22aliasedlinewidthrange%22%3A%22%5B1%2C1%5D%22%2C%22aliasedpointsizerange%22%3A%22%5B1%2C1024%5D%22%2C%22alphabits%22%3A8%2C%22antialiasing%22%3A%22yes%22%2C%22bluebits%22%3A8%2C%22depthbits%22%3A24%2C%22greenbits%22%3A8%2C%22maxanisotropy%22%3A16%2C%22maxcombinedtextureimageunits%22%3A32%2C%22maxcubemaptexturesize%22%3A16384%2C%22maxfragmentuniformvectors%22%3A1024%2C%22maxrenderbuffersize%22%3A16384%2C%22maxtextureimageunits%22%3A16%2C%22maxtexturesize%22%3A16384%2C%22maxvaryingvectors%22%3A30%2C%22maxvertexattribs%22%3A16%2C%22maxvertextextureimageunits%22%3A16%2C%22maxvertexuniformvectors%22%3A4096%2C%22maxviewportdims%22%3A%22%5B16384%2C16384%5D%22%2C%22redbits%22%3A8%2C%22renderer%22%3A%22WebKitWebGL%22%2C%22shadinglanguageversion%22%3A%22WebGLGLSLES1.0(OpenGLESGLSLES1.0Chromium)%22%2C%22stencilbits%22%3A0%2C%22vendor%22%3A%22WebKit%22%2C%22version%22%3A%22WebGL1.0(OpenGLES2.0Chromium)%22%2C%22vertexshaderhighfloatprecision%22%3A23%2C%22vertexshaderhighfloatprecisionrangeMin%22%3A127%2C%22vertexshaderhighfloatprecisionrangeMax%22%3A127%2C%22vertexshadermediumfloatprecision%22%3A23%2C%22vertexshadermediumfloatprecisionrangeMin%22%3A127%2C%22vertexshadermediumfloatprecisionrangeMax%22%3A127%2C%22vertexshaderlowfloatprecision%22%3A23%2C%22vertexshaderlowfloatprecisionrangeMin%22%3A127%2C%22vertexshaderlowfloatprecisionrangeMax%22%3A127%2C%22fragmentshaderhighfloatprecision%22%3A23%2C%22fragmentshaderhighfloatprecisionrangeMin%22%3A127%2C%22fragmentshaderhighfloatprecisionrangeMax%22%3A127%2C%22fragmentshadermediumfloatprecision%22%3A23%2C%22fragmentshadermediumfloatprecisionrangeMin%22%3A127%2C%22fragmentshadermediumfloatprecisionrangeMax%22%3A127%2C%22fragmentshaderlowfloatprecision%22%3A23%2C%22fragmentshaderlowfloatprecisionrangeMin%22%3A127%2C%22fragmentshaderlowfloatprecisionrangeMax%22%3A127%2C%22vertexshaderhighintprecision%22%3A0%2C%22vertexshaderhighintprecisionrangeMin%22%3A31%2C%22vertexshaderhighintprecisionrangeMax%22%3A30%2C%22vertexshadermediumintprecision%22%3A0%2C%22vertexshadermediumintprecisionrangeMin%22%3A31%2C%22vertexshadermediumintprecisionrangeMax%22%3A30%2C%22vertexshaderlowintprecision%22%3A0%2C%22vertexshaderlowintprecisionrangeMin%22%3A31%2C%22vertexshaderlowintprecisionrangeMax%22%3A30%2C%22fragmentshaderhighintprecision%22%3A0%2C%22fragmentshaderhighintprecisionrangeMin%22%3A31%2C%22fragmentshaderhighintprecisionrangeMax%22%3A30%2C%22fragmentshadermediumintprecision%22%3A0%2C%22fragmentshadermediumintprecisionrangeMin%22%3A31%2C%22fragmentshadermediumintprecisionrangeMax%22%3A30%2C%22fragmentshaderlowintprecision%22%3A0%2C%22fragmentshaderlowintprecisionrangeMin%22%3A31%2C%22fragmentshaderlowintprecisionrangeMax%22%3A30%7D%2C%22touch%22%3A%7B%22maxTouchPoints%22%3A0%2C%22touchEvent%22%3Afalse%2C%22touchStart%22%3Afalse%7D%2C%22video%22%3A%7B%22ogg%22%3A%22probably%22%2C%22h264%22%3A%22probably%22%2C%22webm%22%3A%22probably%22%7D%2C%22audio%22%3A%7B%22ogg%22%3A%22probably%22%2C%22mp3%22%3A%22probably%22%2C%22wav%22%3A%22probably%22%2C%22m4a%22%3A%22maybe%22%7D%2C%22vendor%22%3A%22GoogleInc.%22%2C%22product%22%3A%22Gecko%22%2C%22productSub%22%3A%2220030107%22%2C%22browser%22%3A%7B%22ie%22%3Afalse%2C%22chrome%22%3Atrue%2C%22webdriver%22%3Atrue%7D%2C%22window%22%3A%7B%22historyLength%22%3A4%2C%22hardwareConcurrency%22%3A4%2C%22iframe%22%3Afalse%7D%2C%22fonts%22%3A%22Calibri%3BCentury%3BHaettenschweiler%3BMarlett%3BPristina%3BSimHei%3BZWAdobeF%22%7D%2C%22cookies%22%3A1%2C%22setTimeout%22%3A0%2C%22setInterval%22%3A0%2C%22appName%22%3A%22Netscape%22%2C%22platform%22%3A%22Win32%22%2C%22syslang%22%3A%22zh-CN%22%2C%22userlang%22%3A%22zh-CN%22%2C%22cpu%22%3A%22%22%2C%22productSub%22%3A%2220030107%22%2C%22plugins%22%3A%7B%220%22%3A%22ChromePDFPlugin%22%2C%221%22%3A%22ChromePDFViewer%22%2C%222%22%3A%22NativeClient%22%7D%2C%22mimeTypes%22%3A%7B%220%22%3A%22application%2Fpdf%22%2C%221%22%3A%22PortableDocumentFormatapplication%2Fx-google-chrome-pdf%22%2C%222%22%3A%22NativeClientExecutableapplication%2Fx-nacl%22%2C%223%22%3A%22PortableNativeClientExecutableapplication%2Fx-pnacl%22%7D%2C%22screen%22%3A%7B%22width%22%3A1920%2C%22height%22%3A1080%2C%22colorDepth%22%3A24%7D%2C%22fonts%22%3A%7B%220%22%3A%22Calibri%22%2C%221%22%3A%22Cambria%22%2C%222%22%3A%22Times%22%2C%223%22%3A%22Constantia%22%2C%224%22%3A%22LucidaBright%22%2C%225%22%3A%22Georgia%22%2C%226%22%3A%22SegoeUI%22%2C%227%22%3A%22Candara%22%2C%228%22%3A%22TrebuchetMS%22%2C%229%22%3A%22Verdana%22%2C%2210%22%3A%22Consolas%22%2C%2211%22%3A%22LucidaConsole%22%2C%2212%22%3A%22LucidaSansTypewriter%22%2C%2213%22%3A%22DejaVuSansMono%22%2C%2214%22%3A%22CourierNew%22%2C%2215%22%3A%22Courier%22%7D%7D")
Out[77]: ‘{“proof”:“b3:1545721180848:HcCAd39S7g05mx3PpKsI”,“fp2”:{“userAgent”:“Mozilla/5.0(WindowsNT10.0;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/71.0.3578.98Safari/537.36”,“language”:“zh-CN”,“screen”:{“width”:1920,“height”:1080,“availHeight”:1040,“availWidth”:1920,“pixelDepth”:24,“innerWidth”:968,“innerHeight”:889,“outerWidth”:1539,“outerHeight”:1020,“devicePixelRatio”:1},“timezone”:8,“indexedDb”:true,“addBehavior”:false,“openDatabase”:true,“cpuClass”:“unknown”,“platform”:“Win32”,“doNotTrack”:“unknown”,“plugins”:“ChromePDFPlugin::PortableDocumentFormat::application/x-google-chrome-pdfpdf;ChromePDFViewer::::application/pdfpdf;NativeClient::::application/x-nacl,application/x-pnacl”,“canvas”:{“winding”:“yes”,“towebp”:true,“blending”:true,“img”:“f41a4128d68ea76d3784bd619744f7b83fe826eb”},“webGL”:{“img”:“bd6549c125f67b18985a8c509803f4b883ff810c”,“extensions”:“ANGLE_instanced_arrays;EXT_blend_minmax;EXT_color_buffer_half_float;EXT_disjoint_timer_query;EXT_frag_depth;EXT_shader_texture_lod;EXT_texture_filter_anisotropic;WEBKIT_EXT_texture_filter_anisotropic;EXT_sRGB;OES_element_index_uint;OES_standard_derivatives;OES_texture_float;OES_texture_float_linear;OES_texture_half_float;OES_texture_half_float_linear;OES_vertex_array_object;WEBGL_color_buffer_float;WEBGL_compressed_texture_s3tc;WEBKIT_WEBGL_compressed_texture_s3tc;WEBGL_compressed_texture_s3tc_srgb;WEBGL_debug_renderer_info;WEBGL_debug_shaders;WEBGL_depth_texture;WEBKIT_WEBGL_depth_texture;WEBGL_draw_buffers;WEBGL_lose_context;WEBKIT_WEBGL_lose_context”,“aliasedlinewidthrange”:"[1,1]",“aliasedpointsizerange”:"[1,1024]",“alphabits”:8,“antialiasing”:“yes”,“bluebits”:8,“depthbits”:24,“greenbits”:8,“maxanisotropy”:16,“maxcombinedtextureimageunits”:32,“maxcubemaptexturesize”:16384,“maxfragmentuniformvectors”:1024,“maxrenderbuffersize”:16384,“maxtextureimageunits”:16,“maxtexturesize”:16384,“maxvaryingvectors”:30,“maxvertexattribs”:16,“maxvertextextureimageunits”:16,“maxvertexuniformvectors”:4096,“maxviewportdims”:"[16384,16384]",“redbits”:8,“renderer”:“WebKitWebGL”,“shadinglanguageversion”:“WebGLGLSLES1.0(OpenGLESGLSLES1.0Chromium)”,“stencilbits”:0,“vendor”:“WebKit”,“version”:“WebGL1.0(OpenGLES2.0Chromium)”,“vertexshaderhighfloatprecision”:23,“vertexshaderhighfloatprecisionrangeMin”:127,“vertexshaderhighfloatprecisionrangeMax”:127,“vertexshadermediumfloatprecision”:23,“vertexshadermediumfloatprecisionrangeMin”:127,“vertexshadermediumfloatprecisionrangeMax”:127,“vertexshaderlowfloatprecision”:23,“vertexshaderlowfloatprecisionrangeMin”:127,“vertexshaderlowfloatprecisionrangeMax”:127,“fragmentshaderhighfloatprecision”:23,“fragmentshaderhighfloatprecisionrangeMin”:127,“fragmentshaderhighfloatprecisionrangeMax”:127,“fragmentshadermediumfloatprecision”:23,“fragmentshadermediumfloatprecisionrangeMin”:127,“fragmentshadermediumfloatprecisionrangeMax”:127,“fragmentshaderlowfloatprecision”:23,“fragmentshaderlowfloatprecisionrangeMin”:127,“fragmentshaderlowfloatprecisionrangeMax”:127,“vertexshaderhighintprecision”:0,“vertexshaderhighintprecisionrangeMin”:31,“vertexshaderhighintprecisionrangeMax”:30,“vertexshadermediumintprecision”:0,“vertexshadermediumintprecisionrangeMin”:31,“vertexshadermediumintprecisionrangeMax”:30,“vertexshaderlowintprecision”:0,“vertexshaderlowintprecisionrangeMin”:31,“vertexshaderlowintprecisionrangeMax”:30,“fragmentshaderhighintprecision”:0,“fragmentshaderhighintprecisionrangeMin”:31,“fragmentshaderhighintprecisionrangeMax”:30,“fragmentshadermediumintprecision”:0,“fragmentshadermediumintprecisionrangeMin”:31,“fragmentshadermediumintprecisionrangeMax”:30,“fragmentshaderlowintprecision”:0,“fragmentshaderlowintprecisionrangeMin”:31,“fragmentshaderlowintprecisionrangeMax”:30},“touch”:{“maxTouchPoints”:0,“touchEvent”:false,“touchStart”:false},“video”:{“ogg”:“probably”,“h264”:“probably”,“webm”:“probably”},“audio”:{“ogg”:“probably”,“mp3”:“probably”,“wav”:“probably”,“m4a”:“maybe”},“vendor”:“GoogleInc.”,“product”:“Gecko”,“productSub”:“20030107”,“browser”:{“ie”:false,“chrome”:true,“webdriver”:true},“window”:{“historyLength”:4,“hardwareConcurrency”:4,“iframe”:false},“fonts”:“Calibri;Century;Haettenschweiler;Marlett;Pristina;SimHei;ZWAdobeF”},“cookies”:1,“setTimeout”:0,“setInterval”:0,“appName”:“Netscape”,“platform”:“Win32”,“syslang”:“zh-CN”,“userlang”:“zh-CN”,“cpu”:"",“productSub”:“20030107”,“plugins”:{“0”:“ChromePDFPlugin”,“1”:“ChromePDFViewer”,“2”:“NativeClient”},“mimeTypes”:{“0”:“application/pdf”,“1”:“PortableDocumentFormatapplication/x-google-chrome-pdf”,“2”:“NativeClientExecutableapplication/x-nacl”,“3”:“PortableNativeClientExecutableapplication/x-pnacl”},“screen”:{“width”:1920,“height”:1080,“colorDepth”:24},“fonts”:{“0”:“Calibri”,“1”:“Cambria”,“2”:“Times”,“3”:“Constantia”,“4”:“LucidaBright”,“5”:“Georgia”,“6”:“SegoeUI”,“7”:“Candara”,“8”:“TrebuchetMS”,“9”:“Verdana”,“10”:“Consolas”,“11”:“LucidaConsole”,“12”:“LucidaSansTypewriter”,“13”:“DejaVuSansMono”,“14”:“CourierNew”,“15”:“Courier”}}’
得到一个字典：
1{‘proof’: ‘b3:1545721180848:HcCAd39S7g05mx3PpKsI’,
2 ‘fp2’: {‘userAgent’: ‘Mozilla/5.0(WindowsNT10.0;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/71.0.3578.98Safari/537.36’,
3 ‘language’: ‘zh-CN’,
4 ‘screen’: {‘width’: 1920,
5 ‘height’: 1080,
6 ‘availHeight’: 1040,
7 ‘availWidth’: 1920,
8 ‘pixelDepth’: 24,
9 ‘innerWidth’: 968,
10 ‘innerHeight’: 889,
11 ‘outerWidth’: 1539,
12 ‘outerHeight’: 1020,
13 ‘devicePixelRatio’: 1},
14 ‘timezone’: 8,
15 ‘indexedDb’: True,
16 ‘addBehavior’: False,
17 ‘openDatabase’: True,
18 ‘cpuClass’: ‘unknown’,
19 ‘platform’: ‘Win32’,
20 ‘doNotTrack’: ‘unknown’,
21 ‘plugins’: ‘ChromePDFPlugin::PortableDocumentFormat::application/x-google-chrome-pdfpdf;ChromePDFViewer::::application/pdfpdf;NativeClient::::application/x-nacl,application/x-pnacl’,
22 ‘canvas’: {‘winding’: ‘yes’,
23 ‘towebp’: True,
24 ‘blending’: True,
25 ‘img’: ‘f41a4128d68ea76d3784bd619744f7b83fe826eb’},
26 ‘webGL’: {‘img’: ‘bd6549c125f67b18985a8c509803f4b883ff810c’,
27 ‘extensions’: ‘ANGLE_instanced_arrays;EXT_blend_minmax;EXT_color_buffer_half_float;EXT_disjoint_timer_query;EXT_frag_depth;EXT_shader_texture_lod;EXT_texture_filter_anisotropic;WEBKIT_EXT_texture_filter_anisotropic;EXT_sRGB;OES_element_index_uint;OES_standard_derivatives;OES_texture_float;OES_texture_float_linear;OES_texture_half_float;OES_texture_half_float_linear;OES_vertex_array_object;WEBGL_color_buffer_float;WEBGL_compressed_texture_s3tc;WEBKIT_WEBGL_compressed_texture_s3tc;WEBGL_compressed_texture_s3tc_srgb;WEBGL_debug_renderer_info;WEBGL_debug_shaders;WEBGL_depth_texture;WEBKIT_WEBGL_depth_texture;WEBGL_draw_buffers;WEBGL_lose_context;WEBKIT_WEBGL_lose_context’,
28 ‘aliasedlinewidthrange’: ‘[1,1]’,
29 ‘aliasedpointsizerange’: ‘[1,1024]’,
30 ‘alphabits’: 8,
31 ‘antialiasing’: ‘yes’,
32 ‘bluebits’: 8,
33 ‘depthbits’: 24,
34 ‘greenbits’: 8,
35 ‘maxanisotropy’: 16,
36 ‘maxcombinedtextureimageunits’: 32,
37 ‘maxcubemaptexturesize’: 16384,
38 ‘maxfragmentuniformvectors’: 1024,
39 ‘maxrenderbuffersize’: 16384,
40 ‘maxtextureimageunits’: 16,
41 ‘maxtexturesize’: 16384,
42 ‘maxvaryingvectors’: 30,
43 ‘maxvertexattribs’: 16,
44 ‘maxvertextextureimageunits’: 16,
45 ‘maxvertexuniformvectors’: 4096,
46 ‘maxviewportdims’: ‘[16384,16384]’,
47 ‘redbits’: 8,
48 ‘renderer’: ‘WebKitWebGL’,
49 ‘shadinglanguageversion’: ‘WebGLGLSLES1.0(OpenGLESGLSLES1.0Chromium)’,
50 ‘stencilbits’: 0,
51 ‘vendor’: ‘WebKit’,
52 ‘version’: ‘WebGL1.0(OpenGLES2.0Chromium)’,
53 ‘vertexshaderhighfloatprecision’: 23,
54 ‘vertexshaderhighfloatprecisionrangeMin’: 127,
55 ‘vertexshaderhighfloatprecisionrangeMax’: 127,
56 ‘vertexshadermediumfloatprecision’: 23,
57 ‘vertexshadermediumfloatprecisionrangeMin’: 127,
58 ‘vertexshadermediumfloatprecisionrangeMax’: 127,
59 ‘vertexshaderlowfloatprecision’: 23,
60 ‘vertexshaderlowfloatprecisionrangeMin’: 127,
61 ‘vertexshaderlowfloatprecisionrangeMax’: 127,
62 ‘fragmentshaderhighfloatprecision’: 23,
63 ‘fragmentshaderhighfloatprecisionrangeMin’: 127,
64 ‘fragmentshaderhighfloatprecisionrangeMax’: 127,
65 ‘fragmentshadermediumfloatprecision’: 23,
66 ‘fragmentshadermediumfloatprecisionrangeMin’: 127,
67 ‘fragmentshadermediumfloatprecisionrangeMax’: 127,
68 ‘fragmentshaderlowfloatprecision’: 23,
69 ‘fragmentshaderlowfloatprecisionrangeMin’: 127,
70 ‘fragmentshaderlowfloatprecisionrangeMax’: 127,
71 ‘vertexshaderhighintprecision’: 0,
72 ‘vertexshaderhighintprecisionrangeMin’: 31,
73 ‘vertexshaderhighintprecisionrangeMax’: 30,
74 ‘vertexshadermediumintprecision’: 0,
75 ‘vertexshadermediumintprecisionrangeMin’: 31,
76 ‘vertexshadermediumintprecisionrangeMax’: 30,
77 ‘vertexshaderlowintprecision’: 0,
78 ‘vertexshaderlowintprecisionrangeMin’: 31,
79 ‘vertexshaderlowintprecisionrangeMax’: 30,
80 ‘fragmentshaderhighintprecision’: 0,
81 ‘fragmentshaderhighintprecisionrangeMin’: 31,
82 ‘fragmentshaderhighintprecisionrangeMax’: 30,
83 ‘fragmentshadermediumintprecision’: 0,
84 ‘fragmentshadermediumintprecisionrangeMin’: 31,
85 ‘fragmentshadermediumintprecisionrangeMax’: 30,
86 ‘fragmentshaderlowintprecision’: 0,
87 ‘fragmentshaderlowintprecisionrangeMin’: 31,
88 ‘fragmentshaderlowintprecisionrangeMax’: 30},
89 ‘touch’: {‘maxTouchPoints’: 0, ‘touchEvent’: False, ‘touchStart’: False},
90 ‘video’: {‘ogg’: ‘probably’, ‘h264’: ‘probably’, ‘webm’: ‘probably’},
91 ‘audio’: {‘ogg’: ‘probably’,
92 ‘mp3’: ‘probably’,
93 ‘wav’: ‘probably’,
94 ‘m4a’: ‘maybe’},
95 ‘vendor’: ‘GoogleInc.’,
96 ‘product’: ‘Gecko’,
97 ‘productSub’: ‘20030107’,
98 ‘browser’: {‘ie’: False, ‘chrome’: True, ‘webdriver’: True},
99 ‘window’: {‘historyLength’: 4, ‘hardwareConcurrency’: 4, ‘iframe’: False},
100 ‘fonts’: ‘Calibri;Century;Haettenschweiler;Marlett;Pristina;SimHei;ZWAdobeF’},
101 ‘cookies’: 1,
102 ‘setTimeout’: 0,
103 ‘setInterval’: 0,
104 ‘appName’: ‘Netscape’,
105 ‘platform’: ‘Win32’,
106 ‘syslang’: ‘zh-CN’,
107 ‘userlang’: ‘zh-CN’,
108 ‘cpu’: ‘’,
109 ‘productSub’: ‘20030107’,
110 ‘plugins’: {‘0’: ‘ChromePDFPlugin’,
111 ‘1’: ‘ChromePDFViewer’,
112 ‘2’: ‘NativeClient’},
113 ‘mimeTypes’: {‘0’: ‘application/pdf’,
114 ‘1’: ‘PortableDocumentFormatapplication/x-google-chrome-pdf’,
115 ‘2’: ‘NativeClientExecutableapplication/x-nacl’,
116 ‘3’: ‘PortableNativeClientExecutableapplication/x-pnacl’},
117 ‘screen’: {‘width’: 1920, ‘height’: 1080, ‘colorDepth’: 24},
118 ‘fonts’: {‘0’: ‘Calibri’,
119 ‘1’: ‘Cambria’,
120 ‘2’: ‘Times’,
121 ‘3’: ‘Constantia’,
122 ‘4’: ‘LucidaBright’,
123 ‘5’: ‘Georgia’,
124 ‘6’: ‘SegoeUI’,
125 ‘7’: ‘Candara’,
126 ‘8’: ‘TrebuchetMS’,
127 ‘9’: ‘Verdana’,
128 ‘10’: ‘Consolas’,
129 ‘11’: ‘LucidaConsole’,
130 ‘12’: ‘LucidaSansTypewriter’,
131 ‘13’: ‘DejaVuSansMono’,
132 ‘14’: ‘CourierNew’,
133 ‘15’: ‘Courier’}}
不得不惊叹获取的信息之丰富，基本上获取了浏览器环境的所有信息和配置，一看这个字典有点熟悉，回忆了一下和浏览器的window.navigator对象数据相似，打开浏览器控制台输入window.navigator得到下面参数：
而其中区分selenium控制与非selenium控制的参数是:
1 ‘browser’: {‘ie’: False, ‘chrome’: True, ‘webdriver’: True},
其中正常浏览器的’webdriver’是undefined,到此这个网站的反扒措施大致清楚了：
IP来源控制，只接受信赖的来源
driver环境监测
那么driver环境监测具体是怎么实现的呢？
我的猜想是采用浏览器认证方式：首次访问浏览器会把浏览器的window.navigator信息发送给服务器验证，验证通过后就返回一个认证过的cookie标识：
为什么会有这种猜想呢？因为看到请求的URL：
PID=EC412E1F-C8F4-3AA3-B67B-48A1565D374B
带有PID很像一个身份ID，同时看待cookie返回了很多ID相关字段，应该猜想到是浏览器的身份认证。
这与之前关注的selenium淘宝登陆和selenium美团采集对selenium的检测一样的原理。
如何突破呢？
针对这个根据这个网站的采集需求：低频、四五千公司，采用requests的Session状态保持，复制一个正常浏览器认证过的cookie添加在访问中即可，事实证明这种方法简单有效，可以成功访问。
还有一些其他突破方法：
在driver执行js
js = r"Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined,});"
self.driver.execute_script(js)
但是这种方法不持久，刷新后会重新检测，恢复真实值。
使用mitmproxy 这个有点复杂在后面在具体说明，也可点击原文链接查看mitmproxy 的案列。