Table of contents
find_element and find_elements usage:
Operation website sample code:
ActionChains(driver) mouse event:
Enter account password to log in:
pyHook listens to the mouse and prints the position
Determine whether the node exists:
Mutual conversion between dictionary dict and json:
Python json.dumps() Chinese garbled problem
python-selenium switch mobile phone mode:
Selenium-TouchActions interface behavior control, gesture control:
selenium-open developer tools (F12):
Get browser Network request and response
Introduction to Selenium
Selenium-WEB automation tool
Introduction: Selenium is a tool for web application testing. Selenium tests run directly in the browser, just like real users.
Supported browsers: IE (7, 8, 9, 10, 11), Mozilla Firefox, Safari, Google Chrome, Opera, etc.
The main function:
- Test compatibility with browsers—Support multiple platforms and multiple browsers, test your application to see if it can work well on different browsers and operating systems.
- Test System Functionality—Create regression tests to verify software functionality and user requirements. Supports automatic recording of actions and automatic generation of test scripts in different languages such as .Net, Java, and Perl.
- The bottom layer of the framework uses JavaScript to simulate real users operating the browser. When the test script is executed, the browser automatically performs operations such as click, input, open, and verification according to the script code, just like real users do, and the application is tested from the perspective of the end user.
- It is possible to automate browser compatibility testing, although there are still subtle differences on different browsers.
- It is easy to use and can write use case scripts in multiple languages such as Java and Python.
Install
python install selenium command:
pip install seleniumDownload chromedriver.exe
The chromedriver.exe file is the driver file for calling chrome. Various versions of Google Chrome correspond to different chromedrivers, so you must choose the correct chromedriver according to your chrome version. Download address of chromedriver:
http://chromedriver.storage.googleapis.com/index.html
Check the current version of Google Chrome on your computer:
Open Google Chrome -> Three dots under the close icon in the upper right corner -> Help -> About Google Chrome
Then open the address just now and find the version number that is the same as or similar to your computer’s Google Chrome version and click on it.
I am a windows system here, so I choose chromedriver_win32.zip to download
Put the downloaded chromedriver.exe in the installation directory of Google Chrome, my path here is
C:\Program Files\Google\Chrome\ApplicationSet the system environment variable and add the path of chromedriver.exe above to Path
mac和linux ：sudo mv chromedriver /usr/local/bin/chromedriver
At this point, the preparatory work is complete.
use
Python code example:
from selenium import webdriver
s = Service(r"C:\Program Files\Google\Chrome\Application\chromedriver.exe")
driver = webdriver.Chrome(service=s)
driver.get("http://www.baidu.com")
driver.find_element_by_id("kw").send_keys("selenium")
# 通过 id=kw 定位到百度的输入框,并通过键盘方法send_keys()向输入框里输入selenium
driver.find_element_by_id("su").click()
# 通过 id=su 定位到搜索按钮，并向按钮发送单击事件（click() ）
# driver.quit()
# 退出并关闭窗口的每一个相关的驱动程序common method
Positioning element method
# 查找单个元素:
find_element_by_id()
find_element_by_name()
find_element_by_xpath()
find_element_by_link_text()
find_element_by_partial_link_text()
find_element_by_tag_name()
find_element_by_class_name()
find_element_by_css_selector()
# 查找多个元素
find_elements_by_name()
find_elements_by_xpath()
find_elements_by_link_text()
find_elements_by_partial_link_text()
find_elements_by_tag_name()
find_elements_by_class_name()
find_elements_by_css_selector()
# 两个私有方法
find_element()
find_elements()find_element and find_elements usage:
from selenium.webdriver.common.by import By
driver.find_element(By.XPATH, '//button[text()="Some text"]')
driver.find_elements(By.XPATH, '//button')
XPATH = "xpath"
LINK_TEXT = "link text"
PARTIAL_LINK_TEXT = "partial link text"
NAME = "name"
TAG_NAME = "tag name"
CLASS_NAME = "class name"
CSS_SELECTOR = "css selector"find_element_by_xpath usage:
绝对路径：login_form = driver.find_element_by_xpath("/html/body/form[1]")
HTML中的第一个表单元素： login_form = driver.find_element_by_xpath("//form[1]")
属性id=loginForm的元素：login_form = driver.find_element_by_xpath("//form[@id='loginForm']")
'//div/*' div下面的所有的元素
'//div//p' 查找div中的p节点，等于 css_selector里的('div p')
'//div/p' 查找div的子节点p； 等价于 css_selector里的('div > p')
'//*[@style]' 查找所有包含style的所有元素，所有的属性要加@，等于('*[style]')：
'//p[@spec='len']' 必须要加引号；等价于 css_selector里的("p[spec='len']")
'//p[@id='kw']' xpath中对于id,class与其他元素一视同仁，没有其他的方法
'//div/p[2]' 选择div下的第二个p节点 ；等价于css_selector里的div>p:nth-of-type(2) 符合p类型的第二个节点
'//div/*[2]' 选择div下第二个元素
'//div/p[position()=2]' position()=2 指定第二个位置； 等价于上面的 //div/p[2]
position()>=2 位置大于等于2
position()<2 位置小于2
position()!=2 位置不等于2
'//div/p[last()]' 选择div下的倒数第一个p节点； last()倒数第一个
'//div/p[last()-1]' 选择div下的倒数第二个p节点；
'//div/p[position()=last()]' 倒数第一个
'//div/p[position()=last()-1]' 倒数第二个
'//div/p[position()>=last()-2]'倒数第一个，第二个，第三个
'//p | //button' 选择所有的p和button，等价于css_selector里的 p, button
'//input[@id='kw' and @class='su']' 选择id=kw 并且 class=su的input元素
'//p[@spec='len']/..' 选择p节点的上层节点 此方法在css_selector中没有
'//p[@spec='len']/../..' 上层节点的上层节点Operation website sample code:
from selenium import webdriver
url = 'http://xxxxxxx'
driver = webdriver.Chrome()
# 隐式等待页面加载完
driver.implicitly_wait(20)
# 设置窗口大小
# driver.set_window_size(1000, 800)
# 全屏操作
# driver.maximize_window()
driver.get(url)
aList = driver.find_elements_by_xpath('//*[@id="list"]//a')
driver.close()Get webpage/text:
driver.page_source 获取整个网页源代码
get_attribute("outerHTML") 输出当前标签的本身和标签内的文本内容，如果有子标签，那么子标签本身和标签内的文本内容也将一起输出
get_attribute('innerHTML') 获取当前标签的文本内容，如果标签内有子标签，会连子标签本身和子标签内的文本内容一起输出
get_attribute('textContent') == .text
get_attribute：这个标签的某个属性的值。
screentshot：获取当前页面的截图。这个方法只能在driver上使用。
获取所有的cookie：
for cookie in driver.get_cookies():
print(cookie)Operation node/get/textbox:
# 获取标签内的属性值
aList[i].get_attribute('src')
# 获取标签下的文本
aList[i].text
# 关闭页面
driver.close()
# 操作输入框：分为两步。第一步：找到这个元素。
# 第二步：使用send_keys(value)，将数据填充进去
inputTag = driver.find_element_by_id('kw')
inputTag.send_keys('python')
# 清除输入框中的内容
inputTag.clear()
# 操作checkbox
# 要选中checkbox标签，在网页中是通过鼠标点击的。因此想要选中checkbox标签，那么先选中这个
# 标签，然后执行click事件
rememberTag = driver.find_element_by_name("rememberMe")
rememberTag.click()
# 选择select，select元素不能直接点击。因为点击后还需要选中元素。这时候selenium就专门
# 为select标签提供了一个类，示例：
from selenium.webdriver.support.ui import Select
# 选中这个标签，然后使用Select创建对象
selectTag = Select(driver.find_element_by_name("jumpMenu"))
# 根据索引选择
selectTag.select_by_index(1)
# 根据值选择
selectTag.select_by_value("http://www.xxx.com")
# 根据可视的文本选择
selectTag.select_by_visible_text("测试")
# 取消选中所有选项
selectTag.deselect_all()
# 按钮点击
inputTag = driver.find_element_by_id('su')
inputTag.click()
# 行为链：
# 有时候在页面中的操作可能要有很多步，那么这时候可以使用鼠标行为链类ActionChains来完成。
# 比如现在要将鼠标移动到某个元素上并执行点击事件。那么示例代码如下：
inputTag = driver.find_element_by_id('kw')
submitTag = driver.find_element_by_id('su')ActionChains(driver) mouse event:
actions = ActionChains(driver) # 实例化一个action对象
# 移动操作
actions.move_to_element(inputTag) # 把鼠标移动到inputTag元素上,鼠标悬停
actions.move_by_offset(10, 20) # 从当前鼠标的位置向水平方向移动10个像素，向竖直方向移动20个像素
actions.send_keys_to_element(inputTag,'python')
actions.send_keys("追光者") # 输入“追光者”
# 全选，复制,粘贴(跨浏览器)
actions.key_down(Keys.CONTROL).send_keys(‘a‘).key_up(Keys.CONTROL) #ctrl+a
actions.key_down(Keys.CONTROL).send_keys(‘c‘).key_up(Keys.CONTROL) #ctrl+c
actions.key_down(Keys.CONTROL,search_key).send_keys(‘v‘).key_up(Keys.CONTROL) #ctrl+v
search_key=driver.find_element_by_id("search-key")
search_key.send_keys(Keys.CONTROL,'v')
key_down,发送，只是按下按键，并不会自动松开，而send_keys发送的键，会实现按下-松开的全过程
# 拖拽操作
actions.drag_and_drop(elem1, elem2).perform() # 把elem1元素拖拽到elem2上
actions.click_and_hold(elem1).release(elem2).perform() # 在elem1元素上按下鼠标左键不松，在elem2元素上释放，实现拖拽操作
# 点击操作
actions.click(submitTag) # 单击
actions.click_and_hold(element) # 鼠标左键按下不放
actions.context_click(element) # 右键点击。
actions.double_click(element) # 鼠标双击。
# 按下鼠标左键在一个元素上,release()释放
ActionChains(driver).click_and_hold(element).perform()
ActionChains(driver).release(element).perform()
actions.perform() # 执行所有ActionChains 中存储的行为For more methods, please check the official documentation: 7. WebDriver API — Selenium Python Bindings 2 documentation
Slide to unlock:
time.sleep(3)
button = driver.find_element_by_xpath('//*[@id="login-slider"]/div[1]/div[2]/div')
action = ActionChains(driver) # 实例化一个action对象
action.click_and_hold(button).perform() # 鼠标左键按下不放
action.move_by_offset(400, 0).perform() # 平行移动鼠标
action.reset_actions()Enter account password to log in:
inputTag = driver.find_element_by_xpath('//*[@id="loginPage"]/ul/li[1]/div/input')
inputTag.send_keys('admin')
inputTag2 = driver.find_element_by_xpath('//*[@id="loginPage"]/ul/li[2]/div/input')
inputTag2.send_keys('123456')
driver.find_element_by_xpath('//*[@id="loginPage"]/div/button').click()Sometimes Python is also used to control mouse clicks, please move to pyautogui this article: PyautoGui automates mouse and keyboard operations and organizes commonly used functions
pyHook listens to the mouse and prints the position
import pythoncom
import PyHook3 as pyHook
# 监听到鼠标事件调用
def onMouseEvent(event):
if (event.MessageName == "mouse left down") and (event.MessageName != "mouse move"):
# 因为鼠标一动就会有很多mouse move，所以把这个过滤下
x,y = pag.position() #返回鼠标的坐标
print(x,y)
return True # 为True才会正常调用，如果为False的话，此次事件被拦截
# 创建管理器
hm = pyHook.HookManager()
# 监听鼠标
hm.MouseAll = onMouseEvent
hm.HookMouse()
# 循环监听
pythoncom.PumpMessages()Determine whether the node exists:
def isElementPresent(driver, path):
#从selenium.common.exceptions 模块导入 NoSuchElementException类
from selenium.common.exceptions import NoSuchElementException
try:
element = driver.find_element_by_xpath(path)
#原文是except NoSuchElementException, e:
except NoSuchElementException as e:
#打印异常信息
# print(e)
#发生了NoSuchElementException异常，说明页面中未找到该元素，返回False
return False
else:
#没有发生异常，表示在页面中找到了该元素，返回True
return True
res = isElementPresent(driver, "/html/div")
if res is True:
print('节点存在')Set proxy ip:
You can also set some browser options, such as the save location after the file is downloaded.
Sometimes some web pages are frequently crawled. After the server finds out that you are a crawler, it will block your ip address. At this time we can change the proxy ip. To change the proxy ip, different browsers have different implementations. Here is an example of Chrome browser:
from selenium import webdriver
# google driver文件位置
s = Service(r"C:\Users\DELL\AppData\Local\Google\Chrome\Application\chromedriver.exe")
# 创建Chrome浏览器配置对象
chromeOptions = webdriver.ChromeOptions()
# 设定下载文件的保存目录为C:\Users\DELL\Desktop\自定义下载位置
# 如果该目录不存在则直接创建
prefs = {"download.default_directory": r"C:\Users\DELL\Desktop\自定义下载位置"}
# 将自定义设置添加到chrome配置对象实例中
chromeOptions.add_experimental_option("prefs", prefs)
# 设置代理ip
chromeOptions.add_argument("--proxy-server=http://110.73.2.248:8123")
# 启动带有自定义设置的chrome浏览器
# driver=webdriver.Chrome(executable_path='C:\Users\DELL\AppData\Local\Google\Chrome\Application\chromedriver.exe',chrome_options=chromeOptions)
driver = webdriver.Chrome(service=s, chrome_options=chromeOptions)
driver.set_window_size(1200, 1000)
driver.get('http://xxxxx')tab page switching
Sometimes there are many sub-tab pages in the window. At this time, it is definitely necessary to switch. Selenium provides a function called switch_to_window to switch, which page to switch to, can be found from driver.window_handles. The sample code is as follows
# 当前打开的所有窗口
windows = driver.window_handles
# 转换到最新打开的窗口
driver.switch_to.window(windows[-1])
for handle in windows : # 切换窗口
if handle != driver.current_window_handle:
print('switch to second window', handle)
driver.close() # 关闭第一个窗口
driver.switch_to.window(handle) # 切换到第二个窗口html to string:
driver.find_element_by_xpath('/html/body').get_attribute("outerHTML").__str__().replace('"', "'")Mutual conversion between dictionary dict and json:
import json
json.loads() 将json转换为dict
json.dumps() 将dict转换为json
json.load() 将json文件转换为dict
json.dump() 将dict转换为json文件 person.json
# 类对象转换为json
person_json = json.dumps(person.__dict__)
# 或者
# 第二个参数传递转换函数，或者使用default=lambda o: o.__dict__
person_json = json.dumps(person, default=convert2json)
# 将person转换为dict
def convert2json(person):
return {
'name': person.name,
'age': person.age,
'email': person.email
}
# dict/对象转换为json文件
with open('person.json', 'w') as f:
json.dump(person, f)
# 将json文件转换为dict/对象
import json
with open('person.json', 'r') as f:
print(json.load(f))Python json.dumps() Chinese garbled problem
json.dumps The encoding used by default during serialization asciineeds to be specified if you want to output real Chinese ensure_ascii=False: in-depth analysis, it should be because it dJSON object is not a simple unicodeimplementation, but contains mixed unicodeencodings and utf-8strings that have been encoded.
Writing:
Python2 ：json.dumps(odata, ensure_ascii=False).decode('utf8')
json.dumps(odata, ensure_ascii=False).decode('utf8').encode('gb2312')
Python3 ：json.dumps(odata, ensure_ascii=False)python-selenium switch mobile phone mode:
# 手机模式
option = webdriver.ChromeOptions()
option.add_argument('disable-infobars')
mobile_emulation = {"deviceName": "iPhone 6"}
option.add_experimental_option('mobileEmulation', mobile_emulation)
driver = webdriver.Chrome(chrome_options=option)Selenium-TouchActions interface behavior control, gesture control:
# 行为控制
perform --- 执行所有准备好的Actio
# 手势控制
tap --- 在指定元素上敲击
double_tap --- 在指定元素上双敲击
tap_and_hold --- 在指定元素上点击但不释放
move --- 手势移动指定偏移(未释放)
release --- 释放手势
scroll --- 手势点击并滚动
scroll_from_element --- 从某个元素位置开始手势点击并滚动
long_press --- 长按元素
flick --- 手势滑动
flick_element --- 从某个元素位置开始手势滑动
例：
from selenium.webdriver.common.touch_actions import TouchActions
Action = TouchActions(driver)
Action.double_tap(driver.find_element_by_xpath('//*[@id="bw"]/div[1]/a[2]'))
flick_element(on_element, xoffset, yoffset, speed)；
on_element #操作元素定位
xoffset #x轴偏移量
yoffset #y轴偏移量
speed #速度
注意：向上滑动为负数，向下滑动为正数
Action = TouchActions(driver)
"""从button元素像下滑动200元素，以50的速度向下滑动"""
Action.flick_element(button, 0, 200, 50).perform()selenium-open developer tools (F12):
option = webdriver.ChromeOptions()
# 开启开发者工具（F12）
option.add_argument("--auto-open-devtools-for-tabs")
driver = webdriver.Chrome(chrome_options=option)Get browser Network request and response
Selenium-Get requested interface data information
Browsermob-Proxy is an open source Java-based proxy service based on LittleProxy. The specific process of Browsermob-Proxy is somewhat similar to Flidder or Charles. That is to open a port and exist as a standard proxy. When the HTTP client (browser, etc.) sets this proxy, it can grab all the request details and get the returned content.
Installation:
Download the compressed package directly from the project's github: Releases · lightbody/browsermob-proxy · GitHub , supporting Linux and Windows.
Install the corresponding python package:
pip install browsermob-proxyFull code:
import json
from browsermobproxy import Server
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
if __name__ == '__main__':
# 开启Proxy
server = Server(r'D:\usr\data\browser\browsermob-proxy-2.1.4\bin\browsermob-proxy.bat')
server.start()
proxy = server.create_proxy()
# 配置Proxy启动WebDriver
chrome_options = Options()
chrome_options.add_argument('--proxy-server={0}'.format(proxy.proxy))
# 解决 您的连接不是私密连接问题
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--ignore-urlfetcher-cert-requests')
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.implicitly_wait(20)
proxy.new_har("douyin", options={'captureHeaders': True, 'captureContent': True})
driver.get(url)
result = proxy.har
for entry in result['log']['entries']:
_url = entry['request']['url']
# 根据URL找到数据接口
if "/hotel/list" in _url:
_response = entry['response']
_content = _response['content']['text']
# 获取接口返回内容
print(_content)
# 读取信息
person_json = json.loads(_content)
hotels = (person_json["data"])["hotels"]
server.stop()