Introduction and installation:
Selenium, as an automated testing tool, is mainly used for web page testing. In Python crawler, we can use this tool to operate on web page elements, such as starting the browser, opening the web page, forward and backward, positioning elements, keyboard input With mouse click operation etc. It can be said that as long as the operation can be performed in the browser, we can use selenium to achieve
OK. First, we need to install selenium.
If Python and pip are installed on your computer, we only need to call
pip install selenium
This command on it
after the installation is complete we need to test
from selenium import webdriver
调用FireFox浏览器
broswer=webdriver.Firefox()
打开百度
broswer.get('http://www.baidu.com')
显示百度的源代码
print(broswer.page_source)In this way, we can open Baidu's page through Firefox and display the source code of Baidu.
PS: Because the page is opened through the browser, the source code at this time is the source code after JS rendering.
If there is an error when the code is running, the system cannot start Firefox, and the error code is
selenium.common.exceptions.WebDriverException: Message: ‘geckodriver’ executable needs to be in PATH.
Reason 1: The version of selenium is too low to support the corresponding version of Firefox; Solution: Download the latest selenium and install it (this reason is generally unlikely)
Reason 2. You are using a version of Firefox 47 or higher, you need Download the third-party driver, geckodriver, and then place geckodriver.exe in the same directory of python.exe or add the directory where geckodriver.exe is located to the environment variable.
Element operation
When we selenium installation is complete, we can use it to operate the website
First, we know that pages are spliced together by a variety of elements, you need to modify pages, we first need to locate you need Modified web page elements.
Let’s take Baidu as an example, we can see that Baidu’s search box is such an element
input id=”kw” name=”wd” class=”s_ipt” value=”” maxlength=”255” autocomplete=”off”
Next, we can locate this element, and then operate the search box
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
broswer=webdriver.Firefox()
broswer.get('http://www.baidu.com')
通过id定位百度搜索框
searchItem=broswer.find_element_by_id('kw')
等待3秒
time.sleep(3)
向搜索框发送文字：图片
searchItem.send_keys('图片')
点击回车
searchItem.send_keys(Keys.ENTER)
显示当前页面的源代码
print(broswer.page_source)Run the code, the system will first open a Firefox browser, and the browser will automatically open the Baidu page, wait three seconds later, the search box will automatically enter the text and click the Enter
Why should
time.sleep(3)
This operation, because the web page takes time to load, we must ensure that all the elements of the web page are loaded before we can proceed.
Just now we have used the Enter key to complete the search, and then we will operate by clicking the search button
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
broswer=webdriver.Firefox()
broswer.get('http://www.baidu.com')
通过id定位百度搜索框
searchItem=broswer.find_element_by_id('kw')
通过class定位“百度一下”按钮
searchButton=broswer.find_element_by_class_name('bg s_btn')
等待3秒
time.sleep(3)
向搜索框发送文字：图片
searchItem.send_keys('图片')
点击搜索按钮
searchButton.click()
显示当前页面的源代码
print(broswer.page_source)
The above code before the code can be achieved and the same operation, but we are to complete the search by clicking on the search button
which is page after the search is completed
Regarding the positioning method of page elements in Selenium , you can refer to this blog
. Eight common methods of Selenium Webdriver element positioning
Frame switching
Page often contains a lot iframe framework, many elements have been placed among the framework
, for example, we need to simulate landing page QQ space
We log in by clicking on the logged-in QQ avatar. First of all, I have obtained the element id of the avatar as'img_out_1356306040', and then we can proceed.
from selenium import webdriver
import time
broswer=webdriver.Firefox()
broswer.get(url='https://i.qq.com')
time.sleep(2)
broswer.find_element_by_id('img_out_1356306040').click()When we run the program, the computer will have this error
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="img_out_1356306040"]This code indicates that no such element with id=”img_out_1356306040” was found.
Let’s take a closer look at the composition of this page
It can be found that the position of the login avatar is placed in an iframe frame with id="login_frame", so we need to locate this frame first, and then we can locate specific elements in the frame
from selenium import webdriver
import time
broswer=webdriver.Firefox()
broswer.get(url='https://i.qq.com')
time.sleep(2)
定位到'login_frame'框架
broswer.switch_to.frame('login_frame')
broswer.find_element_by_id('img_out_1356306040').click()
返回至默认框架
broswer.switch_to.default_content()
print(broswer.page_source)Run the above code, we can successfully log in to the QQ space
to sum up:
The above is the simple usage of selenium. We can use selenium to perform various operations on web pages. I feel that the operations that the browser can perform are as if they can be done through selenium. In short, it is a very powerful tool. Through selenium, we can further Use python crawler to crawl Internet data