Selenium is an open source, free, simple and flexible automated testing tool with good support for web browsers. It is very practical in UI automation, crawler and other scenarios. Being able to master and use Selenium tools proficiently can greatly improve efficiency.
Introduction to Selenium
Selenium supports multiple platforms, multiple browsers, and multiple languages to implement automated testing. It is an open source and portable web testing framework that supports parallel test execution, thereby reducing time and improving testing efficiency. Using it, we can write relevant automated programs so that the program can operate the Web interface in the browser exactly like a human, such as simulating mouse clicks, simulating keyboard input, and so on. Not only can it operate the Web interface, but it can also obtain information from the Web. Relatively speaking, it is easier to use Selenium to obtain information. Its basic principle is that after we write the automation program, we use the browser driver to directly operate the browser. As long as we The information that users can obtain on the browser can be obtained using Selenium.
Environmental preparation
Download the browser driver. Note that the driver version must be consistent with the browser version.
Add the browser driver path to the environment variable path
Install Selenium package pip install selenium
Quick start
Use selenium to control the browser to open Baidu homepage and search for Alipay.
from selenium import webdriver # Import webdriver
import time
driver = webdriver.Chrome() # Get the browser driver
driver.get(" http://www.baidu.com ") # Open Baidu homepage
input_box = driver.find_element_by_id('kw' ) # Get the home page input box element
input_box.send_keys('Alipay') # Enter content into the input box
search_button = driver.find_element_by_id('su') # Get the home page search button element
search_button.click() # Click the search button
time.sleep (5)
driver.quit() # Close the driver
Selenium-API operation [free sharing of Selenium automated testing learning resources at the end of the article]
element wait
show wait
Set a timeout, and check whether the element exists every time. If it exists, execute the subsequent content. If the maximum time (timeout) is exceeded, a timeout exception (TimeoutException) will be thrown. To display waiting, you need to use WebDriverWait with until or not until.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('Http://www.baidu.com')
# Check whether the element exists every 0.5 seconds. If it is not found within 5 seconds, an exception will be thrown.
element = WebDriverWait(driver, 5, 0.5).until(
expected_conditions.presence_of_element_located((By.ID, 'kd')), message='Element not found')
implicit wait
Implicit waiting also specifies a timeout. If the specified element has not been loaded beyond this time, a NoSuchElementException exception will be thrown. In addition to the different exceptions thrown, there is another point: implicit wait is global, that is, during the running process, if the element can be located, it will not affect the code running, but if it cannot be located, it will poll The method continuously accesses the element until the element is found. If the specified time exceeds, an exception is thrown. Use implicitly_wait() to implement implicit waiting, which is much easier to use than explicit waiting.
driver.implicitly_wait(5)
forced wait
Use time.sleep() to force waiting, set a fixed sleep time, and wait for elements to load, which will have an impact on the running efficiency of the code.
element positioning
Selenium provides 8 basic element positioning methods, namely id, name, class name, tag name, link text, partial link text, xpath, css selector, where id, name, class name, tag name are based on the tag of the element Or the attributes of the element for positioning; link text, partial link text are positioned according to the text of the hyperlink; xpath is the element path positioning; css is the selector positioning (style positioning).
element = driver.find_element_by_id('kw') # Get the element through the id attribute
element = driver.find_element_by_name('wd') # Get the element through the name attribute
element = driver.find_element_by_class_name('input') # Get the element through the class attribute
element = driver.find_element_by_tag_name('input') # Get the element by tag name
element = driver.find_element_by_link_text('Video') # Get the element through the link text value
element = driver.find_element_by_partial_link_text('view')# Get the element through the partial link text value
element = driver.find_element_by_xpath("//*[@id='kw']") # Get elements through Xpath
element = driver.find_element_by_css_selector('#kw') # Get elements through CSS selector
Element operations
After locating the element and obtaining the element object, we can perform the operations we want on the element. Selenium provides many APIs for us to operate elements. Common operations include click, input, clear, obtain element coordinate values, and obtain element width and height. Value, element attribute value, check whether the element is selected.
elemnet.click() # Click element
element.send_keys('Wuhan') #Input content
element.clear() # Clear content
element.location.get('x') # Get the X-axis coordinate of the upper left corner of the element
element.location.get('y') # Get the Y-axis coordinate of the upper left corner of the element
element.size.get('width') # Get the element width
element.size.get('height') # Get the height of the element
element.is_selected() # Whether the element is selected
Mouse operation
Common mouse operations include: click, right-click, double-click, hover, drag, etc. Selenium encapsulates corresponding operation methods for these mouse operations.
In Selenium, the methods of operating the mouse are encapsulated in the ActionChains class. All mouse event methods provided in the ActionChains class, when called, all behaviors are stored in the ActionChains object, and the perform() method is to actually execute all Mouse events.
from selenium.webdriver.common.action_chains import ActionChains # 导包
actionChains = ActionChains(driver) # Instantiate ActionChains object
actionChains.move_to_element(element).perform() # Mouseover
actionChains.drag_and_drop(elementA, elementB).perform() # Mouse drag
actionChains.context_click(element).perform() # Right mouse click
actionChains.double_click(element).perform() # Double click of mouse
Keyboard operation
Selenium encapsulates the keyboard keys in the Keys class to simulate the input of some keys or key combinations on the keyboard. Use send_Keys+ http://Keys.XXX to implement key combinations on the keyboard such as: Ctrl+C, Ctrl+V.
from selenium.webdriver.common.keys import Keys # 导包
element.send_keys(Keys.BACK_SPACE) # Simulate pressing the backspace key
element.send_keys(Keys.CONTROL, 'a') # Simulate pressing Ctrl+A
element.send_keys(Keys.CONTROL, 'x') # Simulate pressing Ctrl+X
element.send_keys(Keys.CONTROL, 'v') # Simulate pressing Ctrl+V
Browser operations
Selenium also provides corresponding APIs for browser operations. Commonly used ones include maximizing the browser window, setting the browser window size, setting the browser position, controlling the browser forward and backward, and page refresh.
driver.maximize_window() # Maximize the browser window
driver.set_window_size(800, 800) # The browser window is 800 wide and 800 high
driver.set_window_rect(300, 0) # Browser location (300,0)
driver.back() # Browser backs up
driver.forward() # Browser forwards
driver.refresh() # Refresh the page
Other operations
Bookmark page
driver.get_screenshot_as_file('./baidu.png') # Page screenshot
driver.close() # Close the current tab page
driver.quit() # Close all tabs
Clicking a link on a page opens a new tab. At this time, Selenium cannot locate the elements of the new tab. This involves the concept of handle. The handle is the unique identifier of the tab page object. Each tab page has its own handle. You can use the handle to switch tab pages, thereby locating the element of the corresponding tab page.
handles = driver.window_handles # Get the handles of all tab pages
driver.switch_to.window(handles[1]) # Switch to the specified handle tab page
Startup parameters
Chrome Options is a class that configures chrome startup properties. Through this parameter, we can add startup parameters to Chrome.
Set the chrome binary file location (binary_location)
Add startup parameters (add_argument)
Add extension application (add_extension, add_encoded_extension)
Add experimental setting parameters (add_experimental_option)
Set the debugger address (debugger_address)
Commonly used behaviors of Chrome Options generally include the following:
Disable the loading of images and videos: Improve web page loading speed.
Add a proxy: Anti-crawling technology used for FQ to access certain pages, or to cope with IP access frequency restrictions.
Use mobile headers: Visit mobile sites. Generally, the anti-crawling technology of such sites is relatively weak.
Add extensions: Functions like using the browser normally.
Set encoding: deal with Chinese websites and prevent garbled characters.
Prevent JavaScript from executing
Before opening the browser in UI automation, you can add options to configure the browser. By setting different parameters, you can modify the browser's default behavior.
from selenium import webdriver
options = webdriver.ChromeOptions() # Instantiate a startup parameter object
options.add_argument('--headless') # Set the browser to not provide visual pages
options.add_argument('lang=zh_CN.UTF-8') # Set encoding
options.add_argument('--disable-infobars') # Disable strategy
options.add_argument('--no-sandbox') # Solve the error that DevToolsActivePort file does not exist
options.add_argument('window-size=1920x1080') # Specify browser resolution
options.add_argument('--disable-gpu') # Google documentation mentions the need to add this attribute to avoid bugs
options.add_argument('--incognito') # Invisible mode (incognito mode)
options.add_argument('--disable-javascript') # 禁用javascript
options.add_argument('--start-maximized') # Maximize operation (full screen window), if not set, an error will be reported when fetching elements.
options.add_argument('--disable-infobars') # Disable the prompt that the browser is being controlled by an automated program
options.add_argument('--hide-scrollbars') # Hide scroll bars to deal with some special pages
options.add_argument('blink-settings=imagesEnabled=false') # Do not load images, improve speed
options.add_argument('User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36
') #Set UA request header
driver = webdriver.Chrome(chrome_options=options)
Execute JS script
In some cases, the API provided by selenium cannot complete the corresponding operation or the operation is troublesome. In this case, you can use JS scripts to achieve it, such as executing a sliding scroll bar.
js ='window.scrollTo(0,100)' # JS script statement to be executed
driver.execute_script(js) #Execute JS script
Frame switching
Usually most website pages use frame nesting. At this time, even if the content of the frame nested page is displayed, we still cannot directly locate the elements in the frame, such as login windows like Youku.
If you want to operate the elements in the frame at this time, you need to switch to the frame first and then position.
driver.switch_to.frame('alibaba-login-box') # and switch to the specified frame
driver.switch_to.default_content() # Switch back to the default page
Hide fingerprint features
Most browsers launched using slenium will be monitored by the website through some fingerprint characteristics. If the crawler behavior is recognized, the operation of selenium will be rejected. How to avoid this situation, the key point is how to use these characteristics before the browser detects it. Hidden, in fact, predecessors have paved the way for us. The key to solving this problem is actually a stealth.min.js file. This file is for puppeteer. If you use it in Python, you need to execute this file separately. The file acquisition method requires the installation of node.js. The terminal executes npx extract-stealth-evasions to download the file, and add the following code before operating the browser.
with open('stealth.min.js') as f: js = f.read() driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {'source': js})Conclusion
This article only briefly introduces the installation and use of selenium+python, as well as some basic and commonly used API operations. There are also many advanced operations and usages that require more in-depth understanding and learning. In actual application, it may also need to be integrated with other frameworks and tools, which require deeper understanding and learning.
Python interface automated testing from zero foundation to mastery (2023 latest version)