I believe that friends who have played crawlers know selenium, an artifact tool for automated testing. Writing a Python automation script to free your hands is basically a routine operation. If the crawler can't crawl, you can use automated testing to get it together.
Although selenium has complete documentation, it also requires a certain learning cost, and there are still some thresholds for a pure novice.
Recently, Microsoft has open sourced a project called "playwright-python", which is simply a fortress! This project is a pure automation tool for the Python language, which can realize automation without even writing code.
You may find it a bit unbelievable, but it is so powerful. Let's take a look at this artifact together.
1. Introduction to Playwright
Playwright is a powerful Python library that can automatically execute mainstream browser automation operations such as Chromium, Firefox, and WebKit with only one API, and supports running in headless mode and headed mode at the same time.
The automation technology provided by Playwright is green, powerful, reliable and fast, supporting Linux, Mac and Windows operating systems.
2. Playwright use
Install
Playwright The installation is very simple, two steps.
# Install the playwright library pip install playwright
# Install the browser driver file (the installation process is a bit slow)
python -m playwright install
The above two pip operations are installed separately:
to install the Playwright dependent library, Python3.7+ is required
to install the driver files of Chromium, Firefox, WebKit and other browsers
### recording
There is no need to write a line of code to use Playwright, we only need to manually operate the browser, it will record our operations, and then automatically generate code scripts.
The following is the recorded command codegen, just one line.
# Type --help on the command line to see all options
python -m playwright codegen
Codegen usage can be viewed with --help, if it is simple to use, just add the url link directly after the command, if you have other needs, you can add options.
python -m playwright codegen --help
Usage: index codegen [options] [url]
open page and generate code for user actions
Options:
-o, --output <file name> saves the generated script to a file
--target <language> language to use, one of javascript, python, python-async, csharp (default: "python")
-h, --help display help for command
Examples:
$ codegen
$ codegen --target=python
$ -b webkit codegen https://example.com
Options meaning:
-o: save the recorded script to a file
–target: specifies the language for generating the script, there are two kinds of JS and Python, the default is Python
-b: specify the browser driver
For example, I want to search on baidu.com, use the chromium driver, and save the result as a python file of my.py.
python -m playwright codegen --target python -o 'my.py' -b chromium https://www.baidu.com
The browser will automatically open after the command line is entered, and then you can see that every move on the browser will be Automatically translated into code as shown below.
After the end, the browser is automatically closed, and the generated automation script is saved to a py file.
from playwright import sync_playwright
def run(playwright):
browser = playwright.chromium.launch(headless=False)
context = browser.newContext()
# Open new page
page = context.newPage()
page.goto("https://www.baidu.com/")
page.click("input[name=\"wd\"]")
page.fill("input[name=\"wd\"]", "jingdong")
page.click("text=\"JD.COM\"")
# Click //a[normalize-space(.)='JD.COM's official website is faster, better and more economical only for quality life']
with page.expect_navigation():
with page.expect_popup() as popup_info:
page.click("//a[normalize-space(.)='JD.COM official website is faster, better and more economical only for quality life']")
page1 = popup_info.value
# - --------------------
context.close()
browser.close()
with sync_playwright() as playwright:
run(playwright)
Synchronize
The sample code below: Open three browsers in turn, go to Baidu to search, take a screenshot and exit.
from playwright import sync_playwright
with sync_playwright() as p:
for browser_type in [p.chromium, p.firefox, p.webkit]:
browser = browser_type.launch()
page = browser.newPage()
page.goto('https://baidu.com/')
page.screenshot(path=f'example-{browser_type.name}.png')
browser.close()
asynchronous
Asynchronous operations can be combined with asyncio to perform three browser operations at the same time.
import asyncio
from playwright import async_playwright
async def main():
async with async_playwright() as p:
for browser_type in [p.chromium, p.firefox, p.webkit]:
browser = await browser_type.launch()
page = await browser.newPage()
await page.goto('http://baidu.com/')
await page.screenshot(path=f'example-{browser_type.name}.png')
await browser.close()
asyncio.get_event_loop().run_until_complete(main())
mobile terminal
What's more, playwright can also support mobile browser simulation. The following is a piece of code provided by the official document, simulating the Safari browser on the mobile phone iphone 11 pro at a given geographic location, first navigate to maps.google.com, then perform positioning and take a screenshot.
from playwright import sync_playwright
with sync_playwright() as p:
iphone_11 = p.devices['iPhone 11 Pro']
browser = p.webkit.launch(headless=False)
context = browser.newContext(
**iphone_11,
locale='en-US',
geolocation={ 'longitude': 12.492507, 'latitude': 41.889938 },
permissions=['geolocation']
)
page = context.newPage()
page.goto('https://maps.google.com')
page.click('text="Your location"')
page.screenshot(path='colosseum-iphone.png')
browser.close()
In addition, it can also be used with the pytest plug-in. If you are interested, you can try it yourself.
3. Summary
Compared with existing automated testing tools, playwright has many advantages, such as:
cross-browser, support for Chromium, Firefox, WebKit
cross-operating system, support for Linux, Mac, Windows,
can provide the function of recording and generating code, free hands, and
can be used on mobile terminals
. The disadvantage is that the ecology and documentation are not very complete, such as no API Chinese documentation, no good tutorials and examples for learning. But I believe that as more and more people know about it, the future will get better and better.