Getting started with Puppeteer basics, common applications, and writing Puppeteer scripts using Google plug-ins

Preface

Puppeteer has heard of it many times and seen some articles related to it. But I haven’t studied it much, so now let’s learn it briefly.

Introduction

Puppeteer is a Node library that provides a high-level API to control Chromium or Chrome through the DevTools protocol. Puppeteer runs in headless mode by default, but it can be run in "headed" mode by modifying the configuration file.

Function

• Generate page PDF.
• Crawl SPA (Single Page Application) and generate pre-rendered content (i.e. "SSR" (Server Side Rendering)).
• Automatically submit forms, perform UI testing, keyboard input, etc.
• Create an automated testing environment that is constantly updated. Execute tests directly in the latest version of Chrome using the latest JavaScript and browser features.
• Capture the timeline trace of the website to help analyze performance issues.
• Test browser extensions.

Official documentation: https://puppeteer.bootcss.com/

Recommended article: Puppeteer tutorial that even grandma can easily get started with

Prepare

Install

npm i puppeteer

The latest version will be downloaded during installation Chromiumto ensure that the API can be used.

Note: By default, the latest version will be downloaded. If your node version is too low, the following problem will occur. You must either upgrade the node version or download an old version Puppeteer. I updated the node version here.

In addition to the above problem, if you also encounter ERROR: Failed to set up Chrome r116.0.5845.96! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip downloadproblems like this, it is because when you install Puppeteer, it will download the latest version of Chromium (~170MB Mac, ~282MB Linux, ~280MB Win) to ensure that the API can be used .
This shows that the download Chromiumfailed. I finally solved this problem by installing a lower version.Puppeteer

npm install puppeteer@19.11.1

Execute the official example to verify whether the installation is successful

Let’s execute the screenshot example provided by the official:
example.js

const puppeteer = require('puppeteer');

(async () => {
    
    
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({
    
    path: 'example.png'});

  await browser.close();
})();

node example.js

The above picture appears. You have installed it successfully. Now let’s get to the point.

use

Basic usage

const puppeteer = require("puppeteer");

// 初始化函数
const init = async () => {
    
    
  // 启动浏览器
  const browser = await puppeteer.launch({
    
    
    headless: false, // 是否以无头模式运行，默认为 true，除非选择开启了开发者工具
    devtools: true, // 是否自动打开开发者工具，默认为 false
    args: ["--start-maximized"], // 启动时传递给浏览器的额外参数
  });
  // 创建一个标签页
  const page = await browser.newPage();
  // 跳转到相应的网站
  await page.goto("https://example.com");
  // 截图等其他操作
  await page.screenshot({
    
     path: "example.png" });
  // 关闭浏览器
  await browser.close();
};

init();

Headless mode and headless mode
Headless mode and headless mode are two different browser running modes in Puppeteer.

• Headless Mode: Headless mode means running the browser in the background with no visible user interface. It is very useful in scenarios that do not require a graphical interface, such as conducting automated testing, crawling web page data, etc. The advantage of headless mode is to save resources and improve performance because it does not require rendering and displaying the page.

• Headful Mode: Headful mode means running the browser in a regular graphical interface and you can see the actual browser page. This is very helpful for visual operations during debugging and development, such as viewing page layout, debugging JavaScript, etc. Headed mode generally consumes more system resources because it requires page rendering and display.

When running in headless mode, a graphical interface will appear. After the operation is completed,

other common configuration properties of puppeteer.launch will be automatically closed.

//  <boolean> 是否在导航期间忽略 HTTPS 错误. 默认是 false。
// 当设置为true时，Puppeteer在导航过程中会忽略与 HTTPS 相关的错误，例如证书错误或安全警告。这在某些情况下很有用，例如当你访问的网站的 SSL 证书有问题或者过期时，仍然可以继续进行导航和操作。
// 忽略 HTTPS 错误的情况下，你的连接可能是不安全的。因此，在生产环境中，建议不要使用这个选项
ignoreHTTPSErrors

// <?Object> 为每个页面设置一个默认视口大小。包括宽高、缩放的，默认为800 * 600
defaultViewport 

// <Array<string>> 传递给浏览器实例的其他参数。具体见官方文档
args 

Error handling

const {
    
    TimeoutError} = require('puppeteer/Errors');

try {
    
    
 // 等待获取dom元素
  await page.waitForSelector('.foo');
} catch (e) {
    
    
  if (e instanceof TimeoutError) {
    
    
    // 如果超时，做一些处理。
  }
}

page

Page provides methods for manipulating a tabpage or page extension background page. A Browser instance can have multiple Page instances.

In practical applications, almost all common applications are used around pages. If you want to play well Puppeteer, you need to master it wellPage

event

We can listen to events through page.onor and remove events throughpage.oncepage.removeListener

function logRequest(interceptedRequest) {
    
    
  console.log('A request was made:', interceptedRequest.url());
}
page.on('request', logRequest);
// 一段时间后...
page.removeListener('request', logRequest);

common events

• close: triggered when the page is closed
• domcontentloaded: The DOMContentLoaded event is triggered when the page's DOMContentLoaded(When the HTML document has been fully parsed and all deferred scripts have been downloaded and executed. It does not wait for other content such as images, subframes, and asynchronous scripts to complete loading.) event is triggered triggered when.
• error: triggered when the page crashes
• load: Triggered when the load event of the page is triggered
• request: triggered when the page sends a request. The parameter request object is read-only. If you need to intercept and change the request, refer topage.setRequestInterception
• requestfailed: Triggered when the request for the page fails. For example, a request times out
• requestfinished: Triggered when a request for the page is successfully completed.
• response: triggered when a request on the page receives the corresponding response

Common methods

• page.$(selector): This method is executed within the page document.querySelector. If no element matches the specified selector, the return value is null.
• page.$$(selector): This method is executed within the page document.querySelectorAll. If no element matches the specified selector, the return value is [].
• page.$eval(selector, pageFunction[, ...args]): This method is executed within the page document.querySelector, and then the matched element is passed to pageFunction as the first parameter. If the pageFunction returns a Promise, then this method will wait for the promise to complete and return its return value. const searchValue = await page.$eval('#search', el => el.value);Similar to finding this dom element and performing some operations on this dom element.
• page.$$eval(selector, pageFunction[, ...args]): This method is executed within the page Array.from(document.querySelectorAll(selector)), and then the matched element array is passed to pageFunction as the first parameter. If the pageFunction returns a Promise, then this method will wait for the promise to complete and return its return value.
• page.addScriptTag(options): Inject script into the page, which await page.addScriptTag({ path: 'path/to/your/script.js' });is basically the same as script injection in browser plug-ins. The page is modified through the injected script.
• page.addStyleTag(options): Inject styles into the page,await page.addStyleTag({ path: 'path/to/your/styles.css' });
• page.browser(): Get the browser instance to which the current page instance belongs.
• page.click(selector[, options]): Click the specified element. If there are multiple matching natives, click the first one.
• page.content():Return the complete html code
• page.focus(selector): Causes the specified element to gain focus. If there are multiple matching elements, focus will be given to the first element.
• page.goBack([options]): Navigate to the previous page in the page history
• page.goForward([options]): Navigate to the next page in the page history
• page.goto(url[, options]): Jump to the webpage of the specified url
• page.isClosed(): Determine whether the page is closed
• page.evaluate(pageFunction[, ...args]): Execute the method in the context of the page instance and return the result of the method execution
• page.keyboard: Class used to interact with the keyboard
• page.mouse: Class for interacting with the mouse
• page.pdf([options]): Generate a pdf file of the current page. By default, a binary encoding will be returned. Please see the documentation for specific configuration.
• page.screenshot([options]): Generate page screenshots. By default, a binary encoding will be returned. It can also be set to base64. See the documentation for specific configuration.
• page.select(selector, ...values): Used to select the drop-down list in the page. It accepts a selector parameter and one or more value parameters. This method finds the drop-down list element on the page that matches the selector and selects the option in the value parameter. For example: `await page.select('select#myDropdown', 'option1', 'option2');
• page.setExtraHTTPHeaders(headers): Each request initiated by the current page will carry these request headers
• page.setRequestInterception(value): Whether to enable the request interceptor. If the request interceptor is enabled, request.abortthe r equest.continueand request.respondmethod will be activated. This provides the ability to modify the network requests made by the page. Once request interception is enabled, every request will be stopped unless it continues, responds, or aborts
• page.title(): Return to page title
• page.type(selector, text[, options]): used to input content, for example

page.type('#mytextarea', 'Hello'); // 立即输入
page.type('#mytextarea', 'World', {
    
    delay: 100}); // 输入变慢，像一个用户

• page.waitForNavigation([options]): Used to wait for page navigation to be completed, and subsequent operations can only be continued after the new page is loaded.
• page.waitForSelector(selector[, options]): Wait for the element matching the specified selector to appear on the page. If there is already a matching element when this method is called, this method returns immediately. If the specified selector does not appear after the timeout, this method will report an error.

Keyboard

KeyboardProvides an interface to manage virtual keyboards. The high-level interface is , which receives raw characters, and then generates corresponding , and events keyboard.typeon your pagekeydownkeypress/inputkeyup

Common methods

page.type : receive characters

page.type('#mytextarea', 'Hello'); // 立即输入
page.type('#mytextarea', 'World', {
    
    delay: 100}); // 输入变慢，像一个用户

keyboard.down(key[, options]) : Press the specified key

await page.keyboard.down('Shift');

keyboard.up(key) : Release the pressed key

await page.keyboard.up('Shift');

A pressed keyboard.downkey must keyboard.upbe released, otherwise the key will remain pressed.

keyboard.press(key[, options]) : Press and release the key, which is equivalent to the combination of keyboard.downand keyboard.up. You can specify the interval between pressing and releasing.

await page.keyboard.press('KeyA');

Mouse

Mouse is used to simulate a mouse

Common methods

mouse.click(x, y, [options]) : Use the mouse to click the specified location. The default left button is click once.

mouse.down([options]) : Mouse down, the default is the left button, the default is to press once

mouse.up([options]) : When the mouse is raised, the default is the left button, and the default is once

Example

Let’s make a slightly more complicated example:

Step
1. Open the main interface of the novice tutorial.

2. Click the Learn HTML5 tab to enter the corresponding page.

3. Click the HTML image to enter the corresponding page.

4. Download the image to the local computer.

code

const puppeteer = require("puppeteer");

// 初始化函数
const init = async () => {
    
    
  // 启动浏览器
  const browser = await puppeteer.launch({
    
    
    headless: false,
  });
  // 创建一个标签页
  const page = await browser.newPage();
  // 跳转到相应的网站
  await page.goto("https://www.runoob.com/");
  // 元素是动态生成的，需要等待元素完全加载
  await page.waitForSelector(".item-top.item-1");
  // 获取到 学习html5对应的标签
  const html5 = await page.$$(".item-top.item-1");
  // 返回值是CDPElementHandle { handle: CDPJSHandle {} }，一个CDP元素句柄，指向浏览器中实际dom元素的引用
  // 需要使用 evaluate方法来获取对应的dom中的属性，比如元素的innerText属性是在浏览器环境中计算得出的，而不是直接在 Puppeteer 中可见的
  console.log("html5[1]：", html5[1]);
  // 点击该标签
  await html5[1].click();
  // 等待页面渲染出现a标签的元素
  await page.waitForSelector("a");
  // 获取HTML 图像并点击
  const htmlImg = await page.$('a[title="HTML 图像"]');
  console.log("htmlImg:", htmlImg);
  htmlImg.click();
  await page.waitForSelector(".example_result.notranslate img");
  // 获取到图片标签
  const img = await page.$(".example_result.notranslate img");
  const imageSrc = await img.evaluate((el) => el.src);
  console.log("图片地址：", imageSrc);
  // 图片截图
  await img.screenshot({
    
     path: "example.png" });
  // 关闭浏览器
  await browser.close();
};

init();

renderings

important point:

• In order to facilitate development, it is best to set it to headless: falseand do not close the browser. This will make it easier to check the specific effects and see if there are any problems. After confirming that there is no problem with the code headless: true, set it to and close the browser at the end.
• page.$(".example_result.notranslate img")What is obtained is one CDPElementHandle { handle: CDPJSHandle {} }. This is a CDP element handle, a reference to the actual DOM element in the browser. You can directly call clickother methods, but you cannot directly obtain innerTextthe attributes of elements, for example. The explanation found is that innerTextthe attributes of the element are calculated in the browser environment, rather than directly visible in Puppeteer. evaluateThe context object can be manipulated through methods.
• page.waitForSelector("a"), waiting for a certain element to be rendered. Because the code operates directly, the next step may begin before the page is rendered, causing the code to report an error.
• img.evaluate((el) => el.src)Execution context object

Writing Puppeteer scripts using Google plug-ins

What, you can't write scripts? What, you find it troublesome to write scripts? What, can the plug-in write scripts?

That's right, it doesn't matter if you don't know how to write scripts. You can use Google plug-ins to let the plug-ins write scripts for you. Here we need to useHeadless Recorde

Introduction
Headless Recorder is a Chrome browser plug-in that helps you record and play back your actions in the browser. Its main function is to record your actions in the browser and generate replayable code so that you can re-perform these actions at any time.

Headless Recorder is a very practical tool, especially suitable for scenarios such as automated testing, web page presentations, and teaching. Use it to quickly generate replayable code without having to manually write and debug scripts, saving a lot of development and testing time.

The working principle of Headless Recorder is to use the Headless mode of Chrome browser, which can run the browser in the background and simulate user operations, including clicks, inputs, scrolling, etc. Through the recording function of the plug-in, you can perform various operations in the browser, and the plug-in will record these operations and generate corresponding codes.

Once the recording is complete, you can export the generated code to a variety of formats, such as JavaScript, Puppeteer, Playwright, etc., for running and replaying in other environments. This way you can replicate what you did in the browser in different environments without having to repeat the steps manually.

In short, Headless Recorder is a powerful tool that can help you quickly record and play back operations in the browser, improving development efficiency and test quality. Whether it is automated testing or web page presentation, it is a very useful auxiliary tool.

Official address
https://github.com/checkly/headless-recorder

Basic Operation
After turning on Headless Recordethe plug-in, Headless Recordeyour operations in the browser will be recorded. When the recording ends. Will convert your operations in the browser into scripts, for example:

The following is the generated code (needs to be slightly modified)

const init = async () => {
    
    
  const puppeteer = require("puppeteer");
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  const navigationPromise = page.waitForNavigation();

  await page.goto("https://www.runoob.com/");

  await page.setViewport({
    
     width: 1920, height: 969 });

  await page.waitForSelector(
    ".row > .col > .cate1 > .item-top:nth-child(3) > h4"
  );
  await page.click(".row > .col > .cate1 > .item-top:nth-child(3) > h4");

  await navigationPromise;

  await page.waitForSelector(
    ".runoob-col-md2 > .left-column > .sidebar-box > #leftcolumn > a:nth-child(13)"
  );
  await page.click(
    ".runoob-col-md2 > .left-column > .sidebar-box > #leftcolumn > a:nth-child(13)"
  );

  await navigationPromise;

  await page.waitForSelector(
    ".article-body > #content > .example > .example_result > img"
  );
  await page.click(
    ".article-body > #content > .example > .example_result > img"
  );

  const element1 = await page.$(
    ".article-body > #content > .example > .example_result > img"
  );
  await element1.screenshot({
    
     path: "screenshot_1.png" });

  await browser.close();
};

init();

Comparing it, I don’t know how many times better than the script I wrote.

Download
Download The plug-in seems to have been removed from the Google store. You can search directly on Baidu. Of course, you can also download the plug-in I downloaded and the plug-in has been uploaded to Baidu Cloud Disk.

Link: https://pan.baidu.com/s/1k8fvl-CemYOMUedsZmjBnQ
Extraction code: 1234