java Selenium implements simple web page operations

Official Documentation : Getting Started | Selenium 

Selenium is a tool for web application testing. Selenium tests run directly in the browser, just like real users.

Therefore, using this front-end testing tool, you can automatically do many things, such as automatically crawling web content, commonly known as web crawlers. In fact, the search engine itself is a web crawler technology.

Due to the complexity of network information, it is very difficult for users to find the information they need, and the emergence of search engines is to help users organize network information .

1. Simple small case

Logic: Use the Seleium tool to automatically open the Chrome browser, visit the Baidu webpage, enter the four words "Crayon Shin-Chan" in the webpage, and then click "Baidu Click" to search the results.

<!-- 安装依赖 --> 
 <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>4.11.0</version>
    </dependency>

  public static void main(String[] args) {
        //Selenium所做的一切, 就是发送给浏览器命令, 用以执行某些操作
        //或为信息发送请求. 您将使用Selenium执行的大部分操作, 都是以下基本命令的组合
        System.out.println("启动");

        //1. 使用驱动实例开启会话
        WebDriver driver = new ChromeDriver();

        //2. 在浏览器上执行操作 ，导航到一个网址
        driver.get("https://www.baidu.com/");

        //3. 请求浏览器的信息
        String title = driver.getTitle();
        System.out.println(title);

        driver.manage().timeouts().implicitlyWait(Duration.ofMillis(500));

        //4. 打开百度网页，寻找到input输入框，输入内容，之后点击 ”百度一下“ 按钮
        WebElement textBox = driver.findElement(By.id("kw"));
        textBox.sendKeys("蜡笔小新");
        WebElement submitButton = driver.findElement(By.id("su"));
        submitButton.click();
    }

2. Thinking about technical direction

If you think carefully about this automated testing technology, it can actually help people do many things.

 1. For example, if you want to watch an anime video, Tencent is a VIP, and you don’t want to spend money, but there are still many video websites on the Internet that provide free viewing, but some video websites still cannot play it even though they provide the name of the anime .

=>At this time, you can use automated tools to automatically find the website of the animation, click to play one by one, until one or more can be played, then we select one or more and return the URL to the user, and the user clicks It's ready to play.

3. Crawler framework WebMagic (to be learned later)

WebMagic