Selenium+2Captcha automation + verification code recognition practice

This article discusses in depth the use of the Selenium library for web page automation, and combined with the 2Captcha service to crack ReCAPTCHA verification codes. The content covers the basic knowledge of Selenium, the classification of verification codes, the use of 2Captcha services, and detailed explanations through examples. Finally, it summarizes and optimizes the practice, providing readers with a complete practice roadmap for cracking verification codes.

I. Introduction

Automated testing and web scraping are common tasks in modern web development. In these two fields, Selenium is a widely used tool that can simulate browser operation and operate and analyze Web pages. In this article, we'll start by covering the basics of Selenium, and then explore how it can be used to tackle another common web problem: captchas.

1.1 Introduction to Selenium and its application scenarios

Selenium is an automated testing tool mainly used for functional and performance testing of web applications. It can run directly on the browser and supports multiple operating systems, browsers and programming languages. In addition to testing, Selenium is also often used in web crawlers to simulate and automate browser operations.

from selenium import webdriver

driver = webdriver.Firefox()
driver.get('http://www.python.org')
assert "Python" in driver.title
driver.quit()

The Python code above shows a simple Selenium script that launches a Firefox browser, then visits the official Python website and checks if the page title contains the word "Python". Finally, close the browser.

1.2 Purpose and type of verification code

Captcha, the full name is "Completely Automated Public Turing test to tell Computers and Humans Apart", which is a public fully automated Turing test for distinguishing whether a user is a machine or a human. The primary purpose is to prevent malware and automated scripts from harassing, abusing the service, or engaging in other undesirable behavior.

Common types of verification codes include text verification codes, graphic verification codes, sliding verification codes, touch verification codes, etc. Recently, with the development of machine learning technology, such as Google's ReCAPTCHA system, it provides verification code services based on user behavior analysis, which greatly increases the difficulty of cracking.

In the next article, we will focus on how to use Selenium to handle these captchas, especially graphic captchas and ReCAPTCHA captchas.

Two, Selenium knowledge

Selenium is an automated testing tool mainly used for functional testing of web applications. It can simulate real user behavior, such as clicking a button, entering text, selecting a drop-down menu, and so on. Therefore, Selenium is often used in web crawlers to process pages rendered by JavaScript or to simulate user behavior.

2.1 Selenium installation and configuration

First, we need to install Selenium on our machine. Following are the commands to install Selenium in Python environment:

pip install selenium

Then, we also need to download the corresponding browser driver, for example, the driver of Chrome is chromedriver. The download address of the driver can usually be found on the official website of the browser.

2.2 Introduction to WebDriver

WebDriver is the core part of Selenium. It is an interface that defines a series of methods to operate the browser. Each browser has its own implementation of WebDriver, such as ChromeDriver, FirefoxDriver and so on.

Here is a simple example showing how to use WebDriver to open a web page:

from selenium import webdriver

driver = webdriver.Firefox()
driver.get('http://www.google.com/')

The above code first creates an instance of FirefoxDriver, and then calls its getmethod to open the Google homepage.

2.3 Positioning of page elements

Selenium provides a variety of methods to locate page elements, such as by id, name, class name, tag name, link text, partial link text, xpath, css selector, etc.

Here are some examples of positioned elements:

element = driver.find_element_by_id('id_of_element')  # 通过id定位
element = driver.find_element_by_name('name_of_element')  # 通过name定位
element = driver.find_element_by_class_name('class_of_element')  # 通过class name定位
element = driver.find_element_by_tag_name('tag_of_element')  # 通过tag name定位
element = driver.find_element_by_xpath('//div[@class="my_class"]')  # 通过xpath定位
element = driver.find_element_by_css_selector('div.my_class')  # 通过css selector定位

2.4 Operating page elements

After getting the page element, we can operate on it. Common operations include entering text, clicking buttons, getting element text, and so on.

Here are some examples of action elements:

element.send_keys('some text')  # 输入文字
element.click()  # 点击元素
text = element.text  # 获取元素的文本

2.5 Waiting for the page to load

In web crawling, we often encounter situations where we need to wait for a page to load. Selenium provides two ways to wait: explicit wait and implicit wait.

Here is an example of an explicit wait:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, 'id_of_element'))
)

The above code will wait up to 10 seconds until an element with id 'id_of_element' appears on the page.

2.6 Advanced Operations

Selenium also supports some advanced operations, such as executing JavaScript code, manipulating cookies, switching iframes, handling popups, and more.

The following is an example of executing JavaScript code:

driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')

The above code executes a JavaScript script that scrolls the page to the bottom. This is very useful when dealing with some web pages that need to be scrolled.

3. Verification code cracking: Selenium uses 2Captcha service to implement ReCAPTCHA verification code cracking

Captchas are a ubiquitous security mechanism in the online world to distinguish human users from machines. Here, we will combine Selenium and an automatic verification code solution (TwoCaptcha) to show how to crack a common verification code - ReCAPTCHA.

1.1 2Captcha service introduction cn.2captcha.com

2Captcha is a captcha recognition service based on human labor. It provides an API interface that allows developers to send unrecognized verification codes to 2Captcha services. 2Captcha's workers then manually identify and return the results. This service has a high accuracy rate for complex verification codes such as image verification codes, reCAPTCHA, and FunCaptcha. The main advantage of 2Captcha is its excellent accuracy and flexible API, which allows developers to easily integrate and use it in different environments.

cn.2captcha.com

Support verification code type

Support Alipay payment

3.2 Introduction to ReCAPTCHA

ReCAPTCHA is a verification code service launched by Google. Its main feature is to provide a "I am not a robot" check box for users to click. After the user clicks, ReCAPTCHA will evaluate the user's behavior to determine whether the user is human. If the user is judged to be human, then the verification is passed; if it cannot be determined, then an additional challenge is given, such as selecting an image that contains something.

3.3 Using Selenium to simulate user behavior

We can use Selenium to simulate a user clicking the "I'm not a robot" checkbox. In Selenium, we can use clickmethods to simulate clicks, for example:

checkbox = driver.find_element_by_id('recaptcha_check')
checkbox.click()

However, this may not be enough, since ReCAPTCHA analyzes user behavior. For example, if the click is too fast or mechanical, ReCAPTCHA may judge it as robotic behavior.

3.4 Use 2Captcha to automatically solve the verification code

If ReCAPTCHA presents additional challenges, we need to use other tools to solve it. Here we choose to use TwoCaptcha, which is a service that can automatically solve various captchas.

In TwoCaptcha, we need to provide the URL of the website and the sitekey of the website, and then it will return an answer to solve the captcha, we can fill this answer back to the webpage to complete the verification.

Here is a sample code to solve captcha using TwoCaptcha:

solver = TwoCaptcha(API_KEY)
result = solver.recaptcha(sitekey=sitekey, url=url)

3.5 Combining Selenium and 2Captcha to crack the verification code

With Selenium and TwoCaptcha, we can combine to form a complete solution. Specific steps are as follows:

1. Open the web page using Selenium.
2. Find and click the "I'm not a robot" checkbox.
3. If additional challenges arise, solve them using TwoCaptcha and fill the answers back into the web page.

The following code implements the above steps:

from bs4 import BeautifulSoup
from twocaptcha import TwoCaptcha
from selenium import webdriver

API_KEY = 'your_twocaptcha_api_key'
solver = TwoCaptcha(API_KEY)

# 创建一个WebDriver实例
driver = webdriver.Firefox()

# 使用Selenium打开网页
driver.get('https://www.example.com/')

# 找到并点击"我不是机器人"的复选框
checkbox = driver.find_element_by_id('recaptcha_check')
checkbox.click()

# 如果出现额外的挑战，使用TwoCaptcha解决
sitekey = 'sitekey_from_webpage'
url = driver.current_url
result = solver.recaptcha(sitekey=sitekey, url=url)

# 将答案填回网页
driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML="{}"'.format(result['code']))

3.6 Use Selenium to automatically fill in the verification code

Selenium is combined with 2Captcha to automatically fill in the parsed verification code. We can use Selenium to locate the verification code input box and fill in the parsing results.

# 用Selenium定位验证码输入框
input_box = driver.find_element_by_id('captcha-input-box-id')

# 填入解析出的验证码
input_box.send_keys(captcha_solution)

3.7 Processing image verification code

For the image verification code, we can use Selenium to get the image element and save it as a local file. Then, we can upload the local file to 2Captcha for parsing.

# 定位到图片元素
image_element = driver.find_element_by_id('captcha-image-id')

# 将图片保存为本地文件
image_element.screenshot('captcha.png')

# 上传到TwoCaptcha进行解析
solver = TwoCaptcha(API_KEY)
result = solver.normal('captcha.png')

# 填入解析出的验证码
input_box = driver.find_element_by_id('captcha-input-box-id')
input_box.send_keys(result)

3.7 Handling other types of verification codes

In addition to the verification code types mentioned above, there are other types of verification codes, such as text CAPTCHA, reCaptcha V2, reCaptcha V3, HCaptcha, Funcaptcha, and 2Captcha services can all be solved very well.

Four. Summary

Through this article, we have learned how to use the Selenium library to simulate browser operations, and combine the TwoCaptcha service to realize the automatic cracking of ReCAPTCHA verification codes. Now, let's summarize our practices and offer some thoughts on possible future optimizations.

Through Selenium, we can perform various complex controls on the browser, from opening web pages, filling out forms, to simulating clicks, and so on. However, when we encounter challenges like captchas that require human participation, we need to find additional solutions.

2Captcha offers a great solution. It can solve various types of captchas, including ReCAPTCHA which we mentioned in this article. Moreover, the API provided by 2Captcha allows us to easily integrate it into our Selenium scripts.

If it is helpful, please pay more attention to
the personal WeChat public account: [TechLead] Share the full-dimensional knowledge of AI and cloud service research and development, and talk about my unique insight into technology as a TechLead.
TeahLead KrisChang, 10+ years of experience in the Internet and artificial intelligence industry, 10+ years of experience in technology and business team management, Tongji Software Engineering Bachelor, Fudan Engineering Management Master, Alibaba Cloud Certified Cloud Service Senior Architect, AI Product Business with Hundreds of Millions of Revenue principal.