Related use of Selenium technology on Tencent Cloud Server of CentOS6.8 system (in Linux environment)

1. Explanation

If you want to use Selenium directly in the CentOS6.8 environment, generally speaking, there is python on the server, which is quite convenient.

2. Linux related commands during operation

1. Download Google Chrome

yum install https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

2. Check the version of Google Chrome

google-chrome --version

3. Download the compressed package of the corresponding version of Google Drive (or upload locally)

wget https://chromedriver.storage.googleapis.com/113.0.5672.63/chromedriver_linux64.zip

4. Unzip the downloaded file

unzip chromedriver_linux64.zip

5. Mobile download files

mv chromedriver /usr/bin/

6. Give the file execution permission

chmod +x /usr/bin/chromedriver

7. Update pip3 to the highest version

pip3 install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple/

8. Download the Selenium third-party library

Generally speaking, there are two pythons on the server, one is Python3.x, and the other is Python2.x. In most cases, the third-party library Selenium is downloaded using Python3.x.

I generally don’t like individual downloads. , otherwise some dependencies may be missing, so I usually put all the required third-party libraries in a file, and then read and download.

pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple/

9. Formal testing

nohup python3 -u test.py > log.log 2>&1 &

10. Finally, you can test it yourself (Selenium version must be >=4.3.0)

#!/usr/bin/python3
#coding:utf-8
# 浏览器
from selenium import webdriver
# 规避检测
from selenium.webdriver import ChromeOptions
# 无头浏览器
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By

ch_options = webdriver.ChromeOptions()
#为Chrome配置无头模式
ch_options.add_argument("--headless")  
ch_options.add_argument('--no-sandbox')
ch_options.add_argument('--disable-gpu')
ch_options.add_argument('--disable-dev-shm-usage')
# 在启动浏览器时加入配置，这个驱动路径是容器里对应的路径，不是宿主机的路径
dr = webdriver.Chrome(service=Service("/ubuntu/python/chromedriver"),options=ch_options)
#这是测试网站
url = "https://www.baidu.com"
dr.get(url)
#打印源码
print(dr.page_source)

3. Update python3.6 to a higher version under CentOS6.8

1. Explanation

I may feel that some third-party libraries cannot be downloaded with python3.6, which is very troublesome, and I will want to upgrade the original python3 or something. I will also add it as a blogger. I will use python3.7 as an example.

2. Download the compressed package online

wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz

3. Decompress the compressed package

tar -xvf Python-3.7.0.tgz

4. Download related dependent libraries

yum install zlib zlib-devel libffi-devel -y

5. Jump to the corresponding folder

cd Python-3.7.0

6. Install in the /usr/local directory

./configure --prefix=/usr/local/python3

7. Compile and install the program to the specified directory

make && make altinstall

8. Delete some temporary files (delete the execution files and all intermediate target files generated by the source code (C\C++ code))

make clean

9. Delete the original python3 and pip3

rm -rf /usr/bin/python3
rm -rf /usr/bin/pip3

10. Create a soft link

ln -s /usr/local/python3/bin/python3.7 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3.7 /usr/bin/pip3

11. Check the python version

python3 -V

12. View pip version

pip3 --V

13. Basically succeeded

14. If there is a problem with the pip command, the solution

The error ModuleNotFoundError: No module named 'pip._internal.cli' keeps appearing
. Perform these two steps in order to solve the problem

python -m ensurepip

python -m pip install --upgrade pip