Xiaobai learns crawler-using Selenium+Chrome on CentOS without GUI

The crawler proxy IP is provided by the Sesame HTTP service provider


. Are you tossed by various click events of JavaScript in your daily collection? Finally found a Selenium+Chrome can solve the problem!

But another ▄█▀█● fact is in front of you, the server is so no GUI...


well! We have to overcome difficulties! We must not be overwhelmed by this little difficulty

... But the fact that is in front of us is... He can't pretend to be anything! Pit father!

Then I'm here to save you from fire and water!


The server is as follows:
[root@spider01 ~]# hostnamectl
   Static hostname: spider01
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 1c4029c4e7fd42498e25bb75101f85b6
           Boot ID: f5a67454b94b454fae3d75ef1ccab69f
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-514.6.2.el7.x86_64
      Architecture: x86-64


Install Chromeium:
## install yum source
[root@spider01 ~]# sudo yum install -y epel-release
## Install Chrome
[root@spider01 ~]# yum install -y chromium


Go to this place: https://sites.google.com/a/chromium.org/chromedriver/downloads Download the ChromeDriver driver and put it in the /usr/bin/ directory:

The result is as follows:
[root@spider01 ~]# ll /usr/bin/ | grep chrom
-rwxrwxrwx. 1 root 7500280 11 月 29 17:32 chromedriver
lrwxrwxrwx. 1 root root        47 11月 30 09:35 chromium-browser -> /usr/lib64/chromium-browser/chromium-browser.sh


Install XVFB:
[root@spider01 ~]# yum install Xvfb -y
[root@spider01 ~]# yum install xorg-x11-fonts* -y


Create a new file called xvfb-chromium in /usr/bin/ and write the following:
[root@spider01 ~]# cat /usr/bin/xvfb-chromium
#!/bin/bash

_kill_procs() {
  kill -TERM $chromium
  wait $chromium
  kill -TERM $xvfb
}

# Setup a trap to catch SIGTERM and relay it to child processes
trap _kill_procs SIGTERM

XVFB_WHD = $ {XVFB_WHD: -1280x720x16}

# Start Xvfb
Xvfb :99 -ac -screen 0 $XVFB_WHD -nolisten tcp &
xvfb=$!

export DISPLAY=:99

chromium --no-sandbox --disable-gpu$@ &
chromium=$!

wait $chromium
wait $xvfb


change soft link

## Change the soft connection launched by Chrome
[root@spider01 ~]# ln -s /usr/lib64/chromium-browser/chromium-browser.sh /usr/bin/chromium


[root@spider01 ~]# rm -rf /usr/bin/chromium-browser

[root@spider01 ~]# ln -s /usr/bin/xvfb-chromium /usr/bin/chromium-browser

[root@spider01 ~]# ln -s /usr/bin/xvfb-chromium /usr/bin/google-chrome

[root@spider01 ~]# ll /usr/bin/ | grep chrom*
-rwxrwxrwx. 1 root 7500280 11 月 29 17:32 chromedriver
lrwxrwxrwx. 1 root root        47 11月 30 09:47 chromium -> /usr/lib64/chromium-browser/chromium-browser.sh
lrwxrwxrwx. 1 root root        22 11月 30 09:48 chromium-browser -> /usr/bin/xvfb-chromium
-rwxr-xr-x. 1 root root     73848 12月  7 2016 chronyc
lrwxrwxrwx. 1 root root        22 11月 30 09:48 google-chrome -> /usr/bin/xvfb-chromium
-rwxrwxrwx. 1 root root       387 11月 29 18:16 xvfb-chromium
 


Let's see if it can be used:
>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get("http://www.baidu.com")
>>> driver.find_element_by_xpath("./*//input[@id='kw']").send_keys("哎哟卧槽")
>>> driver.find_element_by_xpath("./*//input[@id='su']").click()
>>> driver.page_source


No problem! ! ! !

Well, the deployment is complete! Of course, Docker is such a hot thief for lazy people! Come and see the Docker version here!

docker pull thsheep/chromium-xvfb-py3:master


After Python3.6.3 is integrated with

Chrome , you need to use Dockerfile to repackage and install the Python packages you need.

Incidentally! ! ! ! This thing can be used by friends who are engaged in web testing! ! ! ! ! ! ! !

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=326455648&siteId=291194637