[SOLVED]Ubuntu Server failed to run selenium webdriver using jenkins

I have encountered this problem when using jenkins account to run selenium webdriver on multiple ubuntu servers.

Including ubuntu 16.04, ubuntu 18.04 and ubuntu 20.04 server systems, this is a common problem.

The error message on failure is as follows:

selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Here is the detailed log when I tried to reproduce the problem with a simple script:

(venv) jenkins@guyongqiangx:/scratch/test/sampleKeywordTest$ python3 Others/testSample.py 
Traceback (most recent call last):
  File "Others/testSample.py", line 28, in <module>
    driver = webdriver.Chrome(service=service, options=chrome_options)
  File "/scratch/test/venv/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
    super().__init__(DesiredCapabilities.CHROME['browserName'], "goog",
  File "/scratch/test/venv/lib/python3.8/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
    super().__init__(
  File "/scratch/test/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 277, in __init__
    self.start_session(capabilities, browser_profile)
  File "/scratch/test/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 370, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/scratch/test/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 435, in execute
    self.error_handler.check_response(response)
  File "/scratch/test/venv/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x5565e64e1463 <unknown>
#1 0x5565e62a58d8 <unknown>
#2 0x5565e62cdb6a <unknown>
#3 0x5565e62c8c05 <unknown>
#4 0x5565e630c802 <unknown>
#5 0x5565e630c2af <unknown>
#6 0x5565e6304443 <unknown>
#7 0x5565e62d53c5 <unknown>
#8 0x5565e62d6531 <unknown>
#9 0x5565e6533dce <unknown>
#10 0x5565e6537192 <unknown>
#11 0x5565e651893e <unknown>
#12 0x5565e6538103 <unknown>
#13 0x5565e650bd85 <unknown>
#14 0x5565e65590a8 <unknown>
#15 0x5565e6559239 <unknown>
#16 0x5565e6574492 <unknown>
#17 0x7f78ae631609 start_thread

(venv) jenkins@guyongqiangx:/scratch/test/sampleKeywordTest$ 

But if I test with another account it behaves fine:

(venv) rocky@guyongqiangx:/scratch/test/sampleKeywordTest$ python3 Others/testSample.py 
URL: http://bp3.newbiestart.net/, Title: Home - BP3
Checkbox is not selected, click it
Checkbox is selected
URL: http://bp3.newbiestart.net/, Title: Home - BP3
(venv) rocky@guyongqiangx:/scratch/test/sampleKeywordTest$ 

The only difference between these two tests is that the accounts are different, one is a common account rocky, and the other is a jenkins account.

This problem has been going on for a long time. I have tried all kinds of environmental factors that I can think of, and I have used Baidu and Google to search many webpages according to the previous error message, but it has not been resolved.

Until one day, I suddenly remembered that I should narrow down the scope of the investigation and use jenkins to directly run the Chrome headless browser (headless) to have a look.

Sure enough, running the Chrome headless browser on jenkins also failed, and surprisingly, this run gave a clear error message.

The following are the logs of running the headless browser of Chrome in the command letter using the normal account and the jenkins account respectively:

• Use the rocky account to run google-chrome headless normally

(venv) rocky@guyongqiangx:/scratch/test/sampleKeywordTest$ google-chrome --headless --disable-gpu --dump-dom https://www.baidu.com
[1220/215744.646207:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.
[1220/215744.672649:WARNING:sandbox_linux.cc(380)] InitializeSandbox() called with multiple threads in process gpu-process.
<!DOCTYPE html>
<html><head>
...
... 这中间是正常的网页内容
...
</script><script defer="" src="//hectorstatic.baidu.com/96c9c06653ba892e.js"></script></body></html>
(venv) rocky@guyongqiangx:/scratch/test/sampleKeywordTest$ 

• Run google-chrome headless exception using jenkins account

(venv) jenkins@guyongqiangx:/scratch/test/sampleKeywordTest$ google-chrome --headless --disable-gpu --dump-dom https://www.baidu.com
[1220/215611.305649:ERROR:filesystem_posix.cc(63)] mkdir /tmp/Crashpad/new: Permission denied (13)
[1220/215611.306044:ERROR:socket.cc(120)] recvmsg: Connection reset by peer (104)
Trace/breakpoint trap (core dumped)
(venv) jenkins@guyongqiangx:/scratch/test/sampleKeywordTest$ 

Here it is prompted that /tmp/Crashpad/newthe directory because of "Permission denied".

Go to /tmpthe directory and see that the owner of the "Crashpad" directory is the rocky account, and jenkins does not have any permissions.

(venv) rocky@guyongqiangx:/scratch/test/sampleKeywordTest$ ls -lh /tmp/
total 3.0M
drwxrwxr-x 3 andrew users   4.0K Oct 31 17:52 8863542
drwxrwxr-x 3 andrew users   4.0K Oct 31 18:04 8882707
-rw-r----- 1 andrew users    49K Dec  6 17:13 adb.36670.log
drwx------ 6 rocky  users   4.0K Dec 20 20:53 Crashpad

Compare the environment where jenkins can be executed normally on the virtual machine:

(venv) ygu@bp3tester:/local/bp3Test/sampleKeywordTest$ ls -lh /tmp/
total 3.3M
drwx------ 6 jenkins jenkins 4.0K Dec  3 00:06 Crashpad
drwxr-xr-x 2 jenkins jenkins 4.0K Dec 20 21:45 hsperfdata_jenkins

Here you can see that the normal execution environment on the virtual machine /tmp/Crashpadbelongs to the jenkins account.

Once the cause is found, the solution is easy.

There are several ways:

1. /tmp/CrashpadChange the owner of the directory to jenkins

   $ sudo chown -R jenkins:jenkins /tmp/Crashpad 
   

2. Do not change the owner of /tmp/Crashpadthe directory , but set its permissions to 777, so that the jenkins account can also operate

   $ sudo chmod -R 777 /tmp/Crashpad
   

Problem solved perfectly.

If you fail to run selenium webdriver on ubuntu server, it is recommended to check the following:

1. Is the Chrome browser application installed on the server?

2. Did you download and specify the webdriver corresponding to Chrome when running selenium webdriver?

3. Use jenkins to run selenium in an environment without a graphical interface, you need to set it to headless mode (headless)

   from selenium.webdriver.chrome.service import Service as ChromeService
   from selenium.webdriver.chrome.options import Options as ChromeOptions
   from webdriver_manager.chrome import ChromeDriverManager
   
   chrome_options = ChromeOptions()
   
   chrome_options.add_argument('--no-sandbox')
   chrome_options.add_argument('--headless')
   chrome_options.add_argument('--disable-extensions')
   chrome_options.add_argument('--disable-gpu')
   chrome_options.add_argument('--disable-dev-shm-usage')
   
   # 这里使用 ChromeDriverManager 自动安装 webdriver
   chrome_service = ChromeService(executable_path=ChromeDriverManager().install())
   driver = webdriver.Chrome(service=chrome_service, options=chrome_options)
   ...
   

4. Try running Chrome in headless mode directly from the command line

   $ google-chrome --headless --disable-gpu --dump-dom https://www.baidu.com