How to scroll a web page down until specific button to click?

DaniB :

I am working with Python and Selenium but I got stuck scrapping a page trying to scroll down until selenium finds the classic "Show more" button (that seems to be hidden).

What I have done 'til now is the following:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


from selenium.webdriver.common.keys import Keys


element = browser.find_element_by_xpath('//*[@id="ember371"]/button') # you can use ANY way to locate element
coordinates = element.location_once_scrolled_into_view # returns dict of X, Y coordinates
browser.execute_script('window.scrollTo({}, {});'.format(coordinates['x'], coordinates['y']))

However, whatever I try to do Python throws me several kind of errors, for example the below.

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="ember371"]/button"}
  (Session info: chrome=80.0.3987.122)

Could it be due to the fact that the button is hidden?

Below is the HTML code for the button I am looking for and that I would like to click on.

<button class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid" aria-controls="skill-categories-expanded" aria-expanded="false" data-control-name="skill_details" data-ember-action="" data-ember-action-2182="2182">
                    <span aria-hidden="true">
                    Show more
                  </span>
                  <span class="visually-hidden">
                    BLA BLA BLA
                  </span>
                  <li-icon aria-hidden="true" type="chevron-down-icon" class="pv-skills-section__chevron-icon" size="small"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" data-supported-dps="16x16" fill="currentColor" width="16" height="16" focusable="false">
  <path d="M8 9l5.93-4L15 6.54l-6.15 4.2a1.5 1.5 0 01-1.69 0L1 6.54 2.07 5z"></path>
</svg></li-icon>

</button>

#

It worked with the following:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


# scroll down smoothly
scheight = .1
while scheight < 1.0:
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight*%s);" % scheight)
    scheight += .1
    time.sleep(1)
try:
    browser.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))
    browser.execute_script("arguments[0].click();", WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))
except:
    print('NO')

DebanjanB :

The element Show more within linkedin website is EmberJS enabled element. So to scrollIntoView the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR:

  driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button[aria-controls='skill-categories-expanded'][data-control-name='skill_details']>span"))))
  driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-controls='skill-categories-expanded'][data-control-name='skill_details']>span"))))
  

• Using XPATH:

  driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))
  driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=278949&siteId=1