Getting Started with Web Automation Using Python's Selenium

Automation can be seen as the process of removing human beings from the use of electronic machines or robots to perform tasks.

In this post, we'll look at the automation of network processes.

The ability to have software robots automate processes and tasks on a network is known as network automation.

Using web automation we can do many things eg.

Search the web.
Delete email.
Fill in the form.
Log in to the website.

In modern society, the need for speed to perform repetitive tasks is a must, which makes automation a necessity.

Selenium is a framework for web application testing, automated software testing and scraping the web.

In python, Selenium can be seen as a set of libraries that help developers interact with the network to automate network processes.

Selenium is a very powerful tool when it comes to interacting with web browsers, it supports all modern web browsers and can be coded in various programming languages like Java, Python, C#, etc.

In this guide, we'll look at how to use Selenium to write scripts that automate basic web tasks with Python.

prerequisite

To understand this guide, the reader must be familiar with it.

HTML tags, elements, IDs and classes.
Basic knowledge of the Python programming language

Target

In this guide, we will focus on setting up two Python automation scripts.

One will do a Google search based on the keyword "university" and the other will automatically log into Quora .

By the end of this guide, readers will be able to write capable python scripts.

Find elements in the browser.
Insert text into the table area in the browser.
Click the button in your browser.

The expected result would be.

set environment

First, we need to create a virtual environment in Python.

In order to work with selenium, we will have to install selenium. To install, use the command below.

pip install selenium
复制代码

We also had to install a network driver (a tool required for network automation). Web drivers help us interact with the browser.

If you're using Windows, we'll use a chocolatey package manager called windows package manager to install the network drivers.

To install, we will use the command below.

choco install chromedriver
复制代码

If you are using macOS, we will use the command below.

brew cask install chromedriver
复制代码

chromedriver The version of should be compatible with your browser version.

If you encounter compatibility errors, then download the driver based on your browser version.

automatic google search

Create a file app.py and add the following code.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://www.google.com/')
复制代码

The code snippet above is used to open a browser and request a web page url .

The first line of code imports the web driver from Selenium. The second line opens chrome's web driver driver .

Note: Different browsers have different network drivers. If you prefer to use a different browser, look for the driver's name on the Internet.

For example, we will use firefoxDriver , for the Firefox browser.

On the third line, we use driver , to send a request to the browser to use url .

You can run the code with the command below.

python app.py
复制代码

The above code opens the Chrome browser, as shown in the figure below.

Next, we will enter a keyword into the search bar of the Google website search . To do this we will have to get the elements of the search field by inspecting the page.

To inspect the page, right-click on the Google Sites page, and click Inspect element .

The browser will open a window as shown in the image below.

Before proceeding, we need to understand what is a locator in Selenium.

Locators are the means by which we can identify web elements on a web page. They help us find any element on a web page.

We can use different types of locators to identify elements on a web page. They include - id, class, name, and xpath .

We use them as shown in the image below.

find_element_by_id.
find_element_by_name.
find_element_by_className.
find_element_by_xpath.

From above id, name, and className are HTML attributes used inside HTML tags to control their behavior.

xpath Stands for Extensible Markup Language Path** (XML path**) is a syntax for finding elements on a web page.

To get the element, div hover over the tab and keep opening tabs that highlight the search bar until you find a tab that only highlights the search bar.

Then, right-click the label and click Copy xpath . Next, paste xpath as shown.

searchField = driver.find_element_by_xpath('/html/body/div[1]/div[3]/form/div[1]/div[1]/div[1]/div/div[2]/input')
searchField.send_keys('university')

searchField.submit()
复制代码

From the code snippet above.

xpath We initialized the variable with the copied value searchField .
send_keys() is used to insert text into an object.searchField
searchField.send_keys('university') Insert the value in the search box.university
searchField.submit() Submit a search request.

submit Note: You can also search for buttons and use methods on them if there is such an element on the page click() . However, submit() methods make it easier.

Your full code will look like the snippet below.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://www.google.com/')

searchField = driver.find_element_by_xpath('/html/body/div[1]/div[3]/form/div[1]/div[1]/div[1]/div/div[2]/input')
searchField.send_keys('university')

searchField.submit()
复制代码

If you run your code, it will open the browser, request the google page, enter university a value in the search box, and automatically submit.

Automatically log in to a website

Using what we learned from the previous examples, let's try logging into the Quora website. To do this, let's create a new file in our project directory called main.py . Paste or enter the code below into the file.

from selenium import webdriver

driver = webdriver.Chrome()

driver.get('https://www.quora.com/') # Open Quora website

emailField = driver.find_element_by_xpath('//*[@id="email"]') # HTML tag element for email field
emailField.send_keys('YourEmail') # Login user name

passwordField = driver.find_element_by_xpath('//*[@id="password"]') # HTML tag element for password field
passwordField.send_keys('YourPassword') # Login password

button = driver.find_element_by_xpath('//*[@id="root"]/div[2]/div/div/div/div/div/div[2]/div[2]/div[4]/button/div/div/div') # HTML tag element for button

button.click() # onClick event handler for HTML button
复制代码

From the code snippet above.

First, we import selenium from webdriver .
To avoid multiple use webdriver.Chrome() , we store them in a variable driver .
driver.get('https://www.quora.com/') Send a request to Quora .
emailField = driver.find_element_by_xpath('//*[@id="email"]') By finding the field.xpath email
emailField.send_keys('YourEmail') Insert the email address into the field.email
passwordField = driver.find_element_by_xpath('//*[@id="password"]') By finding the password field.xpath
passwordField.send_keys('YourPassword') Insert the password into the field.password
button = driver.find_element_by_xpath('//*[@id="root"]/div[2]/div/div/div/div/div/div[2]/div[2]/div[4]/button/div/div/div'), by xpath finding login the button.
button.click() Click the button.login

When you run the app, the Chrome browser opens, sends a request to the Quora website, fills in the login details, and logs you into your Quora account.

in conclusion

In summary, we were able to write two Python scripts, do a Google search and log on Quora.

After understanding the above two examples, you will understand how to use selenium.

Go directly to any URL.
Find any HTML element.
Complete and submit any form.

An Introductory Tutorial for Web Automation Using Python's Selenium