What is Web Scraping
This is also referred to as web harvesting and web data extraction.
This is the process of automatically downloading a web page's data and extracting information from it.
Benefits of Web Scraping
Component of applications used for web indexing. e.g. Google
Web and data mining
Online price monitoring
Online price comparison
Product review to watch the competition
Gather real estate listing
Weather data monitoring
Website change detection
Research
Basic Rules for Web Scraping
Always check a website's Terms and Conditions before you scape it to avoid legal issues.
Do not request data from a website too aggressively(spamming) with your program as this may overload and break the website.
Tools used for Web Scraping
- Scrapy
- Scrapy is a free open source application framework.
- It is used for crawling web sites and extracting data.
- Can be installed using pip: pip install scrapy
- Beautiful Soup
- This is a python library used to extract data from HTML and XML files.
- Can be installed using pip: pip install beautifualsoup4(bs4)
Target Website:https://bluelimelearning.github.io/my-fav-quotes/