Tools and Libraries to Install
development tools
python https://www.python.org/
pycharm https://www.jetbrains.com/pycharm/
You can go directly to the official website to download and install
Built-in basic library
urllib re
>>> from urllib.request import urlopen
>>> response = urlopen("http://www.baidu.com")
>>> response
<http.client.HTTPResponse object at 0x1032edb38>
network request library
requests http://cn.python-requests.org/zh_CN/latest/
>>> import requests
>>> response = requests.get("http://www.baidu.com")
>>> response
<Response [200]>
browser tools
selenium https://www.seleniumhq.org/
chromedriver
google official website: https://sites.google.com/a/chromium.org/chromedriver/downloads
Taobao mirror: https://npm.taobao.org/mirrors/chromedriver/
>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get("http://www.baidu.com")
>>> driver.get("https://www.python.org")
>>> html = driver.page_source
phantomjs http://phantomjs.org/
>>> from selenium import webdriver
>>> dirver = webdriver.PhantomJS()
>>> dirver.get("http://www.baidu.com")
>>> html = driver.page_source
Web parsing library
lxml http://lxml.de/
beautifulsoup4 https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/
>>> from bs4 import BeautifulSoup as BS
>>> html = "<html><h1></h1></html>"
>>> soup = BS(html, "lxml")
>>> soup.h1
<h1></h1>
pyquery https://pythonhosted.org/pyquery/
>>> from pyquery import PyQuery as pq
>>> html = "<html><h1>title</h1></html>"
>>> doc = pq(html)
>>> doc("html").text()
'title'
>>> doc("h1").text()
'title'
database
mysql https://dev.mysql.com/downloads/mysql/
redis https://redis.io/
mongobd https://www.mongodb.com/
mac os can be installed using brew https://docs.brew.sh /
Database package:
pymysql
>>> import pymysql https://pypi.org/project/PyMySQL/
>>> conn = pymysql.connect(host="localhost",
user="root", password="123456",
port=3306, db="demo")
>>> cursor = conn.cursor()
>>> sql = "select * from mytable"
>>> cursor.execute(sql)
3
>>> cursor.fetchone()
(1, datetime.date(2018, 4, 14))
>>> cursor.close()
>>> conn.close()
pymongo http://api.mongodb.com/python/current/index.html
>>> import pymongo
>>> client = pymongo.MongoClient("localhost")
>>> db = client["newtestdb"]
>>> db["table"].insert({"name": "Tom"})
ObjectId('5adcb250d7696c839a251658')
>>> db["table"].find_one({"name": "Tom"})
{'_id': ObjectId('5adcb250d7696c839a251658'), 'name': 'Tom'}
say again
>>> import redis
>>> r = redis.Redis("localhost", 6379)
>>> r.set("name", "Tom")
True
>>> r.get("name")
b'Tom'
web framework package:
flask http://docs.jinkan.org/docs/flask/
django https://www.djangoproject.com/
jupyter http://jupyter.org/
run: jupyter notebook
shortcut key adds a line: b
One command to install all the above libraries
pip install requests selenium beautifulsoup4 pyquery pymysql pymongo redis flask django jupyter