1. Write a set of data into mysql in python, the main point is to realize the initial connection between python and MySQL:
import pymysql # Import the pymysql module, so that we can connect to mysql, but we still need to use -u root -p in DOS Enter the password to log in, otherwise the connection will report an error, you can download another module to solve it, but I don’t think it is necessary to log in, otherwise there are too many modules
db = pymysql.Connect(host='localhost', port=3306, user ='root', # connect to the database MySQL
passwd='******', database='db_securities', charset='utf8') #This is the pymysql.Connect function, you can study the parameters in it, just take it directly To use, you only need to change your password is passwd='******' and the database to be connected database='db_securities'
cursor = db.cursor() # quite create a cursor
sql = """ # Typical SQL Statement, in the form of str
insert into t_pe(date, 10yearsyotb, cna, hs300, zz500, zz1000, cyb, kcb, shch, hkt, nsdq100)
values('2022-10-21', 2.7287, 16.36, 10.98, 20.96, 28.35, 45.43, 40.4, 54.72, 36.95, 23.45)
"""
cursor.execute(sql) # Execute the SQL statement is equivalent to ";+enter", when SQL encounters; end, press enter to execute
db.commit() # Submit the command to write to the hard disk, but MySQL submits by default, if you don't change it randomly
cursor.close() # Close the cursor
db.close() # Close the database
2. Python connects to mysql and writes simple crawler data, focusing on json
import requests # crawler requests module
import re # regular matching module
import json # transfer module, which can make dictionary-like strings into dictionaries and dictionary strings Make it into a dictionary, it is very powerful anyway
import pymysql
index_tuple = [] # Create an empty list to put the crawled data
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64)'
'AppleWebKit/537.36 (KHTML , like Gecko) Chrome/86.0.4240.198 Safari/537.36'} # Copy a 'user-agent' to make the crawler simulate the user more realistically, and you can also use cookies and proxy IPs (the free ones have been used badly) rule =
r 'quote: (.*),' # Specify regular matching rules, look at the source code data of the webpage, what is the beginning and end of the data you need, and then match it response = requests.get('
https://xueqiu.com /S/SH000001', headers=headers) # Send a get request, post request requires interactive parameters, for example, if you want to enter a verification code, result =
re.findall(rule, response.text) # Match our results , the matching is a type of ['{target data}']
Take out '{target data}' from the list
data1 = json.loads(data) # Use json to restore it into a dictionary, and then get the value according to the key, which is to repeat the connection between the crawler part and the first part
index_tuple.extend([float(data1['current']), data1[ 'amount']/100000000])
response = requests.get('https://xueqiu.com/S/SZ399006', headers=headers)
result = re.findall(rule, response.text)
data = result[0]
data1 = json.loads(data)
index_tuple.extend([float(data1['current']), data1['amount']/100000000])
index_list = []
asd = round(index_tuple[1]+index_tuple[3] ,2)
date = '2022-10-21'
db = pymysql.Connect(host='localhost', port=3306, user='root', # connect to the database MySQL
passwd='******', database ='db_securities', charset='utf8')db_securities', charset='utf8')
cursor = db.cursor()
sql = "insert into t_index(date, Shanghai Composite Index, GEM Index, 2 market turnover, GEM turnover)" \ "
values('%s', '%.2f', '%.2f', '% .2f', '%.2f')" %(date, index_tuple[0], index_tuple[4], asd, index_tuple[5]) cursor.execute(sql) db.commit() cursor.close
(
)
db
. close()
