Python crawler and pytorch basic knowledge study notes

1. Generate sequential list/tuple

s = [x for x in range(1,10)]

2. Python output formatting

print("Python{} {}/{}/{}".format('学习',2019,9,6))
print("Python%s %d/%2d/%2d"%('学习',2019,9,6))# %s输出字符串，%2d输出2格整数
#此外还有%f，%4.2f等控制小数点后位数

3.Lambda function usage

Used to define an anonymous function. The function function is very simple, there is no need to define a function separately, at this time, lambda comes in handy.

f = lambda x: x*x + 1
f(3)
f(5)

4.map function usage

The map function, used for one-to-one mapping, is very simple. The input parameters are the mapping relationship and sequence (usually a list, but also tuples). A one-to-one mapping of the elements in the sequence through a defined function returns a list.

f = lambda x: x*x + 1
s = [x for x in range(1,10)]
ss = map(f,s)
for i in ss:
	print("函数值：\t%d"%(i))

5.Python special notes

#!/usr/bin/python3 env

This sentence indicates the location of the python compiler. In the Windows environment, this is not necessary, but it must be in the Linux environment. To be compatible with the Linux environment, add this sentence.

#coding=utf-8

This sentence specifies the character encoding, python3 defaults to utf-8, but python2 uses ASCII encoding. In order to be compatible with python2, add this sentence.

6. Use regular expressions to remove non-Chinese characters in the string

import re #引入正则
def remove(text):
	remove_chars = '[A-Z0-9’a-z!"#$%&\'()*+,-./:;<=>?@，。?★、…【】《》？“”‘’！[\\]^_`{|}~]+'
    return re.sub(remove_chars, '', text)

7. Make the statement execute only when it is running, and not be executed when called

if __name__ == "__main__":
	print("Print only in running")
#if下面的语句将在被调用时不会执行

8.Python read and write file method

Basically the same as C language keywords

r	Read only mode	The file must exist (default mode)
w	Write only mode	If the file does not exist, it will be created, if it exists, it will be written after being emptied.
a	Append mode	If the file does not exist, create it, and append the content if it exists
r+	Read and write mode	The file must exist, start from the beginning, overwrite as much as you write, and be readable at the same time
w+	Read and write mode	If the file does not exist, it will be created. If it exists, it will be written and read at the same time.
a+	Read and write mode	Create the file if it does not exist, write it after the content if it exists, and read it

Read and write example:

fr = open('test1.txt','r+')#打开文件
for line in fr: #遍历文件
    print('每行内容为：',line) #打印文件中每一行内容
fr.close()#关闭文件

fw = open('test2.txt','w+')#打开或创建文件
fw.write('Python写入操作')
fw.close()#关闭文件

Note: During the read operation, pay attention to whether the file pointer is at the end of the file. If it is at the end, you need to reset the pointer to the beginning with seek (0) to read the file content, such as: fr.seek(0)

9. TXT file encoding

The txt file defaults to gbk encoding, but when the character encoding is utf-8, an error will be reported. At this time, you need to specify the character encoding as utf-8

with open('test.txt','w',encoding = 'utf-8') as fp:
	fp.write(result)

10. Two ways to reference Item when applying scrapy module

1. The following will prompt a typo in Pycharm, but it can still run

from FindMovies.items import FindmoviesItem

2. Instead of using the following method, there will be no problems

import sys
sys.path.append('E:\\PyCharm2017\\program\\FindMovies\\FindMovies')# 当前items所在的路径
from items import FindmoviesItem

11. The establishment of scrapy framework under multiple python versions of the computer

1. In general, the framework of scrapy is established, just find the path where the program is stored on the command line and enter the following command:

scrapy startProject scrapyDemo	#建立名为scrapyDemo爬虫框架
tree scrapyDemo	#查看框架产生的项目分支

If the following project branch is returned, it means the creation is successful

. 2. If there are multiple versions of python,
Fatal error in launcher: Unable to create process using... may appear . At this time, you need to enter the command to specify the python version To create:

python -m scrapy startproject scrapyDemo

Now it can be created normally

12. Download address and save location of pytorch model

1. The download link of each model of pytorch is as follows:
   link: blog ; directly copy the download address and paste in the browser to download.
2. The storage location of the downloaded model can be determined by the following way:
   input in IDLE

import torchvisvion.models as models
resnet = models.resnet18(pretrained=True)

It will pop up to download the file and its storage location, just copy the downloaded model to the changed path.

13. The difference between CNN and RNN

CNN is the abbreviation of Convolutional Neural Network, and RNN is the abbreviation of Recurrent Neural Network. The limitation of CNN is that its input and output sequence is fixed, while RNN is variable, which is represented graphically as follows:

CNN is one to one, and RNN is one to many, many to one, and many to many. RNN is suitable for scenarios with variable input and output sequences, such as machine translation, text processing, etc. In general, RNN is designed to handle variable length data.

14. Errors encountered and solutions

1.TypeError: not all arguments converted during string formatting

This is a problem that the parameter does not correspond to, and it appears when using% to pass the parameter.
Solution:
Replace% with .format().