Using regular expressions
Regular expressions related knowledge
When writing a program or web page processing strings, there is often a need to find a string meet certain complex rules of regular expressions is the tool for a description of these rules, in other words Regular expressions are a tool that It defines the pattern string matching (how to check whether a character string with a pattern matching part or extracted from or replace a portion where the character string matches the pattern). If you've used Windows operating system files used to find and wildcards when specifying the file name (* and?), Then the regular expressions are also used for tools with a similar matching text, but compared to the regular expression wildcards style more powerful, it can more accurately describe your needs (of course, the price you pay is writing a regular expression is much more complex than a wildcard to play, to know anything to bring you all the benefits of a price, Like learning a programming language), for example, you can write a regular expression used to find all starts with 0, followed by 2-3 digits, then a hyphen "-", the last seven or eight numeric string (like 0813-7654321 or 028-12345678), is not this domestic landline number, please. Initially the computer to do the math and birth, information processing are basically value, and today we are dealing with information in their daily work are basically text data, we want the computer to be able to recognize and process certain patterns in line with text , the regular expression is very important. Today, almost all programming languages provide support for regular expression operations, Python supports regular expressions operated by the standard library re module.
We can consider the following question: We (could be a text file, it could be a news on the network) to get a string from somewhere, hoping to find the phone number and landline number in the string. Of course, we can set the phone number is a 11-bit digital (note not a random 11-digit number, because you have not seen "25012345678" This phone number it) and keep the same pattern described for a landline number, if not use regular expressions to accomplish this task will be very troublesome.
Knowledge about regular expressions, you can read a very famous blog called "regular expressions 30 minutes Guide" , after reading this article you can read the following table, which is our expression of positive a brief summary of some of the basic symbols.
| symbol | Explanation | Examples | Explanation |
|---|---|---|---|
| . | Matches any character | b.t | Match bat / but / b # t / b1t etc. |
| \w | Match letter / number / underscore | b\wt | Match bat / b1t / b_t the like but can not match b # t |
| \s | Match blank characters (including \ r, \ n, \ t, etc.) | love\syou | Match love you |
| \d | Matching numbers | \d\d | Match 01/23/99 etc. |
| \b | Matching word boundaries | \bThe\b | |
| ^ | Matches the beginning of the string | ^The | The match beginning of string |
| $ | The end of the match the string | .exe$ | You can match strings ending in .exe |
| \W | Non-matching letter / number / underscore | b\Wt | Match b # t / b @ t like , but it does not match but / b1t / b_t etc. |
| \S | Matching non-whitespace characters | love\Syou | Love # you can match and so on , but can not match love you |
| \D | Matching non-numeric | \d\D | Match 9a / 3 # / 0F, etc. |
| \B | Matching non-word boundary | \ Bio \ B | |
| [] | Matches any single character from the character set | [aeiou] | Matches any character a vowel |
| [^] | Matches any single character not in the character set | [^aeiou] | Matches any of the non-vowel characters |
| * | Match 0 or more times | \w* | |
| + | Match 1 or more times | \w+ | |
| ? | Match 0 or 1 | \w? | |
| {N} | Match N times | \w{3} | |
| {M,} | Match at least M times | \w{3,} | |
| {M,N} | Matching at least N times at most M times | \w{3,6} | |
| | | Branch | foo|bar | You can match foo or bar |
| (?#) | Note | ||
| (exp) | Exp and matching the captured automatically named group | ||
| (? <name>exp) | Exp group match and capture the named name in | ||
| (?:exp) | Exp match but does not capture the match text | ||
| (?=exp) | Matching the position in front of exp | \b\w+(?=ing) | 可以匹配I'm dancing中的danc |
| (?<=exp) | 匹配exp后面的位置 | (?<=\bdanc)\w+\b | 可以匹配I love dancing and reading中的第一个ing |
| (?!exp) | 匹配后面不是exp的位置 | ||
| (?<!exp) | 匹配前面不是exp的位置 | ||
| *? | 重复任意次，但尽可能少重复 | a.*b a.*?b |
将正则表达式应用于aabab，前者会匹配整个字符串aabab，后者会匹配aab和ab两个字符串 |
| +? | 重复1次或多次，但尽可能少重复 | ||
| ?? | 重复0次或1次，但尽可能少重复 | ||
| {M,N}? | 重复M到N次，但尽可能少重复 | ||
| {M,}? | 重复M次以上，但尽可能少重复 |
说明： 如果需要匹配的字符是正则表达式中的特殊字符，那么可以使用\进行转义处理，例如想匹配小数点可以写成\.就可以了，因为直接写.会匹配任意字符；同理，想匹配圆括号必须写成\(和\)，否则圆括号被视为正则表达式中的分组。
Python对正则表达式的支持
Python提供了re模块来支持正则表达式相关操作，下面是re模块中的核心函数。
| 函数 | 说明 |
|---|---|
| compile(pattern, flags=0) | 编译正则表达式返回正则表达式对象 |
| match(pattern, string, flags=0) | 用正则表达式匹配字符串 成功返回匹配对象 否则返回None |
| search(pattern, string, flags=0) | 搜索字符串中第一次出现正则表达式的模式 成功返回匹配对象 否则返回None |
| split(pattern, string, maxsplit=0, flags=0) | 用正则表达式指定的模式分隔符拆分字符串 返回列表 |
| sub(pattern, repl, string, count=0, flags=0) | 用指定的字符串替换原字符串中与正则表达式匹配的模式 可以用count指定替换的次数 |
| fullmatch(pattern, string, flags=0) | match函数的完全匹配（从字符串开头到结尾）版本 |
| findall(pattern, string, flags=0) | 查找字符串所有与正则表达式匹配的模式 返回字符串的列表 |
| finditer(pattern, string, flags=0) | 查找字符串所有与正则表达式匹配的模式 返回一个迭代器 |
| purge() | 清除隐式编译的正则表达式的缓存 |
| re.I / re.IGNORECASE | 忽略大小写匹配标记 |
| re.M / re.MULTILINE | 多行匹配标记 |
说明： 上面提到的re模块中的这些函数，实际开发中也可以用正则表达式对象的方法替代对这些函数的使用，如果一个正则表达式需要重复的使用，那么先通过compile函数编译正则表达式并创建出正则表达式对象无疑是更为明智的选择。
下面我们通过一系列的例子来告诉大家在Python中如何使用正则表达式。
例子1：验证输入用户名和QQ号是否有效并给出对应的提示信息。
"""
验证输入用户名和QQ号是否有效并给出对应的提示信息
要求：用户名必须由字母、数字或下划线构成且长度在6~20个字符之间，QQ号是5~12的数字且首位不能为0
"""
import re
def main():
username = input('请输入用户名: ')
qq = input('请输入QQ号: ')
# match函数的第一个参数是正则表达式字符串或正则表达式对象
# 第二个参数是要跟正则表达式做匹配的字符串对象
m1 = re.match(r'^[0-9a-zA-Z_]{6,20}$', username)
if not m1:
print('请输入有效的用户名.')
m2 = re.match(r'^[1-9]\d{4,11}$', qq)
if not m2:
print('请输入有效的QQ号.')
if m1 and m2:
print('你输入的信息是有效的!')
if __name__ == '__main__':
main()提示： 上面在书写正则表达式时使用了“原始字符串”的写法（在字符串前面加上了r），所谓“原始字符串”就是字符串中的每个字符都是它原始的意义，说得更直接一点就是字符串中没有所谓的转义字符啦。因为正则表达式中有很多元字符和需要进行转义的地方，如果不使用原始字符串就需要将反斜杠写作\\，例如表示数字的\d得书写成\\d，这样不仅写起来不方便，阅读的时候也会很吃力。
例子2：从一段文字中提取出国内手机号码。
下面这张图是截止到2017年底，国内三家运营商推出的手机号段。
import re
def main():
# 创建正则表达式对象 使用了前瞻和回顾来保证手机号前后不应该出现数字
pattern = re.compile(r'(?<=\D)1[34578]\d{9}(?=\D)')
sentence = '''
重要的事情说8130123456789遍，我的手机号是13512346789这个靓号，
不是15600998765，也是110或119，王大锤的手机号才是15600998765。
'''
# 查找所有匹配并保存到一个列表中
mylist = re.findall(pattern, sentence)
print(mylist)
print('--------华丽的分隔线--------')
# 通过迭代器取出匹配对象并获得匹配的内容
for temp in pattern.finditer(sentence):
print(temp.group())
print('--------华丽的分隔线--------')
# 通过search函数指定搜索位置找出所有匹配
m = pattern.search(sentence)
while m:
print(m.group())
m = pattern.search(sentence, m.end())
if __name__ == '__main__':
main()说明： 上面匹配国内手机号的正则表达式并不够好，因为像14开头的号码只有145或147，而上面的正则表达式并没有考虑这种情况，要匹配国内手机号，更好的正则表达式的写法是：
(?<=\D)(1[38]\d{9}|14[57]\d{8}|15[0-35-9]\d{8}|17[678]\d{8})(?=\D)，国内最近好像有19和16开头的手机号了，但是这个暂时不在我们考虑之列。
例子3：替换字符串中的不良内容
import re
def main():
sentence = '你丫是傻叉吗? 我操你大爷的. Fuck you.'
purified = re.sub('[操肏艹]|fuck|shit|傻[比屄逼叉缺吊屌]|煞笔',
'*', sentence, flags=re.IGNORECASE)
print(purified) # 你丫是*吗? 我*你大爷的. * you.
if __name__ == '__main__':
main()说明： re模块的正则表达式相关函数中都有一个flags参数，它代表了正则表达式的匹配标记，可以通过该标记来指定匹配时是否忽略大小写、是否进行多行匹配、是否显示调试信息等。如果需要为flags参数指定多个值，可以使用按位或运算符进行叠加，如
flags=re.I | re.M。
例子4：拆分长字符串
import re
def main():
poem = '窗前明月光，疑是地上霜。举头望明月，低头思故乡。'
sentence_list = re.split(r'[，。, .]', poem)
while '' in sentence_list:
sentence_list.remove('')
print(sentence_list) # ['窗前明月光', '疑是地上霜', '举头望明月', '低头思故乡']
if __name__ == '__main__':
main()后话
如果要从事爬虫类应用的开发，那么正则表达式一定是一个非常好的助手，因为它可以帮助我们迅速的从网页代码中发现某种我们指定的模式并提取出我们需要的信息，当然对于初学者来收，要编写一个正确的适当的正则表达式可能并不是一件容易的事情（当然有些常用的正则表达式可以直接在网上找找），所以实际开发爬虫应用的时候，有很多人会选择Beautiful Soup或Lxml来进行匹配和信息的提取，前者简单方便但是性能较差，后者既好用性能也好，但是安装稍嫌麻烦，这些内容我们会在后期的爬虫专题中为大家介绍。