任务3：论文代码统计
任务主题：论文代码统计，统计所有论文出现代码的相关统计；
任务内容：使用正则表达式统计代码连接、页数和图表数据；
任务成果：学习正则表达式统计；
import pandas as pd
import numpy as np
import re
import json
import matplotlib.pyplot as plt
data = []
with open(r'arxiv-metadata-oai-2019.json','r') as f:
for idx,line in enumerate(f):
d = json.loads(line)
d = {
'abstract': d['abstract'],'categories': d['categories'],'comments': d['comments']}
data.append(d)
data = pd.DataFrame(data)
data.head()
| abstract | categories | comments | |
|---|---|---|---|
| 0 | We systematically explore the evolution of t... | astro-ph | 15 pages, 15 figures, 3 tables, submitted to M... |
| 1 | Cofibrations are defined in the category of ... | math.AT | 27 pages |
| 2 | We explore the effect of an inhomogeneous ma... | astro-ph | 6 pages, 3 figures, accepted in A&A |
| 3 | This paper has been removed by arXiv adminis... | gr-qc | This submission has been withdrawn by arXiv ad... |
| 4 | The most massive elliptical galaxies show a ... | astro-ph | 32 pages (referee format), 9 figures, ApJ acce... |
统计论文页数，也就是在comments字段中抽取pages和figures和个数，首先完成字段读取
对pages进行抽取：
# 使用正则表达式匹配，XX pages
data['pages'] = data['comments'].apply(lambda x: re.findall('[1-9][0-9]* pages', str(x)))
data['pages'].head()
0 [15 pages]
1 [27 pages]
2 [6 pages]
3 []
4 [32 pages]
Name: pages, dtype: object
# 筛选出有pages的论文
data = data[data['pages'].apply(len) > 0]
data.head()
| abstract | categories | comments | pages | |
|---|---|---|---|---|
| 0 | We systematically explore the evolution of t... | astro-ph | 15 pages, 15 figures, 3 tables, submitted to M... | [15 pages] |
| 1 | Cofibrations are defined in the category of ... | math.AT | 27 pages | [27 pages] |
| 2 | We explore the effect of an inhomogeneous ma... | astro-ph | 6 pages, 3 figures, accepted in A&A | [6 pages] |
| 4 | The most massive elliptical galaxies show a ... | astro-ph | 32 pages (referee format), 9 figures, ApJ acce... | [32 pages] |
| 5 | Differential and total cross-sections for ph... | nucl-ex | 8 pages, 13 figures | [8 pages] |
# 由于匹配得到的是一个list，如['19 pages']，需要进行转换
data['pages'] = data['pages'].apply(lambda x: float(x[0].replace(' pages', '')))
data['pages'].head()
0 15.0
1 27.0
2 6.0
4 32.0
5 8.0
Name: pages, dtype: float64
#对pages进行统计：
data['pages'].describe().astype(int)
count 80696
mean 18
std 20
min 1
25% 9
50% 14
75% 24
max 1958
Name: pages, dtype: int32
接下来按照分类统计论文页数，选取了论文的第一个类别的主要类别：
# 选择主要类别
data['categories'] = data['categories'].apply(lambda x: x.split(' ')[0])
data['categories'] = data['categories'].apply(lambda x: x.split('.')[0])
# 每类论文的平均页数
plt.figure(figsize=(12, 6))
data.groupby(['categories'])['pages'].mean().plot(kind='bar')
<matplotlib.axes._subplots.AxesSubplot at 0x2060e0629a0>
对论文图表个数进行抽取：
data['figures'] = data['comments'].apply(lambda x: re.findall('[1-9][0-9]* figures', str(x)))
data = data[data['figures'].apply(len) > 0]
data['figures'] = data['figures'].apply(lambda x: float(x[0].replace(' figures', '')))
data['figures'].head()
0 15.0
2 3.0
4 9.0
5 13.0
8 4.0
Name: figures, dtype: float64
最后我们对论文的代码链接进行提取，为了简化任务我们只抽取github链接：
# 筛选包含github的论文
data_with_code = data.copy()
data_with_code = data[
(data.comments.str.contains('github')==True)|
(data.abstract.str.contains('github')==True)
]
data_with_code['text'] = data_with_code['abstract'].fillna('') + data_with_code['comments'].fillna('')
data_with_code.shape
<ipython-input-15-e5d9adba01fd>:7: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
data_with_code['text'] = data_with_code['abstract'].fillna('') + data_with_code['comments'].fillna('')
(441, 6)
# 使用正则表达式匹配论文
pattern = '[a-zA-z]+://github[^\s]*'
data_with_code['code_flag'] = data_with_code['text'].str.findall(pattern).apply(lambda x: 0 if len(x)<1 else 1)
data_with_code['code_flag'].describe()
<ipython-input-21-a53c25917903>:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
data_with_code['code_flag'] = data_with_code['text'].str.findall(pattern).apply(lambda x: 0 if len(x)<1 else 1)
count 441.000000
mean 0.884354
std 0.320163
min 0.000000
25% 1.000000
50% 1.000000
75% 1.000000
max 1.000000
Name: code_flag, dtype: float64
data_with_code = data_with_code[data_with_code['code_flag'] >= 1]
plt.figure(figsize=(12, 6))
data_with_code.groupby(['categories'])['code_flag'].count().plot(kind='bar')
<matplotlib.axes._subplots.AxesSubplot at 0x2060cbfdd30>
