Apo B raking programmer master station that "scared to Heaven" with the Python!

Author | Xu Lin

This article is reproduced from an authorized data Sen Lin (ID: shujusenlin)

Foreword

B station of the recent New Year's Eve because of its unique and creative swept the major video sites, the company has brought tremendous positive impact on the share price also rose, surely we are not regret it sooner buy stock B station:

However, New Year's Eve is not the B station today we have to discuss, but the core resources B stations: "scared to Heaven," the lady owners, a question was inspired by the article know almost hot standings:

data collection

The above problem answer received a total of 859, the data is also derived from this article, as many will be answered with the main link ID lady reflected in the answer, as shown below:

We can be the lady of the main problems that occur id space was crawling, but not all answers will be taken into account with this ID, so some bold fonts were extracted derive the names of some lady of the Lord, as supplementary data:

The answer to the above is a typical case, which refers to the students get very fire before Cook's birthday wishes, some code to extract data is as follows:

# 开始爬取数据
driver = webdriver.Chrome()
driver.maximize_window()
url = 'https://www.zhihu.com/question/291506148'
js='window.open("'+url+'")'
driver.execute_script(js)
for i in range(1000):
     time.sleep(1)
     js="var q=document.documentElement.scrollTop=10000000"  
     driver.execute_script(js)
     print(i)

# 整理数据
all_html = [k.get_property('innerHTML') for k in driver.find_elements_by_class_name('AnswerItem')]
all_text = ''.join(all_html)
pat = '/space.bilibili.com/\d+'
spaces = list(set([k for k in re.findall(pat,all_text)]))

Now to get to these "scared to Heaven," the lady owners of the ID, the next step is to take personal space they need to climb the B station, to get more detailed information:

The above personal space is B station renowned scientist manual Geng, from which we can get the number of manual Geng fans, video main types (always thought it should be science and technology, did not expect to actually live, B station moral integrity possible) and all videos average player amount, the number of playing screen, the number of comments, as a basis for the subsequent rank, part of the code as follows:

upstat = pd.DataFrame(columns=['name','fans','face','main_type','total_video',
                               'total_play', 'total_comment'])
for i in range(len(spaces)):
    try:
        time.sleep(1)
        space_id = str(spaces[i].replace('/space.bilibili.com/',''))
        url= 'https://api.bilibili.com/x/web-interface/card?mid={}&jsonp=jsonp&article=true'.format(space_id)
        html = requests.get(url=url, cookies=cookie, headers=header).content
        data = json.loads(html.decode('utf-8'))['data']
        this_name = data['card']['name']
        this_fans = data['card']['fans']
        this_face = data['card']['face']
        this_video = int(data['archive_count'])
        total_page = int((this_video-1)/30)+1
        video_list=[]
        for j in range(total_page):
            url = 'https://api.bilibili.com/x/space/arc/search?mid={}&ps=30&tid=0&pn={}&keyword=&order=click&jsonp=jsonp'.format(space_id,str(j+1))
            html = requests.get(url=url, cookies=cookie, headers=header).content
            data = json.loads(html.decode('utf-8'))
            if j == 0 :
                 type_list = data['data']['list']['tlist']
            this_list = data['data']['list']['vlist']
            video_list = video_list + [ this_list [k] for k in range(len(this_list))]
        type_list = list(type_list.values())
        type_list = {type_list[k]['name']:int(type_list[k]['count']) for k in range(len(type_list))}
        this_type = max(type_list,key=type_list.get)
        this_play = sum([video_list[k]['play'] for k in range(len(video_list)) if video_list[k]['play'] != '--'])
        this_comment = sum([video_list[k]['comment'] for k in range(len(video_list)) if video_list[k]['comment'] != '--'])
        upstat = upstat.append({'name':this_name,
                               'fans':this_fans,
                               'face':this_face,
                               'main_type':this_type,
                               'total_video':this_video,
                               'total_play':this_play,
                               'total_comment':this_comment},
                              ignore_index=True)
        print('success:'+str(i))
    except:
        print('fail:'+str(j))
        continue

Eventually we will get the B station more than 200 owners of old lady "scared to Heaven" of information, an overview of the data as follows:

General overview

After obtaining these data, we first look at these "scared to Heaven" main types of video distribution lady owners released:

Since the classification B station in life, all-inclusive, hand Geng Qi and plums are classified into lifestyle, think about all fantasy, so this type of video is more than a grouping, accounting for another class of digital technology and is also very much , confirms the B station is a good learning site concludes, are interested can refer to another article: do you believe in visiting the station B can also learn programming?

除此之外的视频可统称为娱乐类，包括了游戏、影视这些，之后会将视频类型按照科技、生活、娱乐进行大的划分，寻找各个类别最为“惊为天人”的阿婆主。

在开始进行正式的排名前，首先用Python将这些阿婆主的头像进行拼接，获得下面的图片，看看一眼望去有多少是你非常熟悉的阿婆主：

这部分代码如下：

i = 0 
for i in range(upstat.shape[0]):
    loc = 'D:/爬虫/惊为天人/'+upstat['name'][i]+'.jpg'
 # request.urlretrieve(upstat['face'][i],loc)
    img = mpimg.imread(loc)[:,:,0:3]
    img = cv2.resize(img, (500,500),interpolation=cv2.INTER_CUBIC)
    if i % 20 == 0:
        row_img=img
    elif i == 19:
        row_img=np.hstack((row_img,img))
        all_img = row_img
    elif i % 20 == 19:
        row_img=np.hstack((row_img,img))
        all_img = np.vstack((all_img,row_img))
    else:
        row_img=np.hstack((row_img,img))
    i = i+1    
plt.axis('off')
plt.margins(0,0)
plt.imshow(all_img)
plt.savefig('头像.png',dpi=1000)