Topics requirements:
1, data cleaning: Cleaning in accordance with the data, and import data washing hive data repository .
Two-stage data cleaning:
( 1 ) First stage: the required information is extracted from the original log
ip: 199.30.25.88
time: 10/Nov/2016:00:01:03 +0800
traffic: 62
Article: Article This article was / 11325
Video: Video / 3235
( 2 ) The second stage: to do fine operation based on information extracted from the
ip ---> City City ( IP )
date--> time:2016-11-10 00:01:03
day: 10
traffic:62
type:article/video
id:11325
( . 3 ) Hive database table structure :
create table data( ip string, time string , day string, traffic bigint,
type string, id string )
2 , the data processing:
· Statistical most popular video / article Top10 visits ( Video / Article This article was )
· According to the statistics of the most popular cities Top10 course ( ip )
· According to traffic statistics of the most popular Top10 course ( traffic )
3 , Data Visualization: The statistical results poured MySql database, unfolded through a graphical display mode.
The results and screenshot: currently only completed the first step
After cleaning the data:
Create a table in the hive and import data:
