MovieLens 1M数据集含有来自6000名用户对4000部电影的100万条评分数据。分为三个表：评分，用户信息，电影信息。这些数据都是dat文件格式。
读取3个数据集：
#coding=gbk
# MovieLens 1M数据集含有来自6000名用户对4000部电影的100万条评分数据。
# 分为三个表：评分，用户信息，电影信息。这些数据都是dat文件格式
# ，可以通过pandas.read_table将各个表分别读到一个pandas DataFrame对象中
import pandas as pd
import time
start = time.clock()
filename1 =r'D:\datasets\users.dat'
filename2 = r'D:\datasets\ratings.dat'
filename3 = r'D:\datasets\movies.dat'
pd.options.display.max_rows = 10
uname = ['user_id','gender','age','occupation','zip']
users = pd.read_table(filename1, sep='::', header = None, names=uname, engine='python')
print(users.head()) #年龄和职业都是使用编码的形式给出来的
# user_id gender age occupation zip
# 0 1 F 1 10 48067
# 1 2 M 56 16 70072
# 2 3 M 25 15 55117
# 3 4 M 45 7 02460
# 4 5 M 25 20 55455
print(users.shape) # (6040, 5)
rnames = ['user_id','movie_id','rating','timestamp']
ratings = pd.read_table(filename2, header =None, sep='::',names=rnames, engine= 'python')
print(ratings.head())
# user_id movie_id rating timestamp
# 0 1 1193 5 978300760
# 1 1 661 3 978302109
# 2 1 914 3 978301968
# 3 1 3408 4 978300275
# 4 1 2355 5 978824291
# print(ratings.shape) #(1000209, 4)
mnames = ['movie_id','title','genres'] # genres 表示影片的体裁是什么
movies = pd.read_table(filename3, header = None, sep='::', names = mnames, engine='python')
# print(movies.head())
# movie_id title genres
# 0 1 Toy Story (1995) Animation|Children's|Comedy
# 1 2 Jumanji (1995) Adventure|Children's|Fantasy
# 2 3 Grumpier Old Men (1995) Comedy|Romance
# 3 4 Waiting to Exhale (1995) Comedy|Drama
# 4 5 Father of the Bride Part II (1995) Comedy
# print(movies.shape) #(3883, 3)年龄和职业都是使用编码的形式给出来的：
- Age is chosen from the following ranges:
* 1: "Under 18"
* 18: "18-24"
* 25: "25-34"
* 35: "35-44"
* 45: "45-49"
* 50: "50-55"
* 56: "56+"
- Occupation is chosen from the following choices:
* 0: "other" or not specified
* 1: "academic/educator"
* 2: "artist"
* 3: "clerical/admin"
* 4: "college/grad student"
* 5: "customer service"
* 6: "doctor/health care"
* 7: "executive/managerial"
* 8: "farmer"
* 9: "homemaker"
* 10: "K-12 student"
* 11: "lawyer"
* 12: "programmer"
* 13: "retired"
* 14: "sales/marketing"
* 15: "scientist"
* 16: "self-employed"
* 17: "technician/engineer"
* 18: "tradesman/craftsman"
* 19: "unemployed"
* 20: "writer"使用merge 函数将3个表进行合并
#使用merge 函数将3个表进行合并
data = pd.merge(pd.merge(ratings, users), movies)
# print(data.head())
# user_id movie_id rating timestamp gender age occupation zip \..
# 0 1 1193 5 978300760 F 1 10 48067
# 1 2 1193 5 978298413 M 56 16 70072
# 2 12 1193 4 978220179 M 25 12 32793
# 3 15 1193 4 978199279 M 25 7 22903
# 4 17 1193 5 978158471 M 50 1 95350
# print(data.iloc[0])
# user_id 1
# movie_id 1193
# rating 5
# timestamp 978300760
# gender F
# age 1
# occupation 10
# zip 48067
# title One Flew Over the Cuckoo's Nest (1975)
# genres Drama
# Name: 0, dtype: object使用透视表，按性别计算每部电影的平均得分
#index 表示索引，values表示所要进行分析的数据， columns允许选择一个或多个列,以columns作为分组的列
mean_ratings = data.pivot_table(values ='rating', index='title', columns ='gender', aggfunc='mean')
# print(mean_ratings.head())
# gender F M
# title
# $1,000,000 Duck (1971) 3.375000 2.761905
# 'Night Mother (1986) 3.388889 3.352941
# 'Til There Was You (1997) 2.675676 2.733333
# 'burbs, The (1989) 2.793478 2.962085
# ...And Justice for All (1979) 3.828571 3.689024使用选择的数据进行分析
#过滤掉评分数据不足250 条的电影
ratings_by_title = data.groupby('title').size()
print(ratings_by_title[:3])
# title
# $1,000,000 Duck (1971) 37
# 'Night Mother (1986) 70
# 'Til There Was You (1997) 52
# dtype: int64
active_titles = ratings_by_title.index[ratings_by_title >= 250] #找出其评论大于250 的索引
print(active_titles[:3])
# Index([''burbs, The (1989)', '10 Things I Hate About You (1999)',
# '101 Dalmatians (1961)'],
# dtype='object', name='title')
#可以以active_titles 中的电影作为索引，选择出 mean_ratings 中的电影
mean_ratings = mean_ratings.loc[active_titles]
print(mean_ratings[:5])
# gender F M
# title
# 'burbs, The (1989) 2.793478 2.962085
# 10 Things I Hate About You (1999) 3.646552 3.311966
# 101 Dalmatians (1961) 3.791444 3.500000
# 101 Dalmatians (1996) 3.240000 2.911215
# 12 Angry Men (1957) 4.184397 4.328421
#查看女性观众喜欢的电影，可以按 F 列进行降序排列
top_ratings = mean_ratings.sort_values(by="F", ascending = False)
print(top_ratings[:3])
# gender F M
# title
# Close Shave, A (1995) 4.644444 4.473795
# Wrong Trousers, The (1993) 4.588235 4.478261
# Sunset Blvd. (a.k.a. Sunset Boulevard) (1950) 4.572650 4.464589
#计算男性观众和女性观众分歧最大的电影
mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
sort_by_diff = mean_ratings.sort_values(by='diff')
print(sort_by_diff[:3])
# gender F M diff
# title
# Dirty Dancing (1987) 3.790378 2.959596 -0.830782
# Jumpin' Jack Flash (1986) 3.254717 2.578358 -0.676359
# Grease (1978) 3.975265 3.367041 -0.608224
#对行进行反序操作， 取出前3行，得到是男性更喜欢的电影，而女性观众相反
print(sort_by_diff[::-1][:3])
# gender F M diff
# title
# Good, The Bad and The Ugly, The (1966) 3.494949 4.221300 0.726351
# Kentucky Fried Movie, The (1977) 2.878788 3.555147 0.676359
# Dumb & Dumber (1994) 2.697987 3.336595 0.638608
#计算得分数据的标准差，找出分歧最大的电影
rating_std = data.groupby('title')['rating'].std()
rating_std = rating_std.loc[active_titles]
print(rating_std.sort_values(ascending=False)[:3])
# title
# Dumb & Dumber (1994) 1.321333
# Blair Witch Project, The (1999) 1.316368
# Natural Born Killers (1994) 1.307198
# Name: rating, dtype: float64
end = time.clock()
spending_time = end - start
print('花费的时间为：%.2f'%spending_time + 's')
# 花费的时间为：11.13s