我们的目标是把Netflix数据集的格式转换成：用户id、物品id、评分、时间戳格式。在开始转换之前，先下载Netflix数据集：netflix-prize-data。点击“Download”，下载文件archive.zip并解压。
我们只选用combined_data的4个文件此时文件目录如下：
接下来开始转换：
1、先导入需要用到的包
from datetime import datetime
import pandas as pd
import numpy as np
2、读入combined_data_1-4的数据
df1 = pd.read_csv('./combined_data_1.txt', header = None, names = ['user_id', 'rating', 'timestamp'], usecols=[0,1,2]) # 读入combined_data_1
# df2 = pd.read_csv('./combined_data_2.txt', header = None, names = ['user_id', 'rating', 'timestamp'], usecols=[0,1,2]) # 读入combined_data_2
# df3 = pd.read_csv('./combined_data_3.txt', header = None, names = ['user_id', 'rating', 'timestamp'], usecols=[0,1,2]) # 读入combined_data_3
# df4 = pd.read_csv('./combined_data_4.txt', header = None, names = ['user_id', 'rating', 'timestamp'], usecols=[0,1,2]) # 读入combined_data_4
df1['rating'] = df1['rating'].astype(float)
# df2['rating'] = df2['rating'].astype(float)
# df3['rating'] = df3['rating'].astype(float)
# df4['rating'] = df4['rating'].astype(float)
print('Dataset 1 shape: {}'.format(df1.shape))
# print('Dataset 2 shape: {}'.format(df2.shape))
# print('Dataset 3 shape: {}'.format(df3.shape))
# print('Dataset 4 shape: {}'.format(df4.shape))
print('-Dataset examples-')
print(df1.iloc[::5000000, :])
输出：
Dataset 1 shape: (24058263, 3)
-Dataset examples-
user_id rating timestamp
0 1: NaN NaN
5000000 2560324 4.0 2005-12-06
10000000 2271935 2.0 2005-04-11
15000000 1921803 2.0 2005-01-31
20000000 1933327 3.0 2004-11-10
3、合成4个数据集并重构索引
df = df1
# df.append(df2)
# df.append(df3)
# df.append(df4)
df.index = np.arange(0, len(df))
print('Full dataset shape: {}'.format(df.shape))
print('-Dataset examples-')
print(df.iloc[::5000000, :])
输出：
Full dataset shape: (24058263, 3)
-Dataset examples-
user_id rating timestamp
0 1: NaN NaN
5000000 2560324 4.0 2005-12-06
10000000 2271935 2.0 2005-04-11
15000000 1921803 2.0 2005-01-31
20000000 1933327 3.0 2004-11-10
4、数据清洗，去除rating为0的数据行
df_nan = pd.DataFrame(pd.isnull(df.rating))
df_nan = df_nan[df_nan['rating'] == True]
df_nan = df_nan.reset_index()
item_np = []
item_id = 1
for i,j in zip(df_nan['index'][1:],df_nan['index'][:-1]):
# 使用numpy
temp = np.full((1,i-j-1), item_id)
item_np = np.append(item_np, temp)
item_id += 1
# 考虑最后一条记录和其长度
# 使用numpy
last_record = np.full((1,len(df) - df_nan.iloc[-1, 0] - 1),item_id)
item_np = np.append(item_np, last_record)
print('Item numpy: {}'.format(item_np))
print('Length: {}'.format(len(item_np)))
输出：
Item numpy: [1.000e+00 1.000e+00 1.000e+00 ... 4.499e+03 4.499e+03 4.499e+03]
Length: 24053764
5、将物品（电影）id加入dataframe
def time2stamp(cmnttime): # 时间转时间戳函数
cmnttime = datetime.strptime(cmnttime,'%Y-%m-%d')
stamp = int(datetime.timestamp(cmnttime))
return stamp
df = df[pd.notnull(df['rating'])].copy()
df['item_id'] = item_np.astype(int)
df['user_id'] = df['user_id'].astype(int)
df = df.loc[:,['user_id', 'item_id', 'rating', 'timestamp']] # 交换两列位置
df['timestamp'] = df['timestamp'].astype(str).apply(time2stamp) # 时间转成时间戳
print('-Dataset examples-')
print(df.iloc[::5000000, :])
输出：
-Dataset examples-
user_id item_id rating timestamp
1 1488844 1 3.0 1125936000
5000996 501954 996 2.0 1093449600
10001962 404654 1962 5.0 1125244800
15002876 886608 2876 2.0 1127059200
20003825 1193835 3825 2.0 1060704000
6、保存dataframe
# df.sort_values(by=["user_id", "timestamp"], ascending=[True, True]) # 先按用户id排序，然后按时间戳排序
df.to_csv('./ratings.dat', sep=',', index=0, header=0)
完成
