hadoop平台使用python编写mapreduce二次排序小程序

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/w1573007/article/details/71211331

接上一个博文的环境
使用的是官网的专利使用数据，这里只截取了一部分

3858241,956203
3858241,1324234
3858241,3398406
3858241,3557384
3858241,3634889
3858242,1515701
3858242,3319261
3858242,3668705
3858242,3707004
3858243,2949611
3858243,3146465
3858243,3156927
3858243,3221341
3858243,3574238
3858243,3681785
3858243,3684611
3858244,14040
3858244,17445

mapper.py

  1 #!/usr/bin/env python
  2 import sys
  3 list1=[]
  4 for line in sys.stdin:
  5     line=line.strip()
  6     words = line.split("\n")
  7     list1.append(words[0])
  8 for x in xrange(len(list1)):
  9     print list1[x]

reducer.py

1 #!/usr/bin/env python
  2 from operator import itemgetter
  3 import sys
  4 dick1={}
  5 for line in sys.stdin:
  6     words = line.split("\n")
  7     if words[0][0]=='\t':
  8         continue
  9     else:
 10 
 11         word =words[0].split(",")
 12 #        print word[1]
 13 #        print "%s\t%s" % (word[0],word[1])     
 14         if str(word[0]) in dick1:
 15             dick1[word[0]].append(int((word[1].split("\t"[0]))[0]))
 16         else:
 17             dick1[word[0]]=[int((word[1].split("\t"[0]))[0])]
 18 for one in dick1:
 19     dick1[one].sort()  #进行单个键内对多个值的排序
 20 dick2=sorted(dick1.items(), key=lambda d:d[0], reverse = False)   #进行键的排序
 21 for one in dick2:
 22     print one

输出结果

[hadoop@mylab-client py]$ hadoop fs -cat out/part-00000
('3858241', [956203, 1324234, 3398406, 3557384, 3634889])   
('3858242', [1515701, 3319261, 3668705, 3707004])   
('3858243', [2949611, 3146465, 3156927, 3221341, 3574238, 3681785, 3684611])    
('3858244', [14040, 17445, 2211676])    
[hadoop@mylab-client py]$ 

再进行格式化输出即可