Mahout推荐算法之SlopOne
一、 算法原理
有别于基于用户的协同过滤和基于item的协同过滤，SlopeOne采用简单的线性模型估计用户对item的评分。如下图，估计UserB对ItemJ的偏好
图（1）
在真实情况下，该方法有如下几个问题：
1. 为什么要选择UserA计算？
2. 对大量稀疏的情况如何处理，而这种情况是最为普遍的。
图（2）
Item1和item2的相似度：（（5-3）+（3-4））/2=0.5
Item1和Item3的相似度：（5-2）/1=3
Lucy对Item1的评估预估计为：（（2+0.5）*2+（3+5）*1）/(2+1)=4.333
Item3和Item1的相似度：（2-3）/1=-1
Item3和Item2的相似度：（5-2）/1=3
Make对item3的评分预估计为：（（4+3）*1+（3-1）*1）/(1+1)=4.5
通过以上例子可以看出:需要计算item对之间的平均差别，以及item对之间的差别次数。
Mahout给出的训练伪代码：
| for every item i for every other item j for every user u expressing preference for both i and j add the difference in u’s preference for i and j to an average
|
推荐伪代码：
for every item i the user u expresses no preference for for every item j that user u expresses a preference for find the average preference difference between j and i add this diff to u’s preference value for j add this to a running average return the top items, ranked by these averages |
二、 单机模型实现
(一) 构建difference
1. 单机模型构建（MemoryDiffStorage）
private void buildAverageDiffs() throws TasteException { log.info("Building average diffs..."); try { buildAverageDiffsLock.writeLock().lock(); averageDiffs.clear(); long averageCount = 0L; LongPrimitiveIterator it = dataModel.getUserIDs(); while (it.hasNext()) { averageCount = processOneUser(averageCount, it.nextLong()); } pruneInconsequentialDiffs(); updateAllRecommendableItems(); } finally { buildAverageDiffsLock.writeLock().unlock(); } } private void pruneInconsequentialDiffs() { // Go back and prune inconsequential diffs. "Inconsequential" means, here, only represented by one // data point, so possibly unreliable Iterator<Map.Entry<Long,FastByIDMap<RunningAverage>>> it1 = averageDiffs.entrySet().iterator(); while (it1.hasNext()) { FastByIDMap<RunningAverage> map = it1.next().getValue(); Iterator<Map.Entry<Long,RunningAverage>> it2 = map.entrySet().iterator(); while (it2.hasNext()) { RunningAverage average = it2.next().getValue(); if (average.getCount() <= 1) { it2.remove(); } } if (map.isEmpty()) { it1.remove(); } else { map.rehash(); } } averageDiffs.rehash(); } private void updateAllRecommendableItems() throws TasteException { FastIDSet ids = new FastIDSet(dataModel.getNumItems()); for (Map.Entry<Long,FastByIDMap<RunningAverage>> entry : averageDiffs.entrySet()) { ids.add(entry.getKey()); LongPrimitiveIterator it = entry.getValue().keySetIterator(); while (it.hasNext()) { ids.add(it.next()); } } allRecommendableItemIDs.clear(); allRecommendableItemIDs.addAll(ids); allRecommendableItemIDs.rehash(); } private long processOneUser(long averageCount, long userID) throws TasteException { log.debug("Processing prefs for user {}", userID); // Save off prefs for the life of this loop iteration PreferenceArray userPreferences = dataModel.getPreferencesFromUser(userID); int length = userPreferences.length(); for (int i = 0; i < length; i++) { // Loop to length-1, not length-2, not for diffs but average item pref float prefAValue = userPreferences.getValue(i); long itemIDA = userPreferences.getItemID(i); FastByIDMap<RunningAverage> aMap = averageDiffs.get(itemIDA); if (aMap == null) { aMap = new FastByIDMap<RunningAverage>(); averageDiffs.put(itemIDA, aMap); } for (int j = i + 1; j < length; j++) { // This is a performance-critical block long itemIDB = userPreferences.getItemID(j); RunningAverage average = aMap.get(itemIDB); if (average == null && averageCount < maxEntries) { average = buildRunningAverage(); aMap.put(itemIDB, average); averageCount++; } if (average != null) { average.addDatum(userPreferences.getValue(j) - prefAValue); } } RunningAverage itemAverage = averageItemPref.get(itemIDA); if (itemAverage == null) { itemAverage = buildRunningAverage(); averageItemPref.put(itemIDA, itemAverage); } itemAverage.addDatum(prefAValue); } return averageCount; } private RunningAverage buildRunningAverage() { return stdDevWeighted ? new FullRunningAverageAndStdDev() : new FullRunningAverage(); } |
2. MapReduce模式构建（FileDiffStorage）
用MapReduce模式计算difference的部分参看下文。该方式是离线计算模式，不能实施更新，适合大数据量。由于mapreduce模式计算了所有item之间的全部值，故比单机模式更准确。构建好之后拷贝到本地，使用用FileDiffStorage(newFile("diff"), 500) 即可。FileDiffStorage不支持添加和删除pereference(实际上也是不能这么做的);
| private void buildDiffs() { if (buildAverageDiffsLock.writeLock().tryLock()) { try {
averageDiffs.clear(); allRecommendableItemIDs.clear();
FileLineIterator iterator = new FileLineIterator(dataFile, false); String firstLine = iterator.peek(); while (firstLine.isEmpty() || firstLine.charAt(0) == COMMENT_CHAR) { iterator.next(); firstLine = iterator.peek(); } long averageCount = 0L; while (iterator.hasNext()) { averageCount = processLine(iterator.next(), averageCount); }
pruneInconsequentialDiffs(); updateAllRecommendableItems();
} catch (IOException ioe) { log.warn("Exception while reloading", ioe); } finally { buildAverageDiffsLock.writeLock().unlock(); } } }
private long processLine(String line, long averageCount) {
if (line.isEmpty() || line.charAt(0) == COMMENT_CHAR) { return averageCount; }
String[] tokens = SEPARATOR.split(line); Preconditions.checkArgument(tokens.length >= 3 && tokens.length != 5, "Bad line: %s", line);
long itemID1 = Long.parseLong(tokens[0]); long itemID2 = Long.parseLong(tokens[1]); double diff = Double.parseDouble(tokens[2]); int count = tokens.length >= 4 ? Integer.parseInt(tokens[3]) : 1; boolean hasMkSk = tokens.length >= 5;
if (itemID1 > itemID2) { long temp = itemID1; itemID1 = itemID2; itemID2 = temp; }
FastByIDMap<RunningAverage> level1Map = averageDiffs.get(itemID1); if (level1Map == null) { level1Map = new FastByIDMap<RunningAverage>(); averageDiffs.put(itemID1, level1Map); } RunningAverage average = level1Map.get(itemID2); if (average != null) { throw new IllegalArgumentException("Duplicated line for item-item pair " + itemID1 + " / " + itemID2); } if (averageCount < maxEntries) { if (hasMkSk) { double mk = Double.parseDouble(tokens[4]); double sk = Double.parseDouble(tokens[5]); average = new FullRunningAverageAndStdDev(count, diff, mk, sk); } else { average = new FullRunningAverage(count, diff); } level1Map.put(itemID2, average); averageCount++; }
allRecommendableItemIDs.add(itemID1); allRecommendableItemIDs.add(itemID2);
return averageCount; }
private void pruneInconsequentialDiffs() { // Go back and prune inconsequential diffs. "Inconsequential" means, here, only represented by one // data point, so possibly unreliable Iterator<Map.Entry<Long,FastByIDMap<RunningAverage>>> it1 = averageDiffs.entrySet().iterator(); while (it1.hasNext()) { FastByIDMap<RunningAverage> map = it1.next().getValue(); Iterator<Map.Entry<Long,RunningAverage>> it2 = map.entrySet().iterator(); while (it2.hasNext()) { RunningAverage average = it2.next().getValue(); if (average.getCount() <= 1) { it2.remove(); } } if (map.isEmpty()) { it1.remove(); } else { map.rehash(); } } averageDiffs.rehash(); }
private void updateAllRecommendableItems() { for (Map.Entry<Long,FastByIDMap<RunningAverage>> entry : averageDiffs.entrySet()) { allRecommendableItemIDs.add(entry.getKey()); LongPrimitiveIterator it = entry.getValue().keySetIterator(); while (it.hasNext()) { allRecommendableItemIDs.add(it.next()); } } allRecommendableItemIDs.rehash(); } |
(二) 估值
private float doEstimatePreference(long userID, long itemID) throws TasteException { double count = 0.0; double totalPreference = 0.0; PreferenceArray prefs = getDataModel().getPreferencesFromUser(userID); RunningAverage[] averages = diffStorage.getDiffs(userID, itemID, prefs); int size = prefs.length(); for (int i = 0; i < size; i++) { RunningAverage averageDiff = averages[i]; if (averageDiff != null) { double averageDiffValue = averageDiff.getAverage(); if (weighted) { double weight = averageDiff.getCount(); if (stdDevWeighted) { double stdev = ((RunningAverageAndStdDev) averageDiff).getStandardDeviation(); if (!Double.isNaN(stdev)) { weight /= 1.0 + stdev; } // If stdev is NaN, then it is because count is 1. Because we're weighting by count, // the weight is already relatively low. We effectively assume stdev is 0.0 here and // that is reasonable enough. Otherwise, dividing by NaN would yield a weight of NaN // and disqualify this pref entirely } totalPreference += weight * (prefs.getValue(i) + averageDiffValue); count += weight; } else { totalPreference += prefs.getValue(i) + averageDiffValue; count += 1.0; } } } if (count <= 0.0) { RunningAverage itemAverage = diffStorage.getAverageItemPref(itemID); return itemAverage == null ? Float.NaN : (float) itemAverage.getAverage(); } else { return (float) (totalPreference / count); } } |
(三) 推荐
对于在线推荐系统，允许只有一个SlopeOneRecommender实例。
| 方法签名 |
说明 |
备注 |
| public void setPreference(long userID, long itemID, float value) |
添加偏好，线上系统经常需要。 |
动态添加偏好，添加之后会更新ItemID的和其他Item之间的相似度 |
| public void removePreference(long userID, long itemID) |
删除偏好，很少用。 |
删除偏好后，会更新itemId和其他Item之间的相似度 |
| public List<RecommendedItem> recommend(long userID, int howMany, IDRescorer rescorer) |
提供推荐。IDRescorer用于商业规则，调整item的得分 |
1.获取userId还未评分的item作为候选。2.估计每个Item的得分，选取topk 返回。 |
| public float estimatePreference(long userID,long itemID) |
估计userId对ItemId的评分 |
如userId对itemId有真实的值，则返回，否则估计。 |
1. 推荐接口
public List<RecommendedItem> recommend(long userID, int howMany, IDRescorer rescorer) throws TasteException { Preconditions.checkArgument(howMany >= 1, "howMany must be at least 1"); log.debug("Recommending items for user ID '{}'", userID); FastIDSet possibleItemIDs = diffStorage.getRecommendableItemIDs(userID); TopItems.Estimator<Long> estimator = new Estimator(userID); List<RecommendedItem> topItems = TopItems.getTopItems(howMany, possibleItemIDs.iterator(), rescorer, estimator); log.debug("Recommendations are: {}", topItems); return topItems; } |
2. 获取推荐候选项
public FastIDSet getRecommendableItemIDs(long userID) throws TasteException { FastIDSet result; try { buildAverageDiffsLock.readLock().lock(); result = allRecommendableItemIDs.clone(); } finally { buildAverageDiffsLock.readLock().unlock(); } Iterator<Long> it = result.iterator(); while (it.hasNext()) { if (dataModel.getPreferenceValue(userID, it.next()) != null) { it.remove(); } } return result; } |
3. 估计候选项的得分，返回topK个推荐列表
public static List<RecommendedItem> getTopItems(int howMany, LongPrimitiveIterator possibleItemIDs, IDRescorer rescorer, Estimator<Long> estimator) throws TasteException { Preconditions.checkArgument(possibleItemIDs != null, "argument is null"); Preconditions.checkArgument(estimator != null, "argument is null"); Queue<RecommendedItem> topItems = new PriorityQueue<RecommendedItem>(howMany + 1, Collections.reverseOrder(ByValueRecommendedItemComparator.getInstance())); boolean full = false; double lowestTopValue = Double.NEGATIVE_INFINITY; while (possibleItemIDs.hasNext()) { long itemID = possibleItemIDs.next(); if (rescorer == null || !rescorer.isFiltered(itemID)) { double preference; try { preference = estimator.estimate(itemID); } catch (NoSuchItemException nsie) { continue; } double rescoredPref = rescorer == null ? preference : rescorer.rescore(itemID, preference); if (!Double.isNaN(rescoredPref) && (!full || rescoredPref > lowestTopValue)) { topItems.add(new GenericRecommendedItem(itemID, (float) rescoredPref)); if (full) { topItems.poll(); } else if (topItems.size() > howMany) { full = true; topItems.poll(); } lowestTopValue = topItems.peek().getValue(); } } } int size = topItems.size(); if (size == 0) { return Collections.emptyList(); } List<RecommendedItem> result = Lists.newArrayListWithCapacity(size); result.addAll(topItems); Collections.sort(result, ByValueRecommendedItemComparator.getInstance()); return result; } |
三、 MapReduce实现(计算diff)
1. 计算每个user的item之间的差值
Map: 输入，文本文件，格式为:userId\t itemId\t val 输出：key userId,value itemId\t val |
Reduce: for(user u :users){ items of u for(int I =0 ;i<items.size;i++){ itema =items[i]; for(int j =i+1;j<items.size;j++){ itemb= items[j]; itemABdiff =itemb-itema; out.write(itemA\t itemb, itemABdiff); } } } |
2. 计算itemPair的全局平均
Map:输出数据不做处理,将item相同的数据传递到同一个reduce中。 |
Reduce: 输入 key itemA\t itemb ,val itemABdiff 计算改组数据的平均值（FullRunningAverageAndStdDev） 输出： key EntityEntityWritable ,valueFullRunningAverageAndStdDevWritable |
四、 实例演示
(一) 单机模式
MemoryDiffStorage mds =new MemoryDiffStorage(new FileDataModel(new File("pereference")), Weighting.WEIGHTED, 1000); SlopeOneRecommender sr =new SlopeOneRecommender(new FileDataModel(new File("pereference")),Weighting.WEIGHTED,Weighting.WEIGHTED,mds); System.out.println(sr.recommend(1, 10,new IDRescorer() { @Override public double rescore(long id, double originalScore) { int clickCount =10;//id的点击量 return originalScore*clickCount; } @Override public boolean isFiltered(long id) { //如果id和要推荐的item的id属于同一个类型，return false ,否则return true ; return false; } })); |
(二) MapReduce模式
String [] arg ={"-i","p","-o","diff"}; SlopeOneAverageDiffsJob.main(arg); DiffStorage ds =new FileDiffStorage(new File("diff"), 1000); SlopeOneRecommender sr =new SlopeOneRecommender(new FileDataModel(new File("pereference")),Weighting.WEIGHTED,Weighting.WEIGHTED,mds); System.out.println(sr.recommend(1, 10,new IDRescorer() { @Override public double rescore(long id, double originalScore) { int clickCount =10;//id的点击量 return originalScore*clickCount; } @Override public boolean isFiltered(long id) { //如果id和要推荐的item的id属于同一个类型，return false ,否则return true ; return false; } })); |
五、 参考文献
1. http://en.wikipedia.org/wiki/Slope_One
2. DanielLemire, Anna Maclachlan, SlopeOne Predictors for Online Rating-Based Collaborative Filtering
3. PuWang, HongWu Ye, A Personalized Recommendation Algorithm Combining Slope OneScheme and User Based Collaborative Filtering
4. DeJiaZhang, An Item-based Collaborative Filtering Recommendation AlgorithmUsing Slope One Scheme Smoothing
5. Mi,Zhenzhen and Xu, Congfu, A Recommendation Algorithm Combining Clustering Methodand Slope One Scheme
1. BadrulM. Sarwar, George Karypis, Joseph A. Konstan, John Riedl: Item-basedcollaborative filtering recommendation algorithms
2. GregLinden, Brent Smith, Jeremy York, "Amazon.com Recommendations:Item-to-Item Collaborative Filterin