一、项目需求

1.    需求链接

https://tianchi.aliyun.com/getStart/information.htm?raceId=231522

2.    需求内容

竞赛题目

在真实的业务场景下，我们往往需要对所有商品的一个子集构建个性化推荐模型。在完成这件任务的过程中，我们不仅需要利用用户在这个商品子集上的行为数据，往往还需要利用更丰富的用户行为数据。定义如下的符号：
U——用户集合
I——商品全集
P——商品子集，P ⊆ I
D——用户对商品全集的行为数据集合
那么我们的目标是利用D来构造U中用户对P中商品的推荐模型。

数据说明

本场比赛提供20000用户的完整行为数据以及百万级的商品信息。竞赛数据包含两个部分。

第一部分是用户在商品全集上的移动端行为数据（D）,表名为tianchi_fresh_comp_train_user_2w，包含如下字段：

字段	字段说明	提取说明
user_id	用户标识	抽样&字段脱敏
item_id	商品标识	字段脱敏
behavior_type	用户对商品的行为类型	包括浏览、收藏、加购物车、购买，对应取值分别是1、2、3、4。
user_geohash	用户位置的空间标识，可以为空	由经纬度通过保密的算法生成
item_category	商品分类标识	字段脱敏
time	行为时间	精确到小时级别

第二个部分是商品子集（P）,表名为tianchi_fresh_comp_train_item_2w，包含如下字段： 

字段	字段说明	提取说明
item_id	商品标识	抽样&字段脱敏
item_ geohash	商品位置的空间标识，可以为空	由经纬度通过保密的算法生成
item_category	商品分类标识	字段脱敏

训练数据包含了抽样出来的一定量用户在一个月时间（11.18~12.18）之内的移动端行为数据（D），评分数据是这些用户在这个一个月之后的一天（12.19）对商品子集（P）的购买数据。参赛者要使用训练数据建立推荐模型，并输出用户在接下来一天对商品子集购买行为的预测结果。 

评分数据格式
具体计算公式如下：参赛者完成用户对商品子集的购买预测之后，需要将结果放入指定格式的数据表（非分区表）中，要求结果表名为：tianchi_mobile_recommendation_predict.csv，且以utf-8格式编码；包含user_id和item_id两列（均为string类型）,要求去除重复。例如：

 

评估指标

比赛采用经典的精确度(precision)、召回率(recall)和F1值作为评估指标。具体计算公式如下：

其中PredictionSet为算法预测的购买数据集合，ReferenceSet为真实的答案购买数据集合。我们以F1值作为最终的唯一评测标准。

二、协同过滤推荐算法原理及实现流程

1.    基于用户的协同过滤推荐算法

基于用户的协同过滤推荐算法通过寻找与目标用户具有相似评分的邻居用户，通过查找邻居用户喜欢的项目，推测目标用户也具有相同的喜好。基于用户的协同过滤推荐算法基本思想是：根据用户-项目评分矩阵查找当前用户的最近邻居，利用最近邻居的评分来预测当前用户对项目的预测值，将评分最高的N个项目推荐给用户，其中的项目可理解为系统处理的商品。其算法流程图如下图1所示。

图1基于用户的协同过滤推荐算法流程

package entity;public class Item {private String itemId;private String itemGeoHash;private String itemCategory;public String getItemId() {return itemId;}public void setItemId(String itemId) {this.itemId = itemId;}public String getItemGeoHash() {return itemGeoHash;}public void setItemGeoHash(String itemGeoHash) {this.itemGeoHash = itemGeoHash;}public String getItemCategory() {return itemCategory;}public void setItemCategory(String itemCategory) {this.itemCategory = itemCategory;}@Overridepublic String toString() {return "item [itemId=" + itemId + ", itemGeoHash=" + itemGeoHash+ ", itemCategory=" + itemCategory + "]";}}

package entity;public class Score implements Comparable<Score> {private String userId;      // 用户标识private String itemId;      // 商品标识private double score;public String getUserId() {return userId;}public void setUserId(String userId) {this.userId = userId;}public String getItemId() {return itemId;}public void setItemId(String itemId) {this.itemId = itemId;}public double getScore() {return score;}public void setScore(double score) {this.score = score;}@Overridepublic String toString() {return "Score [userId=" + userId + ", itemId=" + itemId + ", score="+ score + "]";}@Overridepublic int compareTo(Score o) {if ((this.score - o.score) < 0) {return 1;}else if ((this.score - o.score) > 0) {return -1;}else {return 0;}}}

package entity;public class User implements Comparable<User> {private String userId;      // 用户标识private String itemId;      // 商品标识private int behaviorType;   // 用户对商品的行为类型,可以为空,包括浏览、收藏、加购物车、购买，对应取值分别是1、2、3、4.private String userGeoHash; // 用户位置的空间标识private String itemCategory;// 商品分类标识private String time;        // 行为时间private int count;private double weight;      // 权重public String getUserId() {return userId;}public void setUserId(String userId) {this.userId = userId;}public String getItemId() {return itemId;}public void setItemId(String itemId) {this.itemId = itemId;}public int getBehaviorType() {return behaviorType;}public void setBehaviorType(int behaviorType) {this.behaviorType = behaviorType;}public String getUserGeoHash() {return userGeoHash;}public void setUserGeoHash(String userGeoHash) {this.userGeoHash = userGeoHash;}public String getItemCategory() {return itemCategory;}public void setItemCategory(String itemCategory) {this.itemCategory = itemCategory;}public String getTime() {return time;}public void setTime(String time) {this.time = time;}@Overridepublic String toString() {return "User [userId=" + userId + ", itemId=" + itemId+ ", behaviorType=" + behaviorType + ", count=" + count + "]";}public int getCount() {return count;}public void setCount(int count) {this.count = count;}public double getWeight() {return weight;}public void setWeight(double weight) {this.weight = weight;}@Overridepublic int compareTo(User o) {return (int)((-1) * (this.weight - o.weight));}}

2.    工具类

文件处理工具:

package util;import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.PrintStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Set;import entity.Item;
import entity.Score;
import entity.User;public class FileTool {public static FileReader fr=null;public static BufferedReader br=null;public static String line=null;public static FileOutputStream fos1 = null,fos2 = null,fos3 = null;public static PrintStream ps1 = null,ps2 = null,ps3 = null;public static int count = 0;/** * 初始化写文件器(单一指针)* */public static void initWriter1(String writePath) {try {fos1 = new FileOutputStream(writePath);ps1 = new PrintStream(fos1);} catch (FileNotFoundException e) {e.printStackTrace();}}/** * 关闭文件器(单一指针)* */public static void closeRedaer() {try {br.close();fr.close();} catch (IOException e) {e.printStackTrace();}}/** * 关闭文件器(单一指针)* */public static void closeWriter1() {try {ps1.close();fos1.close();} catch (IOException e) {e.printStackTrace();}}/** * 初始化写文件器(双指针)* */public static void initWriter2(String writePath1,String writePath2) {try {fos1 = new FileOutputStream(writePath1);ps1 = new PrintStream(fos1);fos2 = new FileOutputStream(writePath2);ps2 = new PrintStream(fos2);} catch (FileNotFoundException e) {e.printStackTrace();}}/** * 关闭文件器(双指针)* */public static void closeWriter2() {try {ps1.close();fos1.close();ps2.close();fos2.close();} catch (IOException e) {e.printStackTrace();}}/** * 初始化写文件器(三指针)* */public static void initWriter3(String writePath1,String writePath2,String writePath3) {try {fos1 = new FileOutputStream(writePath1);ps1 = new PrintStream(fos1);fos2 = new FileOutputStream(writePath2);ps2 = new PrintStream(fos2);fos3 = new FileOutputStream(writePath3);ps3 = new PrintStream(fos3);} catch (FileNotFoundException e) {e.printStackTrace();}}/** * 关闭文件器(三指针)* */public static void closeWriter3() {try {ps1.close();fos1.close();ps2.close();fos2.close();ps3.close();fos3.close();} catch (IOException e) {e.printStackTrace();}}public static List readFileOne(String path,boolean isTitle,String token,String pattern) throws Exception {List<Object> ret = new ArrayList<Object>();fr = new FileReader(path);br = new BufferedReader(fr);int count = 0,i = 0;if (isTitle) {line = br.readLine();count++;}while((line = br.readLine()) != null){String[] strArr = line.split(token);switch (pattern) {case "item":ret.add(ParseTool.parseItem(strArr));break;case "user":ret.add(ParseTool.parseUser(strArr));break;case "score":ret.add(ParseTool.parseScore(strArr));default:ret.add(line);break;}count++;if (count/100000 == 1) {i++;System.out.println(100000*i);count = 0;}}closeRedaer();return ret;}public static void makeSampleData(String inputPath,boolean isTitle,String outputPath,int threshold) throws Exception {fr = new FileReader(inputPath);br = new BufferedReader(fr);initWriter1(outputPath);if (isTitle) {line = br.readLine();}int count = 0;while((line = br.readLine()) != null){ps1.println(line);count++;if (count == threshold) {break;}}closeRedaer();}public static List<String> traverseFolder(String dir) {File file = new File(dir);String[] fileList = null;if (file.exists()) {fileList = file.list();}List<String> list = new ArrayList<String>();for(String path : fileList){list.add(path);}return list;}public static Map<String, List<Score>> loadScoreMap(String path,boolean isTitle,String token) throws Exception {fr = new FileReader(path);br = new BufferedReader(fr);if (isTitle) {line = br.readLine();}Map<String, List<Score>> scoreMap = new HashMap<String, List<Score>>();while((line = br.readLine()) != null){String[] arr = line.split(token);Score score = ParseTool.parseScore(arr);List<Score> temp = new ArrayList<Score>();if (scoreMap.containsKey(score.getUserId())) {temp = scoreMap.get(score.getUserId());}temp.add(score);scoreMap.put(score.getUserId(), temp);}closeRedaer();return scoreMap;}public static Map<String, List<String>> loadPredictData(String path,boolean isTitle,String token) throws Exception {fr = new FileReader(path);br = new BufferedReader(fr);if (isTitle) {line = br.readLine();}Map<String, List<String>> map = new HashMap<String, List<String>>();while((line = br.readLine()) != null){String[] arr = line.split(token);String userId = arr[0];String itemId = arr[1];List<String> temp = new ArrayList<String>();if (map.containsKey(userId)) {temp = map.get(userId);}temp.add(itemId);map.put(userId, temp);count++;}closeRedaer();return map;}public static Map<String, List<String>> loadTestData(Map<String, List<String>> predictMap, String dir, boolean isTitle, String token) throws Exception {List<String> fileList = traverseFolder(dir);Set<String> predictKeySet = predictMap.keySet();Map<String, List<String>> testMap = new HashMap<String, List<String>>();for(String predictKey : predictKeySet){if (fileList.contains(predictKey)) {List<String> itemList = loadTestData(dir + predictKey, isTitle, token);testMap.put(predictKey, itemList);}}return testMap;}public static List<String> loadTestData(String path, boolean isTitle, String token) throws Exception {fr = new FileReader(path);br = new BufferedReader(fr);if (isTitle) {line = br.readLine();}List<String> list = new ArrayList<String>();Set<String> set = new HashSet<String>();while((line = br.readLine()) != null){String[] arr = line.split(token);set.add(arr[1]);count++;}closeRedaer();for(String item : set){list.add(item);}return list;}public static Map<String, Double> loadUser_ItemData(String path,boolean isTitle,String token) throws Exception {fr = new FileReader(path);br = new BufferedReader(fr);if (isTitle) {line = br.readLine();}Map<String, Double> map = new HashMap<String, Double>();while((line = br.readLine()) != null){String[] arr = line.split(token);String itemId = arr[1];double score = Double.valueOf(arr[2]);if(map.containsKey(itemId)){double temp = map.get(itemId);if (temp > score) {score = temp;}}map.put(itemId, score);}closeRedaer();return map;}public static Map<String, Set<String>> loadTestUser(String path,boolean isTitle,String token) throws Exception {fr = new FileReader(path);br = new BufferedReader(fr);int count = 0,i = 0;if (isTitle) {line = br.readLine();count++;}Map<String, Set<String>> map = new HashMap<String, Set<String>>();while((line = br.readLine()) != null){String[] arr = line.split(token);String userId = arr[0];String itemId = arr[1];Set<String> set = new HashSet<String>();if (map.containsKey(userId)) {set = map.get(userId);set.add(itemId);}map.put(userId, set);count++;if (count/100000 == 1) {i++;System.out.println(100000*i);count = 0;}}closeRedaer();return map;}}

解析工具:

package util;import entity.Item;
import entity.Score;
import entity.User;public class ParseTool {public static boolean isNumber(String str) {int i,n;n = str.length();for(i = 0;i < n;i++){if (!Character.isDigit(str.charAt(i))) {return false;}}return true;}public static Item parseItem(String[] contents) {Item item = new Item();if (contents[0] != null && !contents[0].isEmpty()) {item.setItemId(contents[0].trim());}if (contents[1] != null && !contents[1].isEmpty()) {item.setItemGeoHash(contents[1].trim());}if (contents[2] != null && !contents[2].isEmpty()) {item.setItemCategory(contents[2].trim());}return item;}public static User parseUser(String[] contents) {User user = new User();int n = contents.length;if (contents[0] != null && !contents[0].isEmpty()) {user.setUserId(contents[0].trim());}if (contents[1] != null && !contents[1].isEmpty()) {user.setItemId(contents[1].trim());}/*// 2.调用CountFileTest需放开，其它需注释if (contents[2] != null && !contents[2].isEmpty()) {user.setBehaviorType(Integer.valueOf(contents[2].trim()));}// 2.调用CountFileTest需放开，其它需注释if (contents[n-1] != null && !contents[n-1].isEmpty()) {user.setCount(Integer.valueOf(contents[n-1].trim()));}*/// 3.调用PredictTest需放开，其它需注释if (contents[n-1] != null && !contents[n-1].isEmpty()) {user.setWeight(Double.valueOf(contents[n-1].trim()));}/*// 1.调用SpliteFileAndMakeScoreTable需放开，其它需注释if (contents[3] != null && !contents[3].isEmpty()) {user.setUserGeoHash(contents[3].trim());}if (contents[4] != null && !contents[4].isEmpty()) {user.setItemCategory(contents[4].trim());}if (contents[5] != null && !contents[5].isEmpty()) {user.setTime(contents[5].trim());}*/return user;}public static Score parseScore(String[] contents) {Score score = new Score();if (contents[0] != null && !contents[0].isEmpty()) {score.setUserId(contents[0].trim());}if (contents[1] != null && !contents[1].isEmpty()) {score.setItemId(contents[1].trim());}if (contents[2] != null && !contents[2].isEmpty()) {score.setScore(Double.parseDouble(contents[2].trim()));}return score;}
}

3.    数据处理模块:

package service;import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.Map.Entry;import util.FileTool;
import entity.Item;
import entity.Score;
import entity.User;public class DataProcess {public static final double[] w = {0,10,20,30}; public static void output(Map<String, Map<String, List<User>>> userMap,String outputPath) {for(Entry<String, Map<String, List<User>>> entry : userMap.entrySet()){FileTool.initWriter1(outputPath + entry.getKey());Map<String, List<User>> temp = entry.getValue();for(Entry<String, List<User>> tempEntry : temp.entrySet()){List<User> users = tempEntry.getValue();int count = users.size();for(User user : users){FileTool.ps1.print(user.getUserId() + "\t");FileTool.ps1.print(user.getItemId() + "\t");FileTool.ps1.print(user.getBehaviorType() + "\t");//FileTool.ps1.print(user.getUserGeoHash() + "\t");//FileTool.ps1.print(user.getItemCategory() + "\t");//FileTool.ps1.print(user.getTime() + "\t");FileTool.ps1.print(count + "\n");}}}FileTool.closeWriter1();}public static void output(Map<String, Map<String, Double>> scoreTable, String outputPath, Set<String> userSet, Set<String> itemSet, String token) {FileTool.initWriter1(outputPath);for(String itemId: itemSet){FileTool.ps1.print(token + itemId);}FileTool.ps1.println();for(String userId : userSet){FileTool.ps1.print(userId + token);Map<String, Double> itemMap = scoreTable.get(userId);for(String itemId: itemSet){if(itemMap.containsKey(itemId)){FileTool.ps1.print(itemMap.get(itemId));}else {//FileTool.ps1.print(0);}FileTool.ps1.print(token);}FileTool.ps1.print("\n");}}public static void outputUser(List<User> userList) {for(User user : userList){FileTool.ps1.println(user.getUserId() + "\t" + user.getItemId() + "\t" + user.getWeight());}}public static void outputScore(List<Score> scoreList) {for(Score score : scoreList){FileTool.ps1.println(score.getUserId() + "\t" + score.getItemId() + "\t" + score.getScore());}}public static void outputRecommendList(Map<String, Set<String>> map) {for(Entry<String, Set<String>> entry : map.entrySet()){String userId = entry.getKey();Set<String> itemSet = entry.getValue();for(String itemId : itemSet){FileTool.ps1.println(userId + "," + itemId);}}}public static void output(Map<String, Set<String>> map) {for(Entry<String, Set<String>> entry : map.entrySet()){String userId = entry.getKey();Set<String> set = entry.getValue();for(String itemId : set){FileTool.ps1.println(userId + "\t" + itemId);}}}public static Map<String, Map<String, List<User>>> mapByUser(List<User> userList,Set<String> userSet,Set<String> itemSet) {Map<String, Map<String, List<User>>> userMap = new HashMap<>();for(User user: userList){Map<String, List<User>> tempMap = new HashMap<String, List<User>>();List<User> tempList = new ArrayList<User>();if (!userMap.containsKey(user.getUserId())) {}else {tempMap = userMap.get(user.getUserId());if (!tempMap.containsKey(user.getItemId())) {}else {tempList = tempMap.get(user.getItemId());}}tempList.add(user);tempMap.put(user.getItemId(), tempList);userMap.put(user.getUserId(), tempMap);userSet.add(user.getUserId());itemSet.add(user.getItemId());}return userMap;}public static Map<String, Map<String, Double>> makeScoreTable(Map<String, Map<String, List<User>>> userMap) {Map<String, Map<String, Double>> scoreTable = new HashMap<String, Map<String,Double>>();for(Entry<String, Map<String, List<User>>> userEntry : userMap.entrySet()){Map<String, List<User>> itemMap = userEntry.getValue();String userId = userEntry.getKey();Map<String, Double> itemScoreMap = new HashMap<String, Double>();for(Entry<String, List<User>> itemEntry : itemMap.entrySet()){String itemId = itemEntry.getKey();List<User> users = itemEntry.getValue();double weight = 0.0;int maxType = 0;for(User user : users){if (user.getBehaviorType() > maxType) {maxType = user.getBehaviorType();}}int count = users.size();if (maxType != 0) {weight += w[maxType-1];}weight += count;itemScoreMap.put(itemId, weight);}scoreTable.put(userId, itemScoreMap);}return scoreTable;}public static double calculateWeight(int behaviorType, int count) {double weight = w[behaviorType-1] + count;return weight;}public static List<User> reduceUserByItem(List<User> userList) {List<User> list = new ArrayList<User>();Map<String, User> userMap = new LinkedHashMap<String, User>();for(User user : userList){String itemId = user.getItemId();if (!userMap.containsKey(itemId)) {double weight = calculateWeight(user.getBehaviorType(), user.getCount());user.setWeight(weight);userMap.put(itemId, user);list.add(user);}else {User temp = userMap.get(itemId);if (temp.getBehaviorType() < user.getBehaviorType()) {double weight = calculateWeight(user.getBehaviorType(), user.getCount());user.setWeight(weight);userMap.put(itemId, user);list.add(user);}}}userMap.clear();return list;}public static void sortScoreMap(Map<String, List<Score>> scoreMap) {Set<String> userSet = scoreMap.keySet();for(String userId : userSet){List<Score> temp = scoreMap.get(userId);Collections.sort(temp);scoreMap.put(userId, temp);}}public static Map<String, Set<String>> predict(Map<String, List<Score>> scoreMap, List<String> fileNameList, String userDir,int topNUser,int topNItem) throws Exception {Map<String, Set<String>> recommendList = new HashMap<String, Set<String>>();for(Entry<String, List<Score>> entry : scoreMap.entrySet()){String userId1 = entry.getKey();List<Score> list = entry.getValue();int countUser = 0;Set<String> predictItemSet = new LinkedHashSet<String>();for(Score score : list){String userId2 = score.getItemId();if(fileNameList.contains(userId2)){List<User> userList = FileTool.readFileOne(userDir + userId2, false, "\t", "user");int countItem = 0;for(User user : userList){predictItemSet.add(user.getItemId());countItem++;if (countItem == topNItem) {break;}}countUser++;}if (countUser == topNUser) {break;}}recommendList.put(userId1, predictItemSet);}return recommendList;}public static void prediction(Map<String, List<String>> predictMap,int predictN, Map<String, List<String>> referenceMap, int refN) {int count = 0;for(Entry<String, List<String>> predictEntity : predictMap.entrySet()){String userId = predictEntity.getKey();if (referenceMap.containsKey(userId)) {List<String> predictList = predictEntity.getValue();for(String itemId : predictList){if (referenceMap.get(userId).contains(itemId)) {count++;}}}}double precision = (1.0 * count / predictN) * 100;double recall = (1.0 * count / refN) * 100;double f1 = (2 * precision * recall)/(precision + recall);System.out.println("precision="+precision+",recall="+recall+",f1="+f1);}}

4.    计算模块

package service;import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;import entity.Score;
import util.FileTool;public class CalculateSimilarity {public static double EuclidDist(Map<String, Double> userMap1,Map<String, Double> userMap2, Set<String> userSet,Set<String> itemSet) {double sum = 0;for (String itemId : itemSet) {double score1 = 0.0;double score2 = 0.0;if (userMap1.containsKey(itemId) && userMap2.containsKey(itemId)) {score1 = userMap1.get(itemId);score2 = userMap2.get(itemId);} else if (userMap1.containsKey(itemId)) {score1 = userMap1.get(itemId);} else if (userMap2.containsKey(itemId)) {score2 = userMap2.get(itemId);}double temp = Math.pow((score1 - score2), 2);sum += temp;}sum = Math.sqrt(sum);return sum;}public static double CosineDist(Map<String, Double> userMap1,Map<String, Double> userMap2, Set<String> userSet,Set<String> itemSet) {double dist = 0;double numerator = 0; // 分子double denominator1 = 0; // 分母double denominator2 = 0; // 分母for (String itemId : itemSet) {double score1 = 0.0;double score2 = 0.0;if (userMap1.containsKey(itemId) && userMap2.containsKey(itemId)) {numerator++;score1 = userMap1.get(itemId);score2 = userMap2.get(itemId);} else if (userMap1.containsKey(itemId)) {score1 = userMap1.get(itemId);} else if (userMap2.containsKey(itemId)) {score2 = userMap2.get(itemId);}denominator1 += Math.pow(score1, 2);denominator2 += Math.pow(score2, 2);}dist = ((1.0 * numerator) / (Math.sqrt(denominator1) * Math.sqrt(denominator2)));return dist;}public static double execute(Map<String, Double> userMap1,Map<String, Double> userMap2,Set<String> userSet,Set<String> itemSet) {double dist = EuclidDist(userMap1, userMap2, userSet, itemSet);double userScore = 1.0 / (1.0 + dist);// double userScore = CosineDist(userMap1, userMap2, userSet, itemSet);return userScore;}public static void execute(String userId,Map<String, Map<String, Double>> scoreTable,Set<String> userSet, Set<String> itemSet) {for (Entry<String, Map<String, Double>> userEntry : scoreTable.entrySet()) {String userId2 = userEntry.getKey();Map<String, Double> userMap2 = userEntry.getValue();double dist = EuclidDist(scoreTable.get(userId), userMap2, userSet, itemSet);double userScore = 1.0 / (1.0 + dist);// double userScore = CosineDist(userMap1, userMap2, userSet, itemSet);FileTool.ps1.println(userId + "\t" + userId2 + "\t" + userScore);}}public static void execute(Map<String, Map<String, Double>> scoreTable,Set<String> userSet, Set<String> itemSet) {List<Score> similarList = new ArrayList<Score>();for (Entry<String, Map<String, Double>> userEntry1 : scoreTable.entrySet()) {String userId = userEntry1.getKey();execute(userId, scoreTable, userSet, itemSet);}}}

5.    脚本

生成userset和itemset:

package script;import java.util.HashSet;
import java.util.List;
import java.util.Set;import entity.User;
import util.FileTool;public class MakeSet {public static void main(String[] args) throws Exception {String inputDir = args[0];String outputDir = args[1];Set<String> userSet = new HashSet<String>();Set<String> itemSet = new HashSet<String>();List<String> pathList = FileTool.traverseFolder(inputDir);for(String path : pathList){String inputPath = inputDir + path;List<User> list = FileTool.readFileOne(inputPath, false, "\t", "user");for(User user : list){userSet.add(user.getUserId());itemSet.add(user.getItemId());}}FileTool.initWriter1(outputDir+"userSet");for(String userId : userSet){FileTool.ps1.println(userId);}FileTool.closeWriter1();FileTool.initWriter1(outputDir+"itemSet");for(String itemId : itemSet){FileTool.ps1.println(itemId);}FileTool.closeWriter1();}}

map文件构建user-item评分矩阵并计算user间的相似度生成user-user的得分表:

package script;import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;import entity.Item;
import entity.Score;
import entity.User;
import service.CalculateSimilarity;
import service.DataProcess;
import util.FileTool;public class SpliteFileAndMakeScoreTable {public static void main(String[] args) throws Exception {//String inputDir = "data/fresh_comp_offline/";//String outputDir = "data/fresh_comp_offline/sample/";//String inputDir = "data/fresh_comp_offline/sample/";//String outputDir = "data/fresh_comp_offline/sample/out/";String inputDir = args[0];String outputDir = args[1];//String userPath = inputDir + "tianchi_fresh_comp_train_user.csv";String userPath = inputDir + args[2];String outputPath = args[3];//String outputPath = outputDir + "user.csv";//FileTool.makeSampleData(userPath, true, outputPath, 10000);//List<Object> itemList = FileTool.readFileOne(itemPath, true, ",", "item");//List<User> userList = FileTool.readFileOne(userPath, false, ",", "user");List<User> userList = FileTool.readFileOne(userPath, false, ",", "user");Set<String> userSet = new HashSet<String>();Set<String> itemSet = new HashSet<String>();Map<String, Map<String, List<User>>> userMap = DataProcess.mapByUser(userList,userSet,itemSet);userList.clear();DataProcess.output(userMap, outputDir);//生成userToItem的打分表Map<String, Map<String, Double>> scoreTable = DataProcess.makeScoreTable(userMap);//DataProcess.output(scoreTable, outputDir + "scoreTable.csv" , userSet, itemSet, ",");userMap.clear();FileTool.initWriter1(outputPath);CalculateSimilarity.execute(scoreTable, userSet, itemSet);FileTool.closeWriter1();}}

reduce文件:

package script;import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;import entity.User;
import service.DataProcess;
import util.FileTool;public class ReduceFileTest {public static void main(String[] args) throws Exception {//String inputDir = "data/fresh_comp_offline/";//String outputDir = "data/fresh_comp_offline/sample/";//String inputDir = "data/fresh_comp_offline/sample/";//String outputDir = "data/fresh_comp_offline/sample/out/";String inputDir = args[0];String outputDir = args[1];//String userPath = inputDir + "tianchi_fresh_comp_train_user.csv";//String itemPath = inputDir + args[2];//String userPath = inputDir + args[3];List<String> pathList = FileTool.traverseFolder(inputDir);for(String path : pathList){List<User> userList = FileTool.readFileOne(inputDir+path, false, "\t", "user");List<User> list = DataProcess.reduceUserByItem(userList);userList.clear();FileTool.initWriter1(outputDir + path);Collections.sort(list);DataProcess.outputUser(list);FileTool.closeWriter1();list.clear();}}
}

为用户进行推荐，生成预测列表:

package script;import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.Map.Entry;import service.DataProcess;
import util.FileTool;
import entity.Score;public class PredictTest {public static void main(String[] args) throws Exception {//String inputDir = "data/fresh_comp_offline/";//String outputDir = "data/fresh_comp_offline/sample/";//String inputDir = "data/fresh_comp_offline/sample/";//String outputDir = "data/fresh_comp_offline/sample/out/";String inputDir = args[0];String outputDir = args[1];//String userPath = inputDir + "tianchi_fresh_comp_train_user.csv";String inputPath = inputDir + args[2];String outputPath = inputDir + args[3];String userDir = args[4];Map<String, List<Score>> scoreMap = FileTool.loadScoreMap(inputPath, false, "\t");DataProcess.sortScoreMap(scoreMap);List<String> fileNameList = FileTool.traverseFolder(userDir);//我选择推荐该user的最相似的5个user的前5个itemMap<String, Set<String>> predictMap = DataProcess.predict(scoreMap, fileNameList, userDir, 5, 5);FileTool.initWriter1(outputPath);DataProcess.outputRecommendList(predictMap);FileTool.closeWriter1();scoreMap.clear();}
}

计算准确率、召回率、F测度值:

package script;import java.util.List;
import java.util.Map;import service.DataProcess;
import util.FileTool;public class MatchTest2 {public static void main(String[] args) throws Exception {String inputDir = args[0];String inputPath1 = inputDir + args[1];String userDir = args[2];Map<String, List<String>> predictMap = FileTool.loadPredictData(inputPath1, false, ",");int predictN = FileTool.count;System.out.println(predictN);FileTool.count = 0;Map<String, List<String>> referenceMap = FileTool.loadTestData(predictMap, userDir, false, "\t");int referenceN = FileTool.count;System.out.println(referenceN);DataProcess.prediction(predictMap, predictN, referenceMap, referenceN);}}

以上为核心代码，大家可以参考项目源代码地址：

http://download.csdn.net/download/u013473512/10141066

https://github.com/Emmitte/recommendSystem