Several ways to deduplicate java stream

Using the distinct() method of Stream

This method will judge whether it is a duplicate according to the hashCode() and equals() methods of the element. If it is a custom class, you need to override these two methods.

Example:

//利用java8的stream去重
List uniqueList = list.stream()
					.distinct()
					.collect(Collectors.toList());
System.out.println(uniqueList.toString());

Using collectingAndThen() and toCollection() methods

This method can deduplicate based on a certain attribute or multiple attributes of the element, such as name or name+address. This method will use the TreeSet to sort the elements, so the original order cannot be maintained.

Example:

//根据name属性去重
List<User> lt = list.stream().collect(
        collectingAndThen(
                toCollection(() -> new TreeSet<>(Comparator.comparing(User::getName))),
                ArrayList::new)
);
System.out.println("去重后的:" + lt);

Use the filter() method

This method needs to customize a Predicate function, use a Set to record the elements that have appeared, and then filter out duplicate elements.

Example:

//定义一个Predicate函数
private static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    
    
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

//根据age属性去重
list.stream().filter(distinctByKey(s -> s.getAge()))
        .forEach(System.out::println);