List deduplication+Java8-Stream distinct Deduplication based on a certain field in the list

List deduplication + Java8-Stream stream operation List deduplication distinct, and specified field deduplication

Create a new list array:

List list = new ArrayList(); 
list.add(26); 
list.add(39); 
list.add(39); 
list.add(39); 
list.add(39); 
list.add(5); 
list.add(40); 
list.add(39); 
list.add(25); 
System.out.println(list); 

Method 1: Use the new java8 feature stream to deduplicate List [commonly used]

Note: to deduplicate objects, he can only deduplicate all fields in the same object, not for a single deduplication

List newList = list.stream().distinct().collect(Collectors.toList()); 
System.out.println(“java8新特性stream去重:”+newList); 
list.add(39); 

Method 3: Set the set to judge and deduplicate, without disrupting the order

protected final <T> List<T> removeDuplicates(List<T> list) {
    
    
    return new ArrayList<>(new LinkedHashSet<>(list))
}

Java8-Stream stream operation List deduplication distinct, and deduplication of specified fields

Method 1: Java8-Stream stream operation List deduplication distinct, and deduplication of specified fields

The distinct deduplication method of stream is based on the two methods of Object.equals and Object.hashCode to determine whether it is a duplicate.
So we can use this feature to rewrite the two methods of Object.equals and Object.hashCode of pojo to achieve.

1. Rewrite the equals and hashCode methods of the Book class, use the name to determine whether the comparison is the same, and then use the distinct method of the stream to deduplicate

class Book {
    
    
    ...
 
    @Override
    public String toString() {
    
    
        return String.format("(%s,%s,%s)", id, name, DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss").format(createTime.toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime()));
    }
 
    @Override
    public boolean equals(Object o) {
    
    
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        Book book = (Book) o;
        return Objects.equals(name, book.name);
    }
}
 
List<Book> distinctNameBooks1 = books.stream().distinct().collect(Collectors.toList());
System.out.println(distinctNameBooks1);

Summary: By rewriting the equals and hashCode methods and comparing them according to actual needs, you can directly use the distinct method of stream to deduplicate, which is more convenient; sometimes the object class is inconvenient or cannot be modified, such as it has been implemented or the referenced third-party package cannot Modification, this method cannot flexibly deduplicate by field.

2. Through Collectors.toCollection of Collectors.collectingAndThen, use TreeSet to specify fields in the constructor

A field deduplication

List<ProjectInfoVo> vo = jdbcTemplate.query(sql, new BeanPropertyRowMapper<>(ProjectInfoVo.class));
ArrayList<ProjectInfoVo> collect = vo.stream().collect(Collectors.collectingAndThen(
            Collectors.toCollection(() -> new TreeSet<>(
                    Comparator.comparing(p -> p.getProjectId()))), ArrayList::new));

Deduplication of multiple fields

List<ProjectInfoVo> vo = jdbcTemplate.query(sql, new BeanPropertyRowMapper<>(ProjectInfoVo.class));
ArrayList<ProjectInfoVo> collect = vo.stream().collect(Collectors.collectingAndThen(
            Collectors.toCollection(() -> new TreeSet<>(
                    Comparator.comparing(p -> p.getProjectId()+";"+p.getMember()))), ArrayList::new));

Summary:
Using the method provided by stream, the code is very concise, but the disadvantage is that although the deduplication effect is achieved, the order in the list changes, and some scenes need to maintain the order.

3. Custom method Comparator.comparing(p -> p.get***())

Custom method class - distinctByKey

public class StreamUtils {
    
    

 
    public static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    
    
        Map<Object, Boolean> seen = new ConcurrentHashMap<>();
        return t -> seen.putIfAbsent(keyExtractor.apply(t), Boolean.TRUE) == null;
    }
}

single field

List<ProjectInfoVo> acceptances = vo.stream()
	.filter(StreamUtils.distinctByKey(b -> b.getProjectId()))
	.collect(Collectors.toList());

multiple fields

List<ProjectInfoVo> acceptances = vo.stream()
	.filter(StreamUtils.distinctByKey(b -> b.getProjectId()))
	.filter(StreamUtils.distinctByKey(b -> b.getMember()))
	.collect(Collectors.toList());

Summary: By encapsulating and defining a deduplication method, combined with the filter method, it can flexibly deduplicate by field and maintain the order of the original list. The disadvantage is that a HashMap is defined internally, which occupies a certain amount of memory, and there is an additional method definition.

4. Use the filter method of the stream to deduplicate, do not define the deduplication method, and create a HashMap outside

Map<Object, Boolean> map = new HashMap<>();
List<Book> distinctNameBooks4 = books.stream().filter(i -> map.putIfAbsent(i.getName(), Boolean.TRUE) == null).collect(Collectors.toList());
System.out.println(distinctNameBooks4);

Summary: It still cooperates with the filter method to achieve deduplication. There is no separate creation method, and a HashMap is temporarily defined to maintain the order of the original list. The disadvantage is that it takes up a certain amount of memory.