7 ways to achieve array deduplication

foreword

Deduplication is a hot issue that is often encountered in development, but the current situation in the project is that the background interface uses SQL to deduplicate, which is simple and efficient, and basically does not allow front-end processing to deduplicate.

So what happens to the front-end processing deduplication? If each page displays 10 different pieces of data, if the data repetition is serious, then to display 10 pieces of data, it may be necessary to send multiple http requests to be able to filter out 10 different pieces of data, and if it is deduplicated in the background, Only one http request can get 10 different pieces of data.

Of course, this does not mean that the front-end de-duplication is unnecessary, and it still needs to be used proficiently. This article mainly introduces several common methods of array deduplication.

method implementation

Double cycle deduplication

The double for (or while) loop is a clumsy way, and it does this very simply: define an array that contains the first element of the original array, then iterate over the original array, comparing each element in the original array with the new array Compare each element of , add it to the new array if it is not repeated, and finally return the new array; because its time complexity is O(n^2), if the length of the array is very large, it will consume a lot of memory

function unique(arr) {
    if (!Array.isArray(arr)) {
        console.log('type error!')
        return
    }
    let res = [arr[0]]
    for (let i = 1; i < arr.length; i++) {
        let flag = true
        for (let j = 0; j < res.length; j++) {
            if (arr[i] === res[j]) {
                flag = false;
                break
            }
        }
        if (flag) {
            res.push(arr[i])
        }
    }
    return res
}

indexOf method to deduplicate 1

The indexOf() method of an array returns the first occurrence of a specified element in the array. This method first defines an empty array res, and then calls the indexOf method to traverse and judge the original array. If the element is not in res, push it into res, and finally return res to get the deduplicated array.

function unique(arr) {
    if (!Array.isArray(arr)) {
        console.log('type error!')
        return
    }
    let res = []
    for (let i = 0; i < arr.length; i++) {
        if (res.indexOf(arr[i]) === -1) {
            res.push(arr[i])
        }
    }
    return res
}

indexOf method to deduplicate 2

Use indexOf to detect whether the first occurrence of the element in the array is equal to the current position of the element, if not, it means that the element is a duplicate element

function unique(arr) {
    if (!Array.isArray(arr)) {
        console.log('type error!')
        return
    }
    return Array.prototype.filter.call(arr, function(item, index){
        return arr.indexOf(item) === index;
    });
}

Deduplication of adjacent elements

This method first calls the sorting method sort() of the array, and then traverses and compares adjacent elements according to the sorted results. If they are equal, skip changing the elements until the end of the traversal.

function unique(arr) {
    if (!Array.isArray(arr)) {
        console.log('type error!')
        return
    }
    arr = arr.sort()
    let res = [arr[0]]
    for (let i = 1; i < arr.length; i++) {
        if (arr[i] !== arr[i-1]) {
            res.push(arr[i])
        }
    }
    return res
}

Use object properties to deduplicate

Create an empty object, traverse the array, set the value in the array as an attribute of the object, and assign an initial value of 1 to the attribute. Each time it occurs, the corresponding attribute value increases by 1. In this way, the attribute value corresponds to the number of times the element appears. span

function unique(arr) {
    if (!Array.isArray(arr)) {
        console.log('type error!')
        return
    }
    let res = [],
        obj = {}
    for (let i = 0; i < arr.length; i++) {
        if (!obj[arr[i]]) {
            res.push(arr[i])
            obj[arr[i]] = 1
        } else {
            obj[arr[i]]++
        }
    }
    return res
}

set and destructuring assignment to remove duplication

The data type set was added in ES6. One of the biggest features of set is that the data is not repeated. The Set function can accept an array (or array-like object) as a parameter to initialize, and this feature can also be used to deduplicate the array

function unique(arr) {
    if (!Array.isArray(arr)) {
        console.log('type error!')
        return
    }
    return [...new Set(arr)]
}

Array.from and set deduplication

The Array.from method can convert the Set structure into an array result, and we know that the set result is a non-repetitive data set, so it can achieve the purpose of deduplication

function unique(arr) {
    if (!Array.isArray(arr)) {
        console.log('type error!')
        return
    }
    return Array.from(new Set(arr))
}

Summarize

Array deduplication is a hot issue that is often encountered in development. We can choose different implementations according to different application scenarios.