Deduplication of list in python

The premise of deduplication in this article is to ensure that the order remains unchanged. This article gives a variety of implementation methods. Friends who need it can refer to the following

1. Intuitive method

The simplest idea is:

ids = [1,2,3,3,4,2,3,4,5,6,1]
news_ids = []
for id in ids:
    if id not in news_ids:
        news_ids.append(id)
print news_ids

2. Use set

ids = [1,4,3,3,4,2,3,4,5,6,1]
ids = list(set(ids))

The result is that the original order is not maintained

ids = [1,4,3,3,4,2,3,4,5,6,1]
news_ids = list(set(ids))
news_ids.sort(ids.index)

3. Use itertools.grouby

ids = [1,4,3,3,4,2,3,4,5,6,1]
ids.sort()
it = itertools.groupby(ids)
for k, g in it:
    print k

The principle of itertools.groupby can be seen here: http://docs.python.org/2/library/itertools.html#itertools.groupby

4. Use reduce

其中的 lambda x,y:x if y in x else x + [y] 等价于 lambda x,y: y in x and x or x+[y] 。

 

The idea is to change the ids to [[], 1,4,3,...] first, and then use the reduce feature. See here for an explanation of reduce: http://docs.python.org/2/library/functions.html#reduce