1. Iterate over the entire list
# removing duplicated from list
# using naive methods
# initializing list
test_list = [1, 3, 5, 6, 3, 5, 6, 1]
print ("The original list is : " + str(test_list))
# using naive method to remove duplicated from list
res = []
for i in test_list:
if i not in res:
res.append(i)
# printing list after removal
print ("The list after removing duplicates : " + str(res))
# 输出结果：
# 原始列表是：[1, 3, 5, 6, 3, 5, 6, 1]
# 删除重复项后的列表：[1, 3, 5, 6]2. Use set()
One of the biggest disadvantages is that the order of the elements in the list after set is no longer the same as before
# removing duplicated from list
# using set()
# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : " + str(test_list))
# using set()to remove duplicated from list
test_list = list(set(test_list))
# printing list after removal
# distorted ordering
print ("The list after removing duplicates : " + str(test_list))
# 输出结果：
# 原始列表是：[1, 5, 3, 6, 3, 5, 6, 1]
# 删除重复项后的列表：[1, 3, 5, 6]3. Using list comprehension + enumerate()
This method removes duplicate elements based on a list comprehension using an enum. Skips an element by checking if it already exists in the list. This method preserves the order of the elements in the list.
# removing duplicated from list
# using list comprehension + enumerate()
# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : " + str(test_list))
# using list comprehension + enumerate()
# to remove duplicated from list
res = [i for n, i in enumerate(test_list) if i not in test_list[:n]]
# printing list after removal
print ("The list after removing duplicates : " + str(res))4. Use collections.OrderedDict.fromkeys()
This is the fastest way to accomplish your particular task, it first removes duplicates from a list and returns a dictionary, and finally converts it to a list. This method also works with strings, after which the order of the elements in the list is also changed.
# removing duplicated from list
# using collections.OrderedDict.fromkeys()
from collections import OrderedDict
# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : " + str(test_list))
# using collections.OrderedDict.fromkeys()
# to remove duplicated from list
res = list(OrderedDict.fromkeys(test_list))
# printing list after removal
print ("The list after removing duplicates : " + str(res))5. Handling duplicate elements in nested lists
Used for multidimensional list (list nesting) duplicate element removal. This assumes that elements in a list (which is also a list) with the same elements (but not necessarily in the same order) are considered duplicates.
Use the set() + sorted() method:
# removing duplicate sublist
# using set() + sorted()
# initializing list
test_list = [[1, 0, -1], [-1, 0, 1], [-1, 0, 1],
[1, 2, 3], [3, 4, 1]]
# printing original list
print("The original list : " + str(test_list))
# using set() + sorted()
# removing duplicate sublist
res = list(set(tuple(sorted(sub)) for sub in test_list))
# print result
print("The list after duplicate removal : " + str(res))
# 输出结果：
# 原始列表：[[1, 0, -1], [-1, 0, 1], [-1, 0, 1], [1, 2, 3], [3, 4, 1]]
# 去重后的列表：[(-1, 0, 1), (1, 3, 4), (1, 2, 3)]Use the set() + map() + sorted() method:
# removing duplicate sublist
# using set() + map() + sorted()
# initializing list
test_list = [[1, 0, -1], [-1, 0, 1], [-1, 0, 1],
[1, 2, 3], [3, 4, 1]]
# printing original list
print("The original list : " + str(test_list))
# using set() + map() + sorted()
# removing duplicate sublist
res = list(set(map(lambda i: tuple(sorted(i)), test_list)))
# print result
print("The list after duplicate removal : " + str(res))
# 输出结果：
# 原始列表：[[1, 0, -1], [-1, 0, 1], [-1, 0, 1], [1, 2, 3], [3, 4, 1]]
# 去重后的列表：[(-1, 0, 1), (1, 3, 4), (1, 2, 3)]Reference：webqdkf.com