Converting a list to a set changes element order
In Python 3.6, there is another solution for Python 2 and 3:set()
now should keep the order, but
>>> x = [1, 2, 20, 6, 210]
>>> sorted(set(x), key=x.index)
[1, 2, 20, 6, 210]
Remove duplicates and preserve order by below function
def unique(sequence):
seen = set()
return [x for x in sequence if not (x in seen or seen.add(x))]
How to remove duplicates from a list while preserving order in Python
A
set
is an unordered data structure, so it does not preserve the insertion order.This depends on your requirements. If you have an normal list, and want to remove some set of elements while preserving the order of the list, you can do this with a list comprehension:
>>> a = [1, 2, 20, 6, 210] >>> b = set([6, 20, 1]) >>> [x for x in a if x not in b] [2, 210]
If you need a data structure that supports both fast membership tests and preservation of insertion order, you can use the keys of a Python dictionary, which starting from Python 3.7 is guaranteed to preserve the insertion order:
>>> a = dict.fromkeys([1, 2, 20, 6, 210]) >>> b = dict.fromkeys([6, 20, 1]) >>> dict.fromkeys(x for x in a if x not in b) {2: None, 210: None}
b
doesn't really need to be ordered here – you could use aset
as well. Note thata.keys() - b.keys()
returns the set difference as aset
, so it won't preserve the insertion order.In older versions of Python, you can use
collections.OrderedDict
instead:>>> a = collections.OrderedDict.fromkeys([1, 2, 20, 6, 210]) >>> b = collections.OrderedDict.fromkeys([6, 20, 1]) >>> collections.OrderedDict.fromkeys(x for x in a if x not in b) OrderedDict([(2, None), (210, None)])