Python Remove SOME duplicates from a list while maintaining order?

Question

I want to remove certain duplicates in my python list. I know there are ways to remove all duplicates, but I wanted to remove only consecutive duplicates, while maintaining the list order.

For example, I have a list such as the following:

list1 = [a,a,b,b,c,c,f,f,d,d,e,e,f,f,g,g,c,c]

However, I want to remove the duplicates, and maintain order, but still keep the 2 c's and 2 f's, such as this:

wantedList = [a,b,c,f,d,e,f,g,c]

So far, I have this:

z = 0
j=0
list2=[]
for i in list1:
    if i == "c":
        z = z+1
        if (z==1):
            list2.append(i)
        if (z==2):
            list2.append(i)
        else:
            pass
    elif i == "f":
        j = j+1
        if (j==1):
            list2.append(i)
        if (j==2):
            list2.append(i)
        else:
            pass
    else:
        if i not in list2:
            list2.append(i)

However, this method gives me something like:

wantedList = [a,b,c,c,d,e,f,f,g]

Thus, not maintaining the order.

Any ideas would be appreciated! Thanks!

Would it be accurate to say that you want to remove consecutive duplicates? Or is there something special about c and f such that they are treated differently from other elements? — Greg Hewgill
– Greg Hewgill, Commented Jul 23, 2012 at 3:28

Alok Singhal · Accepted Answer · 2012-07-23 03:47:43Z

8

Not completely sure if c and f are special cases, or if you want to compress consecutive duplicates only. If it is the latter, you can use itertools.groupby():

>>> import itertools
>>> list1
['a', 'a', 'b', 'b', 'c', 'c', 'f', 'f', 'd', 'd', 'e', 'e', 'f', 'f', 'g', 'g', 'c', 'c']
>>> [k for k, g in itertools.groupby(list1)]
['a', 'b', 'c', 'f', 'd', 'e', 'f', 'g', 'c']

answered Jul 23, 2012 at 3:47

Alok Singhal

96.8k21 gold badges130 silver badges158 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Greg Hewgill · Accepted Answer · 2012-07-23 03:36:29Z

3

To remove consecutive duplicates from a list, you can use the following generator function:

def remove_consecutive_duplicates(a):
    last = None
    for x in a:
        if x != last:
            yield x
        last = x

With your data, this gives:

>>> list1 = ['a','a','b','b','c','c','f','f','d','d','e','e','f','f','g','g','c','c']
>>> list(remove_consecutive_duplicates(list1))
['a', 'b', 'c', 'f', 'd', 'e', 'f', 'g', 'c']

answered Jul 23, 2012 at 3:36

Greg Hewgill

1.0m192 gold badges1.2k silver badges1.3k bronze badges

Comments

SudoNhim · Accepted Answer · 2012-07-23 03:49:08Z

1

If you want to ignore certain items when removing duplicates...

list2 = []
for item in list1:
    if item not in list2 or item in ('c','f'):
        list2.append(item)

EDIT: Note that this doesn't remove consecutive items

edited Jul 23, 2012 at 3:49

answered Jul 23, 2012 at 3:37

SudoNhim

5804 silver badges17 bronze badges

2 Comments

Greg Hewgill Over a year ago

This doesn't seem to give the desired wantedList value for the sample input list1.

SudoNhim Over a year ago

You are correct! He want's consecutive 'f's and 'c's preserved... question needs to be more specific (and I need to be less hasty)

Chris · Accepted Answer · 2012-07-23 03:47:21Z

0

EDIT Never mind, I read your question wrong. I thought you were wanting to keep only certain sets of doubles.

I would recommend something like this. It allows a general form to keep certain doubles once.

list1 = ['a','a','b','b','c','c','f','f','d','d','e','e','f','f','g','g','c','c']
doubleslist = ['c', 'f']

def remove_duplicate(firstlist, doubles):
    newlist = []
    for x in firstlist:
        if x not in newlist:
            newlist.append(x)
        elif x in doubles:
            newlist.append(x)
            doubles.remove(x)
    return newlist

print remove_duplicate(list1, doubleslist)

edited Jul 23, 2012 at 3:47

answered Jul 23, 2012 at 3:40

Chris

11 bronze badge

Comments

Joe · Accepted Answer · 2012-07-23 04:54:12Z

0

The simple solution is to compare this element to the next or previous element

a=1
b=2
c=3
d=4
e=5
f=6
g=7
list1 = [a,a,b,b,c,c,f,f,d,d,e,e,f,f,g,g,c,c]
output_list=[list1[0]]
for ctr in range(1, len(list1)):
    if list1[ctr] != list1[ctr-1]:
        output_list.append(list1[ctr])
print output_list

answered Jul 23, 2012 at 4:54

Joe

1

Comments

Fuji Komalan · Accepted Answer · 2017-02-18 15:39:38Z

0

list1 = ['a', 'a', 'b', 'b', 'c', 'c', 'f', 'f', 'd', 'd', 'e', 'e', 'f', 'f', 'g', 'g', 'c', 'c']

wantedList = []

for item in list1:   
   if len(wantedList) == 0:
      wantedList.append(item)

   elif len(wantedList) > 0:
      if  wantedList[-1] != item:
          wantedList.append(item)

print(wantedList)

Fetch each item from the main list(list1).
If the 'temp_list' is empty add that item.
If not , check whether the last item in the temp_list is not same as the item we fetched from 'list1'.
if items are different append into temp_list.

answered Feb 18, 2017 at 15:39

Fuji Komalan

2,05519 silver badges26 bronze badges

Collectives™ on Stack Overflow

Python Remove SOME duplicates from a list while maintaining order?

6 Answers 6

Comments

Comments

2 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

Comments

2 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related