Remove duplicates from list python

Question

I'm trying to write a program that removes duplicates from a list, but my program keeps throwing the error "list index out of range" on line 5, if n/(sequence[k]) == 1:. I can't figure this out. Am I right in thinking that the possible values of "k" are 0, 1, and 2? How is "sequence" with any of those as the index outside of the possible index range?

def remove_duplicates(sequence):
    new_list = sequence
    for n in sequence:
        for k in range(len(sequence)):
            if n/(sequence[k]) == 1:
                new_list.remove(sequence[k])
    print new_list

remove_duplicates([1,2,3])

Why not just list(set(your_list))?

Akavall
– Akavall

2015-03-02 03:36:56 +00:00
Commented Mar 2, 2015 at 3:36 — Akavall
– Akavall, Commented Mar 2, 2015 at 3:36
Duplicate of stackoverflow.com/questions/7961363/…

taesu
– taesu

2015-03-02 03:39:01 +00:00
Commented Mar 2, 2015 at 3:39 — taesu
– taesu, Commented Mar 2, 2015 at 3:39

chw21 · Accepted Answer · 2015-03-02 03:50:21Z

6

I strongly suggest Akavall's answer:

list(set(your_list))

As to why you get out of range errors: Python passes by reference, that is sequence and new_list still point at the same memory location. Changing new_list also changes sequence.

And finally, you are comparing items with themselves, and then remove them. So basically even if you used a copy of sequence, like:

new_list = list(sequence)

or

new_list = sequence[:]

It would return an empty list.

answered Mar 2, 2015 at 3:50

chw21

8,1601 gold badge21 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

9769953 Over a year ago

This does not guarantee the same order of the remaining elements.

Others · Accepted Answer · 2015-03-02 03:53:30Z

3

Your error is concurrent modification of the list:

for k in range(len(sequence)):
    if n/(sequence[k]) == 1:
        new_list.remove(sequence[k])

It may seem removing from new_list shouldn't effect sequence, but you did new_list = sequence at the beginning of the function. This means new_list actually literally is sequence, perhaps what you meant is new_list=list(sequence), to copy the list?

If you accept that they are the same list, the error is obvious. When you remove items, the length, and the indexes change.

P.S. As mentioned in a comment by @Akavall, all you need is:

sequence=list(set(sequence))

To make sequence contain no dupes. Another option, if you need to preserve ordering, is:

from collections import OrderedDict
sequence=list(OrderedDict.fromkeys(sequence))

edited Mar 2, 2015 at 3:53

answered Mar 2, 2015 at 3:39

Others

3,0612 gold badges33 silver badges53 bronze badges

4 Comments

Crytera Over a year ago

Oh, thanks. But I still don't understand. Aren't I modifying 'new_list', not 'sequence'?

Others Over a year ago

@Crytera sorry, that was just for example, I've fixed it now if you wanna accept the answer.

Crytera Over a year ago

thanks for the advice, but I wanted more to know exactly why the error was thrown if I'm not modifying 'sequence'.

Others Over a year ago

@Crytera explained your problem

Community · Accepted Answer · 2017-05-23 11:46:24Z

1

If you don't like list(set(your_list)) because it's not guaranteed to preserved order, you could grab the OrderedSet recipe and then do:

from ordered_set import OrderedSet

foo = list("face a dead cabbage")
print foo
print list(set(foo)) # Order might change
print list(OrderedSet(foo)) # Order preserved

edited May 23, 2017 at 11:46

CommunityBot

11 silver badge

answered Mar 2, 2015 at 3:49

kuzzooroo

7,44811 gold badges55 silver badges86 bronze badges

Comments

taesu · Accepted Answer · 2015-03-02 03:53:35Z

0

# like @Akavall suggested
def remove_duplicates(sequence):
    # returns unsorted unique list
    return list(set(sequence))

# create a list, if ele from input not in that list, append.
def remove_duplicates(sequence):
    lst = []
    for i in sequence:
        if i not in lst:
            lst.append(i)
    # returns unsorted unique list
    return lst

edited Mar 2, 2015 at 3:53

answered Mar 2, 2015 at 3:46

taesu

4,5905 gold badges26 silver badges41 bronze badges

Collectives™ on Stack Overflow

Remove duplicates from list python

4 Answers 4

1 Comment

4 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related