Creating a list of dictionaries results in a list of copies of the same dictionary

Question

I want to get all the iframe from a webpage.

Code:

site = "http://" + url
f = urllib2.urlopen(site)
web_content =  f.read()

soup = BeautifulSoup(web_content)
info = {}
content = []
for iframe in soup.find_all('iframe'):
    info['src'] = iframe.get('src')
    info['height'] = iframe.get('height')
    info['width'] = iframe.get('width')
    content.append(info)
    print(info)       

pprint(content)

result of print(info):

{'src': u'abc.com', 'width': u'0', 'height': u'0'}
{'src': u'xyz.com', 'width': u'0', 'height': u'0'}
{'src': u'http://www.detik.com', 'width': u'1000', 'height': u'600'}

result of pprint(content):

[{'height': u'600', 'src': u'http://www.detik.com', 'width': u'1000'},
{'height': u'600', 'src': u'http://www.detik.com', 'width': u'1000'},
{'height': u'600', 'src': u'http://www.detik.com', 'width': u'1000'}]

Why is the value of the content not right? It's suppose to be the same as the value when I print(info).

wjandrea · Accepted Answer · 2021-10-29 05:35:46Z

70

You are not creating a separate dictionary for each iframe, you just keep modifying the same dictionary over and over, and you keep adding additional references to that dictionary in your list.

Remember, when you do something like content.append(info), you aren't making a copy of the data, you are simply appending a reference to the data.

You need to create a new dictionary for each iframe.

for iframe in soup.find_all('iframe'):
    info = {}
    ...

Even better, you don't need to create an empty dictionary first. Just create it all at once:

for iframe in soup.find_all('iframe'):
    info = {
        "src": iframe.get('src'),
        "height": iframe.get('height'),
        "width": iframe.get('width'),
    }
    content.append(info)

There are other ways to accomplish this, such as iterating over a list of attributes, or using list or dictionary comprehensions, but it's hard to improve upon the clarity of the above code.

edited Oct 29, 2021 at 5:35

wjandrea

33.8k10 gold badges69 silver badges105 bronze badges

answered Jul 15, 2012 at 14:25

Bryan Oakley

389k53 gold badges582 silver badges739 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

fragilewindows · Accepted Answer · 2016-11-18 14:48:28Z

44

You have misunderstood the Python list object. It is similar to a C pointer-array. It does not actually "copy" the object which you append to it. Instead, it just store a "pointer" to that object.

Try the following code:

>>> d={}
>>> dlist=[]
>>> for i in xrange(0,3):
    d['data']=i
    dlist.append(d)
    print(d)

{'data': 0}
{'data': 1}
{'data': 2}
>>> print(dlist)
[{'data': 2}, {'data': 2}, {'data': 2}]

So why is print(dlist) not the same as print(d)?

The following code shows you the reason:

>>> for i in dlist:
    print "the list item point to object:", id(i)

the list item point to object: 47472232
the list item point to object: 47472232
the list item point to object: 47472232

So you can see all the items in the dlist is actually pointing to the same dict object.

The real answer to this question will be to append the "copy" of the target item, by using d.copy().

>>> dlist=[]
>>> for i in xrange(0,3):
    d['data']=i
    dlist.append(d.copy())
    print(d)

{'data': 0}
{'data': 1}
{'data': 2}
>>> print dlist
[{'data': 0}, {'data': 1}, {'data': 2}]

Try the id() trick, you can see the list items actually point to completely different objects.

>>> for i in dlist:
    print "the list item points to object:", id(i)

the list item points to object: 33861576
the list item points to object: 47472520
the list item points to object: 47458120

edited Nov 18, 2016 at 14:48

fragilewindows

1,4101 gold badge16 silver badges26 bronze badges

answered Jul 15, 2012 at 14:37

Wang

8,4265 gold badges51 silver badges81 bronze badges

7 Comments

Bryan Oakley Over a year ago

So... you're advocating that the user uses the .copy() method instead of just creating a new dictionary on every iteration? I think that's wrong in this specific case.

Wang Over a year ago

In many cases, you may only change part of the item, in the aspect of performance and concision, I prefer .copy(). In the aspect of education, .copy() also provides a more clear concept.

Bryan Oakley Over a year ago

I think .copy() only makes sense when you actually want to copy something. While agree that in some cases yo umay want to change only part of an item, in the case of this particular question I think the OP clearly was intending to create a new dictionary for each iframe, rather than copy-and-modify. <shrug>

Wang Over a year ago

Well we are completely speculating here ;) No one really sure what the OP wants. It just reminded me my own work, where .copy() is more suitable. Why not keep this option open? Even this is not what he wants, education-wise, readers can learn something.

l1th1um Over a year ago

Thanks for your explanation. I learn a lot for you're example. May intent is to make dictionary for every iframe, but your explanation on .copy() is useful for me

|

fragilewindows · Accepted Answer · 2016-11-18 14:12:09Z

5

If you want one line:

list_of_dict = [{} for i in range(list_len)]

edited Nov 18, 2016 at 14:12

fragilewindows

1,4101 gold badge16 silver badges26 bronze badges

answered Jun 22, 2015 at 14:54

XU Bin

2513 silver badges6 bronze badges

Comments

zenpoy · Accepted Answer · 2012-07-15 14:23:59Z

4

info is a pointer to a dictionary - you keep adding the same pointer to your list contact.

Insert info = {} into the loop and it should solve the problem:

...
content = []
for iframe in soup.find_all('iframe'):
    info = {}
    info['src'] = iframe.get('src')
    info['height'] = iframe.get('height')
    info['width'] = iframe.get('width')
...

answered Jul 15, 2012 at 14:23

zenpoy

20.3k10 gold badges65 silver badges89 bronze badges

Collectives™ on Stack Overflow

Creating a list of dictionaries results in a list of copies of the same dictionary

4 Answers 4

Comments

7 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

7 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related