How to find links and modify an Html using BeautifulSoup in Python

Question

Starting from an Html input like this:

<p>
<a href="http://www.foo.com">this if foo</a>
<a href="http://www.bar.com">this if bar</a>
</p>

using BeautifulSoup, i would like to change this Html in:

<p>
<a href="http://www.foo.com">this if foo[1]</a>
<a href="http://www.bar.com">this if bar[2]</a>
</p>

saving parsed links in a dictionary with a result like this:

links_dict = {"1":"http://www.foo.com","2":"http://www.bar.com"}

Is it possible to do this using BeautifulSoup? Any valid alternative?

danben · Accepted Answer · 2010-05-25 18:28:53Z

4

This should be easy in Beautiful Soup.

Something like:

from BeautifulSoup import BeautifulSoup
from BeautifulSoup import Tag

count = 1
links_dict = {}
soup = BeautifulSoup(text)
for link_tag in soup.findAll('a'):
  if link_tag['href'] and len(link_tag['href']) > 0:
    links_dict[count]  = link_tag['href']  
    newTag = Tag(soup, "a", link_tag.attrs)
    newTag.insert(0, ''.join([''.join(link_tag.contents), "[%s]" % str(count)]))
    link_tag.replaceWith(newTag)
    count += 1

Result of executing this on your text:

>>> soup
<p>
  <a href="http://www.foo.com">this if foo[1]</a>
  <a href="http://www.bar.com">this if bar[2]</a>
</p>

>>> links_dict
{1: u'http://www.foo.com', 2: u'http://www.bar.com'}

The only problem I can foresee with this solution is if your link text contains subtags; then you couldn't do ''.join(link_tag.contents); instead you would need to navigate to the rightmost text element.

edited May 25, 2010 at 18:28

answered May 24, 2010 at 21:29

danben

83.8k18 gold badges127 silver badges149 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

systempuntoout Over a year ago

@danben +1 for the effort. Actually this is like the code i made before asking the question. It does not work because you end up with something like <a href="foo.com[1]">this if foo</a> and this is not what i want.

systempuntoout Over a year ago

@danben do you think is it possible to change the node's content without recreating a new tag?

danben Over a year ago

I was not able to do that, and the documentation suggests that there is not. Why is creating a new Tag undesirable?

systempuntoout Over a year ago

@Danben uhm, because i could have other attributes besides href; a rel="nofollow" for example.Please, have a look to this other question stackoverflow.com/questions/2904542/…

systempuntoout Over a year ago

@Danben Ok i found it; i replaced newTag = Tag(soup, "a", [("href", link_tag['href'])]) with newTag = Tag(soup, "a", link_tag.attrs).Thanks!Please update your code.

|

Collectives™ on Stack Overflow

How to find links and modify an Html using BeautifulSoup in Python

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related