Scraping table values in python

Question

new to SO and having some difficulty with scraping a table from a website using beautifulsoup.

The source html of the table goes something like this (repeated ad-nauseum for every artist/song/album):

<td class="subject">
    <p title="song">song</p>
    <p class="singer" title="artist | album">artist<span class="bar">|</span>album</p>
</td>

And I'm trying to create an output file with all of that information. The code I'm using is:

with open('output.txt', 'w', encoding='utf-8') as f:
for tr in soup.find_all('tr')[1:]:
    tds = tr.find_all('td')
    f.write("Information: %s" % tds[3].text)

which gets me an output like so:

Information: 
song
singer | album

How do I change this to have it all on one line, and to also separate it properly? Ideally my output should be like this:

Song Title: song
Artist: singer
Album Name: album

briancaffey · Accepted Answer · 2018-01-23 21:05:01Z

2

I think you are just close, you just need to process the results of tds. I would do the following:

from bs4 import BeautifulSoup
b = BeautifulSoup(html, 'lxml')

html = """<td class="subject">
    <p title="song">song</p>
    <p class="singer" title="artist | album">artist<span class="bar">|</span>album</p>
</td>"""

tds = b.find_all('td')
data = tds[0]

t = data.text.split('\n')
song = t[1]
artist_album = t[2].split('|')
artist = artist_album[0]
album = artist_album[1]
print("Song:", song)
print("Artist:", artist)
print("Album:", album)

This should give you:

Song: song
Artist: artist
Album: album

answered Jan 23, 2018 at 21:05

briancaffey

2,61811 gold badges40 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ajax1234 · Accepted Answer · 2018-01-23 21:07:37Z

1

You can use regular expressions with BeautifulSoup:

from bs4 import BeautifulSoup as soup
import re
s = """
<td class="subject">
<p title="song">song</p>
<p class="singer" title="artist | album">artist<span class="bar">|</span>album</p>
 </td>
"""
s = soup(s, 'lxml')

data = [list(filter(None, c))[0] for c in [re.findall('title="song">(.*?)</p>|album">(.*?)<span class="bar">|</span>(.*?)</p>', str(i)) for i in s.find_all('td', {'class':'subject'})][0]]
for i in zip(['Song', 'Artist', 'Album'], data):
   print('{}: {}'.format(*i))

Output:

Song: song
Artist: artist
Album: album

answered Jan 23, 2018 at 21:07

Ajax1234

71.7k9 gold badges67 silver badges110 bronze badges

Collectives™ on Stack Overflow

Scraping table values in python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related