2

I'm currently in the process of converting an old bash script of mine into a Python script with added functionality. I've been able to do most things, but I'm having a lot of trouble with Python pattern matching.

In my previous script, I downloaded a web page and used sed to get the elemented I wanted. The matching was done like so (for one of the values I wanted):

PM_NUMBER=`cat um.htm | LANG=sv_SE.iso88591 sed -n 's/.*ol.st.*pm.*count..\([0-9]*\).*/\1/p'`

It would match the number wrapped in <span class="count"></span> after the phrase "olästa pm". The markup I'm running this against is:

<td style="padding-left: 11px;">
    <a href="/abuse_list.php">
        <img src="/gfx/abuse_unread.png" width="15" height="12" alt="" title="9  anmälningar" />
    </a>
</td>
<td align="center">
    <a class="page_login_text" href="/pm.php" title="Du har 3 olästa pm.">
        <span class="count">3</span>
</td>
<td style="padding-left: 11px;" align="center">
    <a class="page_login_text" href="/blogg_latest.php" title="Du har 1 ny bloggkommentar">
        <span class="count">1</span>
</td>
<td style="padding-left: 11px;" align="center">
    <a class="page_login_text" href="/user_guestbook.php" title="Min gästbok">
        <span class="count">1</span>
</td> 
<td style="padding-left: 11px;" align="center">
    <a class="page_login_text" href="/forum.php?view=3" title="Du har 1 ny forumkommentar">
        <span class="count">1</span>
</td> 
<td style="padding-left: 11px;" align="center">
    <a class="page_login_text" href="/user_images.php?user_id=162005&func=display_new_comments" title="Du har 1 ny albumkommentar">
        <span class="count">1</span>
</td> 
<td style="padding-left: 11px;" align="center">
    <a class="page_login_text" href="/forum_favorites.php" title="Du har 2 uppdaterade trådar i &quot;bevakade trådar&quot;">
        <span class="count">2</span>
</td>

I'm hesitant to post this, because it seems like I'm asking for a lot, but could someone please help me with a way to parse this in Python? I've been pulling my hair trying to do this, but regular expressions and I just don't match (pardon the pun). I've spent the last couple of hours experimenting and reading the Python manual on regular expressions, but I can't seem to figure it out.

Just to make it clear, what I need are 7 different expressions for matching the number within <span class="count"></span>. I need to, for example, be able to find the number of unread PMs ("olästa pm").

Improve this question
2
  • 4
    regular expressions and html does not match (pardon the pun) Commented Nov 19, 2010 at 17:27
  • Hello people from the future, maybe you're looking for this? python.org/dev/peps/pep-0636 Commented Jan 9, 2022 at 22:37

4 Answers 4

Reset to default
4

You will not parse html yourself. You will use a html parser built in python to parse the html.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

You will not use an XML parser for HTML. You will use a parser that doesn't break on Javascript (i.e. BeautifulSoup).
I have looked at beautiful soup as a potential solution,but I haven't been able to figure out how to get what I'm looking for yet. I'll keep trying though.
Turns out BeautifulSoup may not be an option, because of a year old regression that basically causes it to throw a hissy fit whenever it encounters certain malformed tags. The html I need it to parse is pretty horrible, which causes it to choke up.
Then you have a problem that needs resolved first. Irregular/malformed html is not going to parse any easier through a regex.
2

You can user lxml to pull out the values you are looking for pretty easily with xpaths

Example

from lxml import html
page = html.fromstring(open("um.htm", "r").read())
matches = page.xpath("//a[contains(@title, 'pm.') or contains(@title, 'ol')]/span")
print [elem.text for elem in matches]
Improve this answer

3 Comments

This seems promising. It does fetch the PMs as it should. However, I'm having trouble adapting it to fetch the other values. I figured it would be as simple as replacing 'pm' with whatever the other values are (like "bloggkommentar" for blog comments, etc.). However, that doesn't seem to return any matches (once I got rid of or contains(@title, 'ol'), since that always returned the pm value (because it was the first)).
Marking this as correct. Using xpath and matching using the link urls worked beautifully! Thank you very much!
Glad to help, xpath and lxml makes for powerful xml processing
1

use either:

parsing HTML with regexes is a recipe for disaster.

Improve this answer

Comments

1

It is impossible to reliably match HTML using regular expressions. It is usually possible to cobble something together that works for a specific page, but it is not advisable as even a subtle tweak to the source HTML can render all your work useless. HTML simply has a more complex structure than Regex is capable of describing.

The proper solution is to use a dedicated HTML parser. Note that even XML parsers won't do what you need, not reliably anyway. Valid XHTML is valid XML, but even valid HTML is not, even though it's quite similar. And valid HTML/XHTML is nearly impossible to find in the wild anyway.

There are a few different HTML parsers available:

  • BeautifulSoup is not in the standard library, but it is the most forgiving parser, it can handle almost all real-world HTML and it's designed to do exactly what you're trying to do.
  • HTMLParser is included in the Python standard library, but it is fairly strict about accepting only valid HTML.
  • htmllib is also in the standard library, but is deprecated.

As other people have suggested, BeautifulSoup is almost certainly your best choice.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.