Manipulate website dropdown menu in Python

Question

I've been searching the web for an answer to the following question, and my lack of success might have to do with my inexperience in web programming/html. If so and there's an obvious answer, sorry for bugging you all. I've been trying to parse a bunch of text from a website, and I found the python script html2text.py, which does a nice job of presenting the website in a way that I can parse. However, in order to get to all the information, I need to access a drop down menu on the site. The drop down menu controls which year of data is displayed, and I want to get all available years. I've seen that with urllib2 I can access a website, and it looks like I can send information to a website that way too, but I haven't been able to figure out exactly how to manipulate the drop down menus. The website I'm interested in is http://www.nfldata.com/nfl-stats/team-stats.aspx if that helps at all. Thanks in advance for any advice you can provide.

-Matt

Thanks for the advice. I learned a lot more about xml/html. I found the select widget that affects which year is displayed, but I still can't quite figure out how to automatically request the website to change that option in python. The statistics I want are dynamically generated when these options are changed. It looks like if I save each webpage (i.e., year) individually I can then extract the information I want from those html files. — Fenix
– Fenix, Commented Feb 20, 2011 at 18:19

jterrace · Accepted Answer · 2011-02-20 03:23:13Z

1

Check out the xml libraries within python. You can take the raw html text and parse it into a Document Object Model (DOM) interface:

http://docs.python.org/library/xml.dom.minidom.html

answered Feb 20, 2011 at 3:23

jterrace

67.5k23 gold badges164 silver badges208 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Matthew Schinckel · Accepted Answer · 2011-02-20 08:08:31Z

1

What you are looking at is not a Drop Down menu, but a Select widget.

You should be able to parse this, and find all of the <option value="X">name<option> entries. The X is what you need.

edited Feb 20, 2011 at 8:08

answered Feb 20, 2011 at 7:59

Matthew Schinckel

35.7k6 gold badges91 silver badges122 bronze badges

Collectives™ on Stack Overflow

Manipulate website dropdown menu in Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related