How to interact with a webpage with python?

Question

Basically, I want to be able to mimic the functions of an internet browser, but without having to actually graphically render it. My ideal scenario would be, for example, a script which opens a connection with a webpage, makes all the necessary calls and interactions, and can list what javascript went off (this is a trivial example, but is the sort of thing I'm looking for). I basically want to be able to "browse" from python while logging various events that happen. The real dream would be able to program in reactions (such as sending a click if there's a box that looks like this or that, for example).

Is there a nice way to do this? Thank you.

Have a look at mechanize.

Sven Marnach
– Sven Marnach

2011-05-06 20:27:07 +00:00
Commented May 6, 2011 at 20:27 — Sven Marnach
– Sven Marnach, Commented May 6, 2011 at 20:27

supersighs · Accepted Answer · 2011-05-06 20:30:37Z

3

You should check out twill. It works on the command line but also has a python api.

answered May 6, 2011 at 20:30

supersighs

9478 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Alex Smith · Accepted Answer · 2011-05-07 04:05:49Z

3

You might want to check out PhantomJS. You script it in JavaScript rather than Python, but it seems to be a nice headless browser based on WebKit.

Another option might be to use Selenium, which has a Python binding available for it too: http://pypi.python.org/pypi/selenium

answered May 7, 2011 at 4:05

Alex Smith

1,53512 silver badges13 bronze badges

Collectives™ on Stack Overflow

How to interact with a webpage with python?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related