Scraping data from webpages with scroller using python

Question

I am interested to scrap all the data from a webpage that when scroll to bottom, you get new data loaded to the page. Here is an example: http://stocktwits.com/symbol/CIEN

As you can see if you scroll down the page, more data will load to the page.

How one can scrap all available data for such a page using Python?

Jonathan Epstein · Accepted Answer · 2015-03-04 15:26:23Z

1

Selenium is recommended for scrolling http://selenium.googlecode.com/svn/trunk/docs/api/py/index.html

answered Mar 4, 2015 at 15:26

Jonathan Epstein

3792 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

TJ1 Over a year ago

Thanks Jonathan, it looks interesting I will look into it.

rufatZZ Over a year ago

Man, can your update your link ? This link doesn't work. i can't see which part of selenium you attached here :/

Jonathan Epstein Over a year ago

selenium-python.readthedocs.io/…

Collectives™ on Stack Overflow

Scraping data from webpages with scroller using python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related