unable Scrape text from website using scrappy

Question

I am new here and I am trying to scrape the nearest station and distance list from this link https://www.onthemarket.com/details/10405122/ I have been stuck here for a day. any help would be apreciated.

I have tried

response.xpath('//div[@class = "tab-content"]/span')

response.xpath('//section//span[@class="poi-name"]')

response.xpath('//section[@class="poi"]/div//text()').extract()

nothing seems to work.

please if you are able to get it please do explain why I failed that would be much apreciated.

You want nth children so you have to use "//" instead "/" for example: //div[@class = "tab-content"]//span — Murat Demir
– Murat Demir, Commented May 13, 2021 at 17:33
Also don't forget to add get() and of the parser response.xpath('//div[@class = "tab-content"]//span').get() if it doesn't work again it is probably a js website. So you have to use selenium instead — Murat Demir
– Murat Demir, Commented May 13, 2021 at 17:34

Serhii Shynkarenko · Accepted Answer · 2021-05-13 19:10:15Z

1

The data is not in the downloaded html:

<ol class="tab-list"></ol><div class="tab-content"></div>

It probably receives the data in another call. Try not hurry up writing the scraper, invest some time to understand how this particular UI works. I would also suggest downloading data via curl or scrapy shell "your_url" (as in this case it will not be downloaded by browser, which renders the page and can trick you like right now).

answered May 13, 2021 at 19:10

Serhii Shynkarenko

7363 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alfin Paul Over a year ago

Thanks, I can't upvote you cause of some restrictions. I wish I could

Collectives™ on Stack Overflow

unable Scrape text from website using scrappy

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related