Scraping a messy javascript-heavy website with python

Question

I was trying to scrape the household links from this page :

https://www.sreality.cz/en/search/to-rent/apartments?page=2

For instance, for the first apartment I would like to obtain the link with:

https://www.sreality.cz/en/detail/lease/flat/1+kt/plzen-jizni-predmesti-technicka/25873756#img=0&fullscreen=false

However the website is quite heavy on javascript. By using requests.get() I only obtain an uninformative chunk of html code:


from requests import get

i = 2
url = f"https://www.sreality.cz/en/search/to-rent/apartments?page={i}"
response = get(url)

print(response.text)

-----------------------------

<!doctype html>
<html lang="{{ html.lang }}" ng-app="sreality" ng-controller="MainCtrl">
<head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1.0,minimal-ui">

    <!--- Nastaveni meta pres JS a ne pres Angular, aby byla nastavena default hodnota pro agenty co nezvladaji PhantomJS --->
    <title ng:bind-template="{{metaSeo.title}}">Sreality.cz â¢ reality a nemovitosti z celÃ© ÄR</title>
    <meta name="description" content="NejvÄtÅ¡Ã nabÃdka nemovitostÃ v ÄR. NabÃzÃme byty, domy, novostavby, nebytovÃ© prostory, pozemky a dalÅ¡Ã reality k prodeji i pronÃ¡jmu. Sreality.cz">
    <meta property="og:title"       content="Sreality.cz â¢ reality a nemovitosti z celÃ© ÄR">
    <meta property="og:type"        content="website">
    <meta property="og:image"       content="https://www.sreality.cz/img/sreality-logo-og.png">

-----------------------------

ETC ...

The question is therefore, how to proceed with some simple scraping activity for websites of this kind ?

Thanks in advance for the help.

Ask the website if they offer an API to access their info.

KIKO Software
– KIKO Software

2021-03-28 09:48:05 +00:00
Commented Mar 28, 2021 at 9:48 — KIKO Software
– KIKO Software, Commented Mar 28, 2021 at 9:48

pb36 · Accepted Answer · 2021-03-28 12:21:28Z

1

I don't think that website has a public API but looking at the API calls from the network tab I could fetch the details for your need and make it as link have a look at the below code.

Let me know if you have any questions :)

import time
import requests
page=2
numberofresults=20
epochmiliseconds=round(time.time() * 1000)
paramsdict={
    "category_main_cb":1,
    "category_type_cb":2,
    "page":page,
    "per_page":numberofresults,
    "tms":epochmiliseconds

}
data=requests.get("https://www.sreality.cz/api/en/v2/estates",params=paramsdict).json()
for lead in data["_embedded"]["estates"]:
    locality=lead["seo"]["locality"]
    name=lead["name"]
    hash_id=lead["hash_id"]
    typedata=[s for s in name.split(" ") if "+" in s][0].replace("\u00a0"," ").split(" ")[0]
    print(f'https://www.sreality.cz/en/detail/lease/flat/{typedata}/{locality}/{hash_id}'))

Output:

edited Mar 28, 2021 at 12:21

answered Mar 28, 2021 at 11:49

pb36

4201 gold badge5 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

into_r Over a year ago

Thank you very much. It works but I am using this has an exercise to get better at web scrapping. So if you could answer the following questions: 1) how did you found out that you should use 1) sreality.cz/api/en/v2/estates 2) how did you find these were the parameters to use ? Thanks in advance

pb36 Over a year ago

1)before calling website open inspect element and go to network tab and you will find this API calls once the website loads it would be something like this prnt.sc/10yes0v (You need to find the correct request in which data is coming from) or you could use tool like fiddler(this is a better approach) to record traffic and see the specific request from where the data is coming. 2)regarding paramsdict first 4 parameters are basic understandable once and 5th "tms" parameter is nothing but current epoch in milliseconds.

DisappointedByUnaccountableMod · Accepted Answer · 2021-04-22 19:30:58Z

0

First ask website, if they provide any API to get the desired information.

To deal with javascript during the scraping only request will not work. You should go Selenium only or for scrapy in combination of scrapy-selenium. These two allow loading of javascript during scraping.

Feel free to ask if you have any other question.

edited Apr 22, 2021 at 19:30

DisappointedByUnaccountableMod

6,8444 gold badges21 silver badges23 bronze badges

answered Mar 28, 2021 at 9:55

Usama Tariq

1795 bronze badges

Collectives™ on Stack Overflow

Scraping a messy javascript-heavy website with python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related