Web scraping with python in javascript dynamic website

Question

I need to scrape all article, title of article and paragraf in this web: https://portaljuridic.gencat.cat/eli/es-ct/l/2014/12/29/19

The problem is than I tried some of div, h3 or p nothing happen add image.

from bs4 import BeautifulSoup
import lxml
import pandas as pd
from tqdm import tqdm_notebook


def parse_url(url):
    response = requests.get(url)
    content = response.content
    parsed_response = BeautifulSoup(content, "lxml")
    return parsed_response


url = "https://portaljuridic.gencat.cat/eli/es-ct/l/2014/12/29/19"

soup = parse_url(url)


article = soup.find("div", {"class":"article-document"})

article

It seems to be a website with javascript, but I don't know how to get it.

balderman · Accepted Answer · 2021-10-11 07:38:37Z

3

The website does 3 API calls in order to get the data.
The code below does the same and get the data.

(In the browser do F12 -> Network -> XHR and see the API calls)

import requests

payload1 = {'language':'ca','documentId':680124}
r1 = requests.post('https://portaldogc.gencat.cat/eadop-rest/api/pjc/getListTraceabilityStandard',data = payload1)
if r1.status_code == 200:
  print(r1.json())

print('------------------')
payload2 = {'documentId':680124,'orderBy':'DESC','language':'ca','traceability':'02'}
r2 = requests.post('https://portaldogc.gencat.cat/eadop-rest/api/pjc/getListValidityByDocument',data = payload2)
if r2.status_code == 200:
  print(r2.json())

print('------------------')

payload3 = {'documentId': 680124,'traceabilityStandard': '02','language': 'ca'}
r3 = requests.post('https://portaldogc.gencat.cat/eadop-rest/api/pjc/documentPJC',data=payload3)
if r3.status_code == 200:
  print(r3.json())

answered Oct 11, 2021 at 7:38

balderman

24k8 gold badges39 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Merinoide Over a year ago

Hi balderman, thanks for your help and explanation. Can I make one question more, I'm sorry I'm really new with this. Some parts of text has special caracters like ' or ` and in the extraction appear &, how can chnage this to specific caracter? Thanks again for the support.

balderman Over a year ago

I am not sure I understand the question. Can you come up with a specific example?

Merinoide Over a year ago

Hi Balderman For exemple when extract first article inside of 'text': '<p align="JUSTIFY">\n\t1. the text start with Aquesta llei té per objecte but in the webside apear 1. Aquesta llei té per objecte: How I can change this to see Aquesta llei té per objecte: instead of Aquesta llei té per objecte. Thanks for your support.

balderman Over a year ago

Well... I have no idea. Sorry.

Merinoide Over a year ago

Hi balderman Oks, well I will look to sse what I find Really thanks for your help and suport!!

Collectives™ on Stack Overflow

Web scraping with python in javascript dynamic website

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related