Extracting data from Script tags in HTML using Python

Question

I am trying to get the data inside these script tags, but I cant seem to be able to convert into json so i can parse it after I read it. The data im interest in is name,image,sku, and price.

HTML:

<script type="application/ld+json">
        {
          "@context": "http://schema.org/",
          "@type": "Product",
          "name": "Key Pouch",
          "image": "https://us.louisvuitton.com/images/is/image/lv/1/PP_VP_L/louis-vuitton-key-pouch-monogram-gifts-for-women--M62650_PM2_Front view.jpg",
          "description": "The Key Pouch in iconic Monogram canvas is a playful yet practical accessory that can carry coins, cards, folded bills and other small items, in addition to keys. Secured with an LV-engraved zip, it can be hooked onto the D-ring inside most Louis Vuitton bags, or used as a bag or belt charm.",
          "sku": "M62650",
          "brand": {
            "@type": "Thing",
            "name": "LOUIS VUITTON"
          },
          "offers": {
            "@type": "Offer",
            "url" : "https://us.louisvuitton.com/eng-us/products/key-pouch-monogram-000941",
            "priceCurrency": "USD",
            "price": "215.00",
            "availability": "http://schema.org/OutOfStock",
            "seller": {
              "@type": "Organization",
              "name": "LOUIS VUITTON"
            }
          }
        }
</script>

Code:

from bs4 import BeautifulSoup as soup
import requests 
import json

HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)'}

req = Request("https://us.louisvuitton.com/eng-us/products/key-pouch-monogram-000941", headers= HEADERS)
webpage = urlopen(req).read()

page_soup = soup(webpage, "html.parser")
data = json.loads(page_soup.find('script', type='application/ld+json').text)

print(data)

Output

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

any help would be appreciated.

Does this answer your question? 'module' object has no attribute 'loads' while parsing JSON using python — Philip Tzou
– Philip Tzou, Commented Aug 18, 2020 at 4:44

Dan-Dev · Accepted Answer · 2020-08-18 05:51:16Z

1

From the documentation at https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text

As of Beautiful Soup version 4.9.0, when lxml or html.parser are in use, the contents of <script>, <style>, and <template> tags are not considered to be ‘text’, since those tags are not part of the human-visible content of the page.

So use html5lib. A working solution is below:

from bs4 import BeautifulSoup as soup
import requests
import json

HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)'}
req = requests.get("https://us.louisvuitton.com/eng-us/products/key-pouch-monogram-000941", headers= HEADERS)
page_soup = soup(req.text, "html5lib")
data = json.loads(page_soup.find('script', type='application/ld+json').text)
print(data)

Outputs:

{'@context': 'http://schema.org/', '@type': 'Product', 'name': 'Key Pouch', 'image': 'https://us.louisvuitton.com/images/is/image/lv/1/PP_VP_L/louis-vuitton-key-pouch-monogram-gifts-for-women--M62650_PM2_Front view.jpg', 'description': 'The Key Pouch in iconic Monogram canvas is a playful yet practical accessory that can carry coins, cards, folded bills and other small items, in addition to keys. Secured with an LV-engraved zip, it can be hooked onto the D-ring inside most Louis Vuitton bags, or used as a bag or belt charm.', 'sku': 'M62650', 'brand': {'@type': 'Thing', 'name': 'LOUIS VUITTON'}, 'offers': {'@type': 'Offer', 'url': 'https://us.louisvuitton.com/eng-us/products/key-pouch-monogram-000941', 'priceCurrency': 'USD', 'price': '215.00', 'availability': 'http://schema.org/OutOfStock', 'seller': {'@type': 'Organization', 'name': 'LOUIS VUITTON'}}}

answered Aug 18, 2020 at 5:51

Dan-Dev

9,5783 gold badges42 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BigO Over a year ago

Thank you for you answer, unfortunately it returns an error on my end.

bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?

. After some google searches it says to use "html.parser" but that returns the error i had before.

Dan-Dev Over a year ago

See pypi.org/project/html5lib use pip install html5lib or see crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser

Collectives™ on Stack Overflow

Extracting data from Script tags in HTML using Python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related