python loop for web-scraping

Question

I'am trying to create a loop to show all values within the li tags to create a DataFrame. Moreover, I can only isolate the code using: new = soup.find("div", class_="PlayerList"). If I use a standard for loop it only shows one value not all values.

The output I would like to show is:

Messi,

Shooting 9,

Passing 9,

Tackle 4,

   <pre>
   import requests
   import pandas as pd
   import numpy as np

   from urllib.request import urlopen
   from bs4 import BeautifulSoup

   main_url = 'https://examplelistpython.000webhostapp.com/messi.html'
   result = requests.get(main_url)
   result.text

   soup = BeautifulSoup(result.text, 'html.parser')
   print(soup.prettify())

   new  = soup.find("div", class_="PlayerList")
   new

    </pre>

<ul class="List">
 <li>
  <div class="PlayerList">
   <div class="HeaderList">
    <span class="player">Messi</span>
</div>
  <div class="PlayerStat">
   <span class="stat">Shooting   <span class="allStatContainer statShooting" data-stat="Shooting">
     9
  </span>
 </span>
</div>
<div class="PlayerStat">
<span class="stat">Passing   <span class="allStatContainer statPassing" data-stat="Passing">
  9
 </span>
</span>
</div>
<div class="PlayerStat">
<span class="stat">Tackle   <span class="allStatContainer statTackle" data-stat="Tackle">
     4
     </span>
   </span>
  </div>
 </li>
</ul>

It that website working? It is showing error while using get() of requests — Prakash Dahal
– Prakash Dahal, Commented Mar 9, 2021 at 19:03

RJ Adriaansen · Accepted Answer · 2021-03-09 19:08:04Z

1

player = [i.text.strip() for i in soup.find_all("span", class_="player")]
shooting = [i.text.strip() for i in soup.find_all("span", class_="allStatContainer statShooting")]
passing = [i.text.strip() for i in soup.find_all("span", class_="allStatContainer statPassing")] 
tackle = [i.text.strip() for i in soup.find_all("span", class_="allStatContainer statTackle")]

df = pd.DataFrame({'Player': player, 'Shooting': shooting, 'Passing': passing, 'Tackle': tackle})

Result:

	Player	Shooting	Passing	Tackle
0	Messi	9	9	4

answered Mar 9, 2021 at 19:08

RJ Adriaansen

9,7092 gold badges16 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python loop for web-scraping

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related