Python for loop not looping through all the elements

Question

I'm trying to code my first webscraper using python and BeautifulSoup.

I'm trying to retrieve all the URLs for all the listings on a webpage but instead of getting an array with all the URLs I only get one URL.

Following is the code I used

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'https://www.pararius.com/apartments/enschede'

uClient = uReq(my_url)
page_html=uClient.read()
uClient.close()

page_soup = soup(page_html,"html.parser")

compartments = page_soup.findAll("li",{"class":"property-list-item-container"})

#Here is where im trying to store all the urls in url_det 
for compartment in compartments:
    url_det = compartment.h2.a["href"]

Any input is appreciated!

marcos · Accepted Answer · 2020-03-14 20:38:49Z

1

Each iteration of the loop will overwrite the content of url_det, use instead a list comprehension to store all the values in a list, for example:

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'https://www.pararius.com/apartments/enschede'

uClient = uReq(my_url)
page_html=uClient.read()
uClient.close()

page_soup = soup(page_html,"html.parser")

compartments = page_soup.findAll("li",{"class":"property-list-item-container"})

url_det = [compartment.h2.a["href"] for compartment in compartments]

print(url_det)
>>> ['/house-for-rent/enschede/PR0001596564/otto-van-taverenstraat', ... , '/house-for-rent/enschede/PR0001594320/hanenberglanden']

answered Mar 14, 2020 at 20:38

marcos

4,5101 gold badge13 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python for loop not looping through all the elements

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related