Skip to main content
Code Review

Questions tagged [web-scraping]

Ask Question

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

609 questions
Newest Active Bountied Unanswered
Filter by
Sorted by
Tagged with
3 votes
1 answer
126 views

I have written a web scraping script using Selenium to crawl blog content from multiple URLs. The script processes URLs in batches of 1000 and uses multithreading with the ThreadPoolExecutor to ...
Minnie's user avatar
  • 31
4 votes
1 answer
105 views

I've been learning some Haskell as an amateur (to be precise: I started programming with this language, and it has been a year or less since I started seriously). So far, I have realised only small ...
user665110's user avatar
  • 141
3 votes
2 answers
100 views

I am writing a crawler that is going to end up in production and I was trying to come up with a way to validate its page visits. It scrapes asp.net pages so each scraping process involves a few ...
Gustavo Costa's user avatar
  • 33
3 votes
1 answer
110 views

Summary: The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
sangharsh's user avatar
  • 269
2 votes
1 answer
80 views

I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
BluffShove's user avatar
  • 21
5 votes
1 answer
220 views

I've wrote a code to scrape the Divar, which is an equivalent of Ebay in Iran. I have a few questions: Am I doing the error handling and logging ok? Is there a better way to optimize this code? (note ...
Amirhossein Rezaei's user avatar
  • 543
4 votes
2 answers
215 views

I added an enum for my webscraper to deserialize data from a JSON field that represents an HTML image size, which can either be an unsigned int like 1080 or a ...
user avatar
2 votes
0 answers
77 views

The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (...
alvas's user avatar
  • 709
3 votes
1 answer
84 views

I'm trying to make my first Python package as a learning experience. There's a lot of things that I suspect I am doing poorly, but this post is specifically about my HttpRequest class. I made this ...
JTB's user avatar
  • 277
4 votes
2 answers
287 views

Made this generator to practice using imports from other modules and better readability for coding. What could I have done better and what did I do wrong? File called test_generator.py ...
Beginner's user avatar
  • 199
3 votes
1 answer
234 views

I have created a simple web scraper that fetches news article previews from nature.com and saves each article to a file containing the article preview text. I am learning independently, so I would ...
razzleDazzle's user avatar
  • 31
2 votes
1 answer
231 views

I'm a newbie in programming, I chose Python. I'm learning on my own. Currently I'm preparing code for a portfolio on github. I will be grateful for any code review, especially in the subject of OOP: ...
Magda's user avatar
  • 29
6 votes
1 answer
865 views

I'm new to web scraping and tried building a web scraper for Amazon customer reviews. The program works fine as is but I wanted to improve the design and get some feedback. The basic idea was to ...
nightstand's user avatar
  • 377
3 votes
2 answers
244 views

When scraping and saving data into a file, Which method is more efficient when saving scraped data to a file? open the file first, scrape, and save the data all ...
Seraph776's user avatar
  • 201
5 votes
1 answer
380 views

I made some C# code to grab youtube thumbnails from urls, I originally made this in python took some time converting it to C#. I am VERY new to C# and did this with minimal help. ...
Joshua Bursill's user avatar
  • 53
2 votes
1 answer
119 views

I recently built my first web scraper in Python and decided to use Stack Overflow (SO) and Stack Exchange (SE) as test websites. Code ...
DialFrost's user avatar
  • 133
1 vote
1 answer
144 views

A task: Сollect data from the site in the following format: book; user; book_rating; comment_rating; publication_date; comment For one book at once several pages of ...
Pom Mop's user avatar
  • 57
3 votes
1 answer
239 views

Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
Salah Zahran's user avatar
  • 33
9 votes
2 answers
2k views

I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback. There is no function but I didn't think it was worth adding. Also there is no test ...
user avatar
1 vote
0 answers
64 views

I am doing web scraping to the next web page (which is my root URL to start scraping tables): https://www.iso.org/standards-catalogue/browse-by-ics.html What I am trying to achieve is to parse the ...
Gescof's user avatar
  • 11
1 vote
1 answer
150 views

I wrote a class on top of BeautifulSoup using the builder design pattern that allows for the navigation of the necp data directory. There are a couple navigation ...
Jason Leaver's user avatar
  • 658
1 vote
0 answers
271 views

I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
Moorgort's user avatar
  • 11
3 votes
0 answers
832 views

You will need to follow this guide. This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
Ξένη Γήινος's user avatar
  • 3,704
3 votes
1 answer
124 views

The following code builds a rudimentary acupuncture database by collecting data from the web. I would like to hear suggestions about improvements to the database structure, code organization, web-...
Sati's user avatar
  • 427
4 votes
2 answers
409 views

I wrote some code to extract the information from a table, but it takes an extremely long time. The table is in the format of a calendar. I need the information on an Excel sheet with column 1 as the ...
Nelj's user avatar
  • 41

1
2 3 4 5
…
13