while loop - sending output to csv file

Question

This goes to a URL from the CSV file , then scrolls down. I'm trying to grab the company URLs off of the page. I can't seem to get it to work. Now if I use just one stand alone URL without pulling it from the CSV, it will print to powershell. Still can't get it to write to CSV.

Here is a couple of URLs that I'm working with:

https://www.facebook.com/search/pages/?q=Los%20Angeles%20remodeling
https://www.facebook.com/search/pages/?q=Boston%20remodeling

The thought I had was that it has to be a loop within a loop. Or, it could be if, elif. I'm not really sure at this point. Any and all suggestions would be appreciated.

import time
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import csv
import requests
from selenium.webdriver.support.ui import WebDriverWait


driver = webdriver.Chrome()
elems = driver.find_elements_by_class_name('_32mo')


chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(chrome_options=chrome_options)


driver.get('https://www.facebook.com')
username = driver.find_element_by_id("email")
password = driver.find_element_by_id("pass")
username.send_keys("*****")
password.send_keys("******")
driver.find_element_by_id('loginbutton').click()
time.sleep(2)



with open('fb_urls.csv') as f_input, open('fb_profile_urls.csv', 'w', newline=)  as f_output:
    csv_input = csv.reader(f_input)
    csv_output = csv.writer(f_output)
    for url in csv_input:
        driver.get(url[0])
        time.sleep(5)
        lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
        match=False
        while(match==False):
            lastCount = lenOfPage
            time.sleep(1)
            lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
            if lastCount==lenOfPage:
                match=True
                for elem in elems:
                    csv_output.(driver.find_elements_by_tag_name('href'))

I'm not sure I understand your question, so a wild guess: open('fb_profile_urls.csv', 'w', newline=) will wipe the file every time you run this script — roganjosh
– roganjosh, Commented Sep 26, 2018 at 18:24
see, that's what i thought as well. But, through research it's what i found in a solution on SO. — RobK
– RobK, Commented Sep 26, 2018 at 18:36
No, it definitely wipes the file each time you run your script — roganjosh
– roganjosh, Commented Sep 26, 2018 at 18:41

G. Anderson · Accepted Answer · 2018-09-26 18:42:44Z

0

Instead of opening the file in write mode open('file','w') open it in append mode open('file','a')

Found in how to add lines to existing file using python

answered Sep 26, 2018 at 18:42

G. Anderson

5,9652 gold badges16 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

RobK Over a year ago

this is almost comical that i just took it at face value that it wouldn't. i know better. even after doing that it's not working. It's just a blank sheet after running

G. Anderson Over a year ago

Silly question, did you repopulate the file with more data after you did the 'w' and before you did the 'a'? When I run fname = 'testfile1.txt' with open(fname,'w') as f: f.write('Some Junk') the file contains Some Junk Then I run with open(fname,'a') as f: f.writelines('\nmorejunk') It contains Some Junk morejunk

RobK Over a year ago

i understand what you're saying. And if i do what you just did, it works fine. It has to be something to do with the loop. I'm not quite sure if it's where i name 'elems' or where i'm writing a 'for' loop at the end to have it print to file. Does that make sense?

G. Anderson Over a year ago

Where it says csv_output.(driver... it doesn;t look like you're calling the writerow() function, thus not giving it anything to write

RobK Over a year ago

tried that...and it doesn't work either. T.T i've been racking my brain for the last few hours on how to get this to work. i think i'm about ready to throw in the towel.

|

Collectives™ on Stack Overflow

while loop - sending output to csv file

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related