0

I am trying to extract medium data and save it to the database. I am able to extract data into lists and clubbed them using zip function. But when I am inserting the data it is not getting inserted.

I have iterated to zip function to save data to a database. But not getting the required outcome. I have crawled medium posts successfully and there is nothing wrong with the sequence of my tables in the database.

import mysql.connector
mydb = mysql.connector.connect(host="localhost", user="root", 
       password="Jai@1998", database="crawldb")
cursor = mydb.cursor()


from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = webdriver.ChromeOptions()
opts.set_headless()
assert opts.headless
browser = webdriver.Chrome(options=opts)
browser.get('https://medium.com/')


comp = browser.find_elements_by_xpath('//div[@class = "streamItem 
       streamItem-extremePostPreview js-streamItem"]')

authors = []
timeline = []

for i in range(len(comp)):
    x = comp[i].find_element_by_xpath('.//div[@class = "postMetaInline 
        postMetaInline-authorLockup u-padding0 u-flexShrink1 u- 
        noWrapWithEllipsis"]')
    authors.append(x)
    y = comp[i].find_element_by_xpath('.//div[@class = "ui-caption u- 
        noWrapWithEllipsis"] ')
    timeline.append(y)




 topics = browser.find_elements_by_xpath('//h2[@class = "ui-h2 ui-xs-h4 
          ui-clamp3"]')
 descriptions = browser.find_elements_by_xpath('//div[@class = "ui- 
                summary ui-clamp2 u-marginTop2"]')


for title, desc, author, time in zip(topics, descriptions, authors, 
                                     timeline):
    print("topics : descriptions : authors : timeline")
    print(title.text + ": " + desc.text + ": "+ author.text + ": " + 
          time.text, '\n')


    sql = "INSERT INTO `DATAS`(topics, descriptions, authors, time) 
           VALUES(%s , %s , %s , %s)", (title.text, desc.text , 
           author.text , time.text)

    try:
        cursor.executemany(sql)
        mydb.commit()
        print("data inserted!!")

    except:
          mydb.rollback()

          mydb.close()                
          print("data not inserted!!")

The output is the extracting data and data not inserted! in SQL.

15
  • Did you try inserting fake data yourself and see if your sql working as intended? Commented Jul 15, 2019 at 6:46
  • Yeah! it is inserting data manually. no problem with that. Commented Jul 15, 2019 at 7:36
  • Remove the try-except block from the for-loop. You are hiding all excpetions/tracebacks that could help you troubleshoot the issue. Hint: you are using executemany incorrectly, in several ways. Commented Jul 15, 2019 at 7:57
  • replacing execute in place of executemany doesn't work too! Commented Jul 15, 2019 at 9:35
  • removing try/except block from for loop still doesn't work!! Commented Jul 15, 2019 at 9:38

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.