I am trying to extract medium data and save it to the database. I am able to extract data into lists and clubbed them using zip function. But when I am inserting the data it is not getting inserted.
I have iterated to zip function to save data to a database. But not getting the required outcome. I have crawled medium posts successfully and there is nothing wrong with the sequence of my tables in the database.
import mysql.connector
mydb = mysql.connector.connect(host="localhost", user="root",
password="Jai@1998", database="crawldb")
cursor = mydb.cursor()
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = webdriver.ChromeOptions()
opts.set_headless()
assert opts.headless
browser = webdriver.Chrome(options=opts)
browser.get('https://medium.com/')
comp = browser.find_elements_by_xpath('//div[@class = "streamItem
streamItem-extremePostPreview js-streamItem"]')
authors = []
timeline = []
for i in range(len(comp)):
x = comp[i].find_element_by_xpath('.//div[@class = "postMetaInline
postMetaInline-authorLockup u-padding0 u-flexShrink1 u-
noWrapWithEllipsis"]')
authors.append(x)
y = comp[i].find_element_by_xpath('.//div[@class = "ui-caption u-
noWrapWithEllipsis"] ')
timeline.append(y)
topics = browser.find_elements_by_xpath('//h2[@class = "ui-h2 ui-xs-h4
ui-clamp3"]')
descriptions = browser.find_elements_by_xpath('//div[@class = "ui-
summary ui-clamp2 u-marginTop2"]')
for title, desc, author, time in zip(topics, descriptions, authors,
timeline):
print("topics : descriptions : authors : timeline")
print(title.text + ": " + desc.text + ": "+ author.text + ": " +
time.text, '\n')
sql = "INSERT INTO `DATAS`(topics, descriptions, authors, time)
VALUES(%s , %s , %s , %s)", (title.text, desc.text ,
author.text , time.text)
try:
cursor.executemany(sql)
mydb.commit()
print("data inserted!!")
except:
mydb.rollback()
mydb.close()
print("data not inserted!!")
The output is the extracting data and data not inserted! in SQL.
try-exceptblock from the for-loop. You are hiding all excpetions/tracebacks that could help you troubleshoot the issue. Hint: you are usingexecutemanyincorrectly, in several ways.