How to stop python script from exiting when an error occurred while scraping data from website

Question

I wrote a python script to scrape data from website with a 1-minute interval. But sometimes error occurred and the script exit. Is there any way to avoid the script from exiting even after the error occurred?

My code:

 from requests
 import Session
 import pandas as pd
 from bs4 import BeautifulSoup
 import re 

 def get_option_chain(symbol, expdate):

  if symbol == 'NIFTY':
  Base_url = ("https://www.nseindia.com/live_market/dynaContent/live_watch/option_chain/optionKeys.jsp?symbol="+symbol+"&date="+expdate)

  

  new_table = pd.DataFrame(columns=col_hdrs_name)

 

  print(new_table)

  new_table.to_csv('Option_Chain_Table_{}.csv'.format(symbol))

 get_option_chain('NIFTY','17OCT2019')


schedule.every(1).minutes.do(get_option_chain,'NIFTY','17OCT2019')

while 1:
        schedule.run_pending()
        time.sleep(1)

Error:

Exception ('Connection aborted.', RemoteDisconnected('Remote end closed connection without 
response')) in getting data  for symbol NIFTY
Traceback (most recent call last):
 File "C:\Python\Python37\NSE scrape fata.py", line 70, in <module>
schedule.run_pending()
File "C:\Python\Python37\lib\site-packages\schedule\__init__.py", line 563, in run_pending
default_scheduler.run_pending()
File "C:\Python\Python37\lib\site-packages\schedule\__init__.py", line 94, in run_pending
 self._run_job(job)
File "C:\Python\Python37\lib\site-packages\schedule\__init__.py", line 147, in _run_job
 ret = job.run()
File "C:\Python\Python37\lib\site-packages\schedule\__init__.py", line 466, in run
 ret = self.job_func()
File "C:\Python\Python37\NSE scrape fata.py", line 41, in get_option_chain
soup = BeautifulSoup(page.content, 'html.parser')
UnboundLocalError: local variable 'page' referenced before assignment

Are you going to show us the real code that caused the exception? — Klaus D.
– Klaus D., Commented Oct 15, 2019 at 11:22

Sid · Accepted Answer · 2019-10-15 11:40:58Z

2

Yes, you can use try...except:

from requests import Session
import pandas as pd
from bs4 import BeautifulSoup
import re 
import time

def get_option_chain(symbol, expdate):
    if symbol == 'NIFTY':
        Base_url = ("https://www.nseindia.com/live_market/dynaContent/live_watch/option_chain/optionKeys.jsp?symbol="+symbol+"&date="+expdate)



new_table = pd.DataFrame(columns=col_hdrs_name)


print(new_table)
new_table.to_csv('Option_Chain_Table_{}.csv'.format(symbol))

get_option_chain('NIFTY','17OCT2019')


schedule.every(1).minutes.do(get_option_chain,'NIFTY','17OCT2019')

while 1:
    try:
        schedule.run_pending()
        time.sleep(1)
    except:
        continue

Also you did not import time module and used time.sleep(1). So import time. I recommend reading this as well.

edited Oct 15, 2019 at 11:40

answered Oct 15, 2019 at 11:14

Sid

2,1891 gold badge15 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to stop python script from exiting when an error occurred while scraping data from website

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related