Fetch Data in CSV files using pandas

Question

I am trying to fetch the stock history data of nifty 50 companies from the website and converting them to CSV. I need to update the same on daily basis. Is there any way, where I can append the current date data to the existing CSV without needing to download it again and again. My code goes like this:-

import os
import csv
import urllib.request as urllib
import datetime as dt
import pandas as pd
import pandas_datareader.data as web
import nsepy as nse

def saveNiftySymbols():
    url = "https://www.nseindia.com/content/indices/ind_nifty50list.csv"
# pretend to be a chrome 47 browser on a windows 10 machine
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}
    req = urllib.Request(url, headers = headers)
# open the url 
    x = urllib.urlopen(req)
    sourceCode = x.read().decode('utf-8') 

    cr = csv.DictReader(sourceCode.splitlines())
    l = [row['Symbol'] for row in cr]
    return l

def fetchDataFromNse(l):
    if not os.path.exists('stock_dfs'):
        os.makedirs('stock_dfs')

    start = dt.datetime(2000, 1, 1)
    end = dt.datetime.today()

    for symbol in l:
        if not os.path.exists('stock_dfs/{}.csv'.format(symbol)):
            df=nse.get_history(symbol,start, end)
            df.to_csv('stock_dfs/{}.csv'.format(symbol))
        else:
            print('Already have {}'.format(symbol))

fetchDataFromNse(saveNiftySymbols())

something like this df.to_csv('filename.csv',mode = 'a',header=False)? — Gayatri
– Gayatri, Commented Oct 11, 2017 at 4:30

rko · Accepted Answer · 2017-10-11 04:49:53Z

1

Try this after market close as NSE is notorious for adding dates without data.
This will only work if you have data already stored for the symbols of NSE. This does not account for any changes in the constituents. This means you'd have to download everything once when NSE changes constituents.

Try this

def saveNiftySymbols():
    url = "https://www.nseindia.com/content/indices/ind_nifty50list.csv"
    # pretend to be a chrome 47 browser on a windows 10 machine
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}          
    req = urllib.request.Request(url, headers = headers)
    url_req = urllib.request.urlopen(req)
    ## Use pandas here. Much more reliable
    table = pd.read_csv(url_req)
    return table.Symbol

def fetchDataFromNse(symbols):
    if not os.path.exists('stock_dfs'):
        os.makedirs('stock_dfs')

    ## given that you have already stored the data, just have the end date
    start = dt.date(2000,3,31)
    end = dt.date.today()
    for symbol in symbols:
        ## you can also convert it to a list if you want.
        df = nse.get_history(symbol, end, end)
        data_to_append = df.to_csv(header= None)
        current_csv = open('stock_dfs/{}.csv'.format(symbol), 'a')
        current_csv.write(data_to_append)
        current_csv.close()



fetchDataFromNse(saveNiftySymbols())

answered Oct 11, 2017 at 4:49

rko

3143 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Udit Hari Vashisht Over a year ago

Thanks for the reply. This helped. But the issue here is that, this command will append the data for the current date. Could not it be modified in the way that it should append data up to current date. means if I run the program after 2-3 days.

rko · Accepted Answer · 2017-10-12 14:55:22Z

0

Yes add this block of code before the two functions and call this when you're fetching Data from NSE

def get_last_date():
    all_files = glob.glob('stock_dfs/*.csv')
    first_csv = open(all_files[0], 'r')
    reader = csv.DictReader(first_csv)
    last_date_str = list(reader)[-1]['Date']
    fmt = '%Y-%m-%d'
    last_date_dt = dt.datetime.strptime(last_date_str, fmt).date()
    return last_date_dt

def fetchDataFromNse(symbols):
    if not os.path.exists('stock_dfs'):
        os.makedirs('stock_dfs')

    ## given that you have already stored the data, just have the end date
    start = dt.date(2000,3,31)

    ### addition here
    new_start = get_last_date()
    end = dt.date.today()
    for symbol in symbols:
        ## you can also convert it to a list if you want.
        df = nse.get_history(symbol, new_start, end)
        data_to_append = df.to_csv(header= None)
        current_csv = open('stock_dfs/{}.csv'.format(symbol), 'a')
        current_csv.write(data_to_append)
        current_csv.close()

Remember this will work reliably only if the constituents are same and are updated together. The second condition is met but you will have to modify for first condition.

answered Oct 12, 2017 at 14:55

rko

3143 silver badges10 bronze badges

1 Comment

Udit Hari Vashisht Over a year ago

Thanks a lot, I modified the function as under to get the same results:- def fetchDataFromNse(symbols): for symbol in symbols: with open('stock_dfs/{}.csv'.format(symbol),'rb') as f: reader=pd.read_csv(f,usecols=[0],infer_datetime_format=True) datelist=reader["Date"].tolist() #datelist.append(reader) s=datelist[-1] sdate=datetime.strptime(s,'%Y-%m-%d') start=sdate+timedelta(days=1)

Collectives™ on Stack Overflow

Fetch Data in CSV files using pandas

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related