Trying to export a SQL table but getting errors with Python + PYODBC and pandas [duplicate]

Question

Hello I am trying to export a very large SQL table (approx 1.7million rows) into either an excel or csv file, my problem is when I dont specify a chunksize I am getting a memory limit error, I am guessing the data is too large to store in memory. Now that I have passed in a chunksize I am getting a different error.

Error - VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. values = np.array([convert(v) for v in values])

Code:

import pyodbc
import pandas as pd


conn = pyodbc.connect('Driver={SQL Server};'
                      'Server=servername;'
                      'Database=DB01;'
                      'Trusted_Connection=yes;')


sql_query = pd.read_sql_query("select * from calltype", conn, chunksize=1000, dtype=object)
df = pd.DataFrame(sql_query)
df.to_excel(r'C:\Users\user\Documents\BackupTest1.xlsx', index=False)
print("Records Exported")'''

ggsmith · Accepted Answer · 2022-06-17 14:05:12Z

0

I would check out this article and see if any of these solutions work for you.

Bypassing Pandas Memory Limitations

The message you are getting is just a warning not an error based on what you wrote so if your data exports correctly this may not be an issue but you do some data validation to check. Alternatively, defining the dtype as an object may make the warning go away.

answered Jun 17, 2022 at 14:05

ggsmith

438 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Trying to export a SQL table but getting errors with Python + PYODBC and pandas [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related