Converting generator from read_sql in pandas to dataframe has failed

Question

I want to read data from my oracle, I use the pandas's read_sql and set the parameter chunksize=20000,

from sqlalchemy import create_engine
import pandas as pd
engine = create_engine("my oracle")
df = pd.read_sql("select clause",engine,chunksize=20000)

It returns a iterator, and I want to convert this generator to a dataframe usingdf = pd.DataFrame(df), but it's wrong, How can the iterator be converted to a dataframe?

giser_yugang · Accepted Answer · 2018-09-03 08:15:11Z

2

This iterator can be concatenated, then it return a dataframe:

df = pd.concat(df)

You can view pandas.concat document.

If you can't use concat directly, try the following:

gens = pd.read_sql("select clause",engine,chunksize=20000)
dflist = []
for gen in gens:
    dflist.append(gen)
df = pd.concat(dflist)

edited Sep 3, 2018 at 8:15

answered Sep 3, 2018 at 7:35

giser_yugang

6,1864 gold badges24 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

littlely Over a year ago

TypeError: cannot concatenate object of type "<class 'generator'>"; only pd.Series, pd.DataFrame objs are valid

giser_yugang Over a year ago

@littlely What is your version? There is no problem on me.

littlely Over a year ago

pandas 0.23 and python 3.6.5

giser_yugang Over a year ago

@littlely Try what I add.

Collectives™ on Stack Overflow

Converting generator from read_sql in pandas to dataframe has failed

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related