PySpark -- Convert List of Rows to Data Frame

Question

The problem I'm actually trying to solve is to take the first/last N rows of a PySpark dataframe and have the result be a dataframe. Specifically, I want to be able to do something like this:

 my_df.head(20).toPandas()

However, because head() returns a list of rows, I get this error:

AttributeError: 'list' object has no attribute 'toPandas'

So, I'm looking either for method that will return the first N rows of PySpark dataframe as a dataframe, or a method for converting these lists of rows into a dataframe. Any ideas?

user6022341 · Accepted Answer · 2016-05-01 18:25:55Z

11

With limit:

>>> df = sc.parallelize((("a", 1), ("b", 2))).toDF()
>>> df.limit(1).toPandas()
  _1  _2
0  a   1

with pd.DataFrame:

>>> pd.DataFrame(df.head(1), columns=df.columns)
  _1  _2
0  a   1

Collectives™ on Stack Overflow

PySpark -- Convert List of Rows to Data Frame

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related