pyspark 'DataFrame' object has no attribute 'pivot'

Question

I'm using pyspark 2.0 I have a df like this:

    +----------+----------+--------
    |pid       |      date| p_category
    +----------+----------+--------
    |    1ba   |2016-09-30|flat
    |    3ed   |2016-09-30|ultra_thin
    +----------+----------+----------

I did a

    df.groupBy("p_category","date") \                        
    .agg(countDistinct("pid").alias('cnt'))

and I got this:

    +-------------+----------+------+
    |p_category   |      date|   cnt|
    +-------------+----------+------+
    |    flat     |2016-09-30|116251|
    |ultra_thin   |2016-09-30|113017|
    +-------------+----------+------+

But I want I pivot table like this:

    +----------+----------+------+
    |date      |      flat|  ultra-thin
    +----------+----------+------+
   2016-09-30  |    116251|113017
    ------------------------------
   df.groupBy("p_category","date") \                        
    .agg(countDistinct("pid").alias('cnt')).pivot("p_category")

I got this error:

'DataFrame' object has no attribute 'pivot'

How could I do a pivot in such case or is there other solution? Thanks

Mariusz · Accepted Answer · 2021-09-27 06:35:31Z

5

You should call pivot on gruped data, so first you need to group by date and then pivot by p_category:

>>> df.groupBy('date').pivot('p_category').agg(countDistinct('pid').alias('cnt')).show()
+----------+----+----------+                                                    
|      date|flat|ultra_thin|
+----------+----+----------+
|2016-09-30|   1|         1|
+----------+----+----------+

edited Sep 27, 2021 at 6:35

answered Dec 13, 2016 at 6:55

Mariusz

14k3 gold badges66 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

hipokito Over a year ago

Looks like a link is outdated but it can still be found on spark.apache.org/docs

Collectives™ on Stack Overflow

pyspark 'DataFrame' object has no attribute 'pivot'

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related