create list of values from array of maps in pyspark

Question

I have a table like this

company_id  | an_array_of_maps
--------------------------------------------------------------
234         | [{"a": "a2", "b": "b2"}, {"a": "a4", "b": "b2"}]
123         | [{"a": "a1", "b": "b1"}, {"a": "a1", "b": "b1"}]
678         | [{"b": "b5", "c": "c5"}, {"b": Null, "c": "c5"}]

and i want to get a table like this (the value of the "a" key in each map)

company_id  | an_array_of_maps
--------------------------------------------------------------
234         | ["a2", "a4"]
123         | ["a1", "a1"]
678         | ["b5", Null]

I tried this df.withColumn("array_of_as", F.expr("filter(an_array_of_maps, x -> x.a)")).show()

but i get the following error:

AnalysisException: cannot resolve 'filter(`an_array_of_maps`, lambdafunction(namedlambdavariable()['a'], namedlambdavariable()))' due to data type mismatch: argument 2 requires boolean type, however, 'lambdafunction(namedlambdavariable()['a'], namedlambdavariable())' is of string type.;

Tytire Recubans · Accepted Answer · 2021-12-18 10:10:34Z

2

Got it - filter is the wrong function. It should be:

(df
  .withColumn("array_of_as", 
              F.expr("transform(an_array_of_maps, x -> x.a)"))
 ).show()

I was not filtering anything i was transforming the list of maps into a list of values of the maps - hence the transform.

answered Dec 18, 2021 at 10:10

Tytire Recubans

1,00713 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

create list of values from array of maps in pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related