Is it possible to store a numpy array in a Spark Dataframe Column?

Question

I have a dataframe and I apply a function to it. This function returns an numpy array the code looks like this:

create_vector_udf = udf(create_vector, ArrayType(FloatType()))
dataframe = dataframe.withColumn('vector', create_vector_udf('text'))
dmoz_spark_df.select('lang','url','vector').show(20)

Now spark seems not to be happy with this and does not accept ArrayType(FloatType()) I get the following error message: net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)

I could just numpyarray.tolist() and return a list version of it, but obviously I would always have to recreate the array if I want to use it with numpy.

so is there a way to store a numpy array in a dataframe column?

Possible duplicate of Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct) — Alper t. Turker
– Alper t. Turker, Commented Feb 8, 2018 at 22:47

Nurlybek Amangeldiuly · Accepted Answer · 2022-07-14 14:06:58Z

2

I suggest transforming the numpy arrays to dense Vectors with VectorsUDT, so the code will look like below:

from pyspark.ml.linalg import Vectors, VectorUDT

arrayUDF = udf(lambda array: Vectors.dense(array)),VectorUDT())

df = df.withColumn('pyspark_arrays', arrayUDF(F.col('vector')))

to transform back to numpy arrays, using .toArray() is possible.

answered Jul 14, 2022 at 14:06

Nurlybek Amangeldiuly

435 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

pissall · Accepted Answer · 2019-10-02 18:09:19Z

1

The source of the problem is that object returned from the UDF doesn't conform to the declared type. create_vector must be not only returning numpy.ndarray but also must be converting numerics to the corresponding NumPy types which are not compatible with DataFrame API.

The only option is to use something like this:

udf(lambda x: create_vector(x).tolist(), ArrayType(FloatType()))

answered Oct 2, 2019 at 18:09

pissall

7,4442 gold badges29 silver badges47 bronze badges

Comments

zmag · Accepted Answer · 2019-11-21 05:49:49Z

0

One way to do that is if you convert each row of the numpy array in DataFrame to list of integer.

df.col_2 = df.col_2.map(lambda x: [int(e) for e in x])

Then, convert it to Spark DataFrame directly

df_spark = spark.createDataFrame(df)
df_spark.select('col_1', explode(col('col_2')).alias('col_2')).show(14)

edited Nov 21, 2019 at 5:49

zmag

8,27112 gold badges39 silver badges47 bronze badges

answered Nov 21, 2019 at 5:22

user1460675

4574 silver badges11 bronze badges

Collectives™ on Stack Overflow

Is it possible to store a numpy array in a Spark Dataframe Column?

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related