How can we create nested arrays in scala from a dataframe?

Question

I have a DataFrame with three columns; How can I convert it to a nested array in Scala - Array[Array[Double]]?

I am getting Array[(Double, Double)] when i try to map:

val x= dataframe.select("time","sex")
  .collect()
  .map(x=>x(0).toString.toDouble ,x(1).toString.toDouble)

rogue-one · Accepted Answer · 2017-02-02 15:08:27Z

1

That is because your map is returning a tuple. since you needed an Array you need to return an Array as shown below.

val df = dataframe.collect().map(x => Array(x.getDouble(0), x.getDouble(1)))

to access the first item (nested Array) in the result use df.head or df(0)

EDIT:

to have the dataset in columnar fashion Array[Array[Double]] where the each column has its own array.

df.foldLeft((Array[Double](),Array[Double]())) {
  case ((col1,col2),(x,y)) => (col1 :+ x) -> (col2 :+ y)
} match {
    case (arr1,arr2) => Array(arr1,arr2)
}

edited Feb 2, 2017 at 15:08

answered Feb 1, 2017 at 16:57

rogue-one

11.6k8 gold badges56 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Deep Over a year ago

How can we access first array from it?

rogue-one Over a year ago

@BejjamDEEPTHIPRIYA can you please elaborate your request?

Deep Over a year ago

From the above command we will getv array[array[double]] how can we access first array ???

Deep Over a year ago

Hi actually I'm looking for nestedarray[array[array1,array2]] array 1 is one Column of dataframe and array 2 is another column and the solution u provided gives me arrays of each row .do u have any suggestion for my qstion if so please let me know

rogue-one Over a year ago

@Deep so you need Array[Array[Double],Array[Double]]. ie an Array with two arrays for each column (ie a column oriented storage)

|

Collectives™ on Stack Overflow

How can we create nested arrays in scala from a dataframe?

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related