607 questions
1
vote
0
answers
48
views
Unable to upload file using spark Java
I'm using Spark java with below dependency, and I'm trying to make one API endpoint to receive an file using multipart file, Below is the code attached, I've tried multiple ways to do that, but ...
2
votes
0
answers
91
views
java.lang.ClassCastException: class java.lang.String cannot be cast to class org.apache.spark.unsafe.types.VariantVal
I evaluate Spark 4 try_variant_get method handling variant type data. First I make sql statements examples.
CREATE TABLE family (
id INT,
data VARIANT
);
INSERT INTO family (id, data)
VALUES
(1, ...
- 1,433
0
votes
0
answers
363
views
java.lang.NoSuchFieldError: LZ4 : spark.sql()
I am trying to run one of my methods which has -
spark.sql("DROP TABLE IF EXISTS " + dbNameString + "." + tableNameString)
When I am running the method, the code breaks on the ...
- 1
0
votes
1
answer
52
views
how to handle exceptions in SparkJava
I'm trying to handle all exception (exception handler) for SparkJava
I am currently using try/catch blocks in each endpoint but that approach doesn't scale properly.
Something that can handle ...
0
votes
1
answer
106
views
Error when working with geoscan on databricks
I used geoscan libraries on Azure Databricks. However, when I configured for my task, I got this error.
`TypeError: 'JavaPackage' object is not callable
------------------------------------------------...
0
votes
1
answer
89
views
Is @Bind needed?
I discovered by accident that in a spring boot project I didn't have to bind argument in a query like the one below.
@SqlQuery("""
select id, name
from organisations
...
- 4,383
-1
votes
1
answer
115
views
Spark Java sum is giving incorrect value
Spark Java sum is giving incorrect value
Java sample code is as below
List<Double> points = Arrays.asList(-6221.4, 6380.46);
Dataset<Row> dt = spark.createDataset(points, Encoders.DOUBLE()...
- 9
0
votes
1
answer
188
views
repartition not working with xml file in Spark
I have dataframe which I want to save as multiple xml files. This is my code:
employees
.repartition(col("first_name"))
.write()
.option(&...
- 3,759
0
votes
2
answers
71
views
Select a list of set columns AND other columns depending on their name/regex (Spark SQL Java)
I have a list of columns that I need to select. I have the field names for each column in this list, so it is easy to select.
public Column[] getSelectColumns()
{
return new Column[]{
...
- 157
0
votes
1
answer
588
views
How to get attribute value with correct datatype from Row in java spark sql
I am using Spark Java (not scala or python).
I have a dataframe (Dataset<Row>) and I want to access specific fields from the Row. I have been able to get the field value if it is a String, but I ...
- 157
1
vote
3
answers
4k
views
Is the Spark Java web framework compatible with Java 17?
Spark - a tiny web framework for Java 8
https://sparkjava.com/
It's always mentioned that this framework is for Java 8. Can anyone confirm that this framework is forward compatible? Can we use the ...
0
votes
1
answer
532
views
Convert a column of Spark dataframe to lowercase
How to convert a column value in Spark dataframe to lowercase/uppercase in Java?
For example, below is the input dataframe:
name | country | src | city | debit
----------------------------...
- 2,170
0
votes
2
answers
189
views
Joining multiple spark dataframes on non-primary key
I am trying to join 1 parent Dataframe with 2 child Dataframes.
Here is how my parent DF look like
PersonId
FirstName
LastName
1
ABC
XYZ
Child DF 1
FirstName
FirstNameMatchedPersonIds
ABC
[1, 10, 20]
...
- 33
0
votes
2
answers
1k
views
spark Connect Two Database tables to produce a third data
DataFrameLoadedFromLeftDatabase=data loaded using DataFrameReader from first database say LeftDB.
I need to
iterate through each row in this dataframe,
connect to a second database say RightDB,
find ...
- 79
0
votes
0
answers
94
views
Spark Java Returns 404 For All Endpoints
I have configured this endpoint and many others just like it. All endpoints return 404. I packaged my .jar in a kubernetes container. In the logs, I see "the requested route [/hello] has not ...
- 120
0
votes
1
answer
293
views
Error running spark simple app without installing spark
I am learning spark. And trying to run a simple spark app that output 5 dataframe rows without installing spark. I know that one can run spark app without installing spark. The app is throwing below ...
- 83
0
votes
1
answer
96
views
Need to return object before/ignoring path in spark java
I want to write my own very basic caching inside my sparkjava server.
My thought is to cache the response somewhere, and have some code in a before block that checks if the incoming path is one that ...
- 2,670
0
votes
2
answers
190
views
How to concatenate null columns in spark dataframe in java?
I am working with spark in java and I want to create a column which is a concatenation of all other column values separated by comma. I have tried few ways to do this but couldn't find a solution.
For ...
- 344
1
vote
0
answers
106
views
Spark Java - strange behavior when uploading files
In my project I want to try to upload files, here is the part of the code responsible for this:
MultipartConfigElement multipartConfigElement =
new MultipartConfigElement(
...
- 11
1
vote
1
answer
284
views
Request body gets lost in http get request. (Java-Spark)
When I sent the following request to my program, the body gets lost:
GET http://localhost:4567/contacts/get-all
Authorization : test
Content-Type : text/plain
Hello
Java Code
public class Test {
...
- 983
0
votes
1
answer
2k
views
How to create a struct column from a list of column names in Spark with Java?
I have a DataFrame with multiple columns, e.g.
root
|-- playerName
|-- country
|-- bowlingAvg
|-- bowlingSR
|-- wickets
|-- battingAvg
|-- battingSR
|-- runs
I also have a list of the column ...
- 45
4
votes
1
answer
2k
views
How to union two dataframes which have same number of columns?
Dataframe df1 contains columns : a, b, c, d, e (Empty dataframe)
Dataframe df2 contains columns : b, c, d, e, _c4 (Contains Data)
I want to do a union on these two dataframes. I tried using
df1.union(...
- 105
0
votes
1
answer
111
views
Can't deploy SparkJava program on Heroku , getting Insufficient privileges error
I have a simple sparkJava endpoint, i am trying to deploy it on Heroku but i am getting the following error.
Failed to deploy application: Insufficient privileges to "spark-heroku-example" ...
0
votes
1
answer
982
views
Creating empty Spark dataframe and adding arbitrary values
I have created an empty dataframe and then trying to add columns and values to the dataframe.
Dataset<Row> runMetadata = sparkSession.emptyDataFrame();
runMetadata = runMetadata.withColumn("...
- 39
2
votes
1
answer
262
views
Spark's DELETE handler returns NULL when accessing any request.queryParams key
I developed a very simple API using the Spark Framework (ref: sparkjava.com). I wrote a delete handler and tested it with Postman.
The Problem
When I try to assign a value using request.queryParams, ...
- 496