0

I have been working on a request that extracts data from an ORACLE 19c instance, and that is processed using aws emr-serveless using spark jdbc connections.

The big picture is that I can't connect to the database because of the error below: 19.15.0.0.1.jar, with Java 17, and I always get an error:

java.sql.SQLRecoverableException: IO Error: Connection reset, Authentication lapse 0 ms.

I've been stuck in it for almost 2 months, and unfortunately I'm not getting past this block. I have an application on emr serveless aws emr-6.13.0, ojdbc10-19.15.0.0.1.jar, with Java 17, and I always get an error.

Here are a few things I've already tested:

  • older versions of emr serveless
  • jdbc8, jdbc10, jdbc11, even compiled in specific versions for oracle 19c.
  • I've used Java 8 and Java 17 as runtime
  • Using pure python, I can connect to the database with Lambda and also as EC2 (I did the test to ensure that other aws services can connect using the necessary VPC).
  • I have ensured that I have network connections, as I can ping, nslookup, and tellnet the destination.

After all, I don't know what to do about this mistake and I don't have any ideas that might be useful.

I'll also make my sparkSubmitParameters available, these are the settings I'm using at the moment.:

--conf spark.driver.extraJavaOptions=-Djavax.net.debug=all 
--conf spark.executor.extraJavaOptions=-Djavax.net.debug=all 
--conf spark.jars=s3://xxxxxxx/emr-serverless/ojdbc10-19.15.0.0.1.jar 
--conf spark.driver.extraJavaOptions=-Djava.security.egd=file:/dev/./urandom 
--conf spark.executor.extraJavaOptions=-Djava.security.egd=file:/dev/./urandom 
--conf spark.driver.extraJavaOptions=-Dsecurerandom.source=file:/dev/./urandom 
--conf spark.executor.extraJavaOptions=-Dsecurerandom.source=file:/dev/./urandom 
--conf spark.dynamicAllocation.enabled=true  
--conf spark.dynamicAllocation.shuffleTracking.enabled=true 
--conf spark.dynamicAllocation.minExecutors=1 
--conf spark.dynamicAllocation.maxExecutors=30 
--conf spark.driver.cores=8 
--conf spark.executor.cores=8 
--conf spark.driver.memory=52g 
--conf spark.executor.memory=52g 
--conf spark.emr-serverless.driver.disk=200g 
--conf spark.emr-serverless.executor.disk=200g 
--conf spark.emr-serverless.driverEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64/ 
--conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64/

Can anyone help me? Any help is welcome.

Improve this question
6
  • Are you able to look at the Oracle logs and see if there's anything relevant? Given the "Connection reset" aspect of the error, it seems like an initial connection is being made? Commented Oct 26, 2023 at 17:17
  • I can request the log, what should I look at in this analysis? Commented Oct 26, 2023 at 17:23
  • Any information or errors around the time your job was trying to connect. I would run a job and then ask for logs between the start/end time of the job and look for any authentication errors. Commented Oct 26, 2023 at 20:49
  • I was able to get the log from a DBA team: VERSION INFORMATION: TNS for Linux: Version 19.0.0.0.0 - Production Oracle Bequeath NT Protocol Adapter for Linux: Version 19.0.0.0.0 TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 Version 19.17.0.0.0 Time: 26-OCT-2023 Tracing not turned on. Tns error struct: ns main err code: 12535 TNS-12535: TNS:operation timed out ns secondary err code: 12606 nt main err code: 0 nt secondary err code: 0 nt OS err code: 0 Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=XX)(PORT=XX)) Commented Oct 27, 2023 at 17:43
  • That error message looks like a timeout error. :\ When you said you did tests with EC2, was it in the exact same VPC/subnet/security group that your EMR Serverless application is using? You can also try the reachability analyzer to verify connectivity: docs.aws.amazon.com/vpc/latest/reachability/… Commented Oct 27, 2023 at 19:42
Related questions
1
Spark History Server very slow when driver running on master node
3
How to hide key password in Spark log?
0
Very long spark process on EMR
Load 6 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.