I'm using atomikos to handle global transaction between 2 different databases. I ran before with more than 1 billion records and this problem just appeared couple days ago. I guess the reason is my table increase the size so it catch atomikos config timeout. Can anyone help me please! For more detail, this is the log that I received:
{"timestamp":"2022-11-29 18:16:09.691","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"insert.job.BagInsertQuartzJob","msg":">>>>> START JOB: bagInsertingJob - CMJobId -28979"}
{"timestamp":"2022-11-29 18:16:09.726","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"o.s.b.c.l.support.SimpleJobLauncher","msg":"Job: [SimpleJob: [name=bagInsertingJob]] launched with the following parameters: [{Time=1669720569691, JobCMId=-28979}]"}
{"timestamp":"2022-11-29 18:16:09.728","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"i.j.l.BagInsertPartitionerAndJobListener","msg":">>> Just Picked a Job: -28979"}
{"timestamp":"2022-11-29 18:16:09.732","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"o.s.batch.core.job.SimpleStepHandler","msg":"Executing step: [bagInsertMasterStep]"}
{"timestamp":"2022-11-29 18:16:09.787","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 0- Start: 0 - End: 10"}
{"timestamp":"2022-11-29 18:16:09.787","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 1- Start: 11 - End: 21"}
{"timestamp":"2022-11-29 18:16:09.787","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 2- Start: 22 - End: 32"}
{"timestamp":"2022-11-29 18:16:09.788","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 3- Start: 33 - End: 39"}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:6] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056980900063 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:7] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056981000065 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:5] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056980900064 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:4] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056980900062 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:17:11.209","level":"ERROR","thread":"[SimpleAsyncTaskExecutor-43] ","logger":"o.s.batch.core.step.AbstractStep","msg":"Encountered an error executing step bagInsertingSlaveStep in job bagInsertingJob"}
org.springframework.transaction.UnexpectedRollbackException: JTA transaction unexpectedly rolled back (maybe due to a timeout); nested exception is javax.transaction.RollbackException: One or more resources refused to commit (possibly because of a timeout in the resource - see the log for details). This transaction has been rolled back instead.
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1038)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at insert.job.step.BagInsertUpdater$$EnhancerBySpringCGLIB$$d6e05d87.write(<generated>)
at insert.job.step.BagInsertUpdater$$FastClassBySpringCGLIB$$e835cd44.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at insert.job.step.BagInsertUpdater$$EnhancerBySpringCGLIB$$efdca957.write(<generated>)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208)
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:138)
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:135)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: javax.transaction.RollbackException: One or more resources refused to commit (possibly because of a timeout in the resource - see the log for details). This transaction has been rolled back instead.
at com.atomikos.icatch.jta.TransactionImp.rethrowAsJtaRollbackException(TransactionImp.java:48)
at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:188)
at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:414)
at com.atomikos.icatch.jta.UserTransactionImp.commit(UserTransactionImp.java:86)
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1035)
... 39 common frames omitted
Caused by: com.atomikos.icatch.RollbackException: One or more resources refused to commit (possibly because of a timeout in the resource - see the log for details). This transaction has been rolled back instead.
at com.atomikos.icatch.imp.ActiveStateHandler.prepare(ActiveStateHandler.java:202)
at com.atomikos.icatch.imp.CoordinatorImp.prepare(CoordinatorImp.java:523)
at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:687)
at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:282)
at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:172)
... 42 common frames omitted
{"timestamp":"2022-11-29 18:17:11.212","level":"INFO","thread":"[SimpleAsyncTaskExecutor-43] ","logger":"o.s.batch.core.step.AbstractStep","msg":"Step: [bagInsertingSlaveStep:partition3] executed in 1m1s410ms"}
I try to expand the config timeout from 1000 to 10000, but it does not work. Hope that anyone can explain what's happened and how can i fix it.
