How can I simulate a Cassandra timeout in a K8s environment?

Question

I am working with a K8s environment running application (WEB API/interface) that interacts with a Cassandra database cluster. I need to simulate network latency or introduce delays specifically for responses from the Cassandra cluster to one of my Kubernetes pods. I cannot touch Cassandra DB cluster or specific apache pods (limitation due to lab usage).

Cassandra version: $ nodetool version ReleaseVersion: 4.1.1 $

Easiest way would be to use "tc"(traffic control) within K8s POD, but it's not available, and there is no possibility to install it (install package comes from R&D and extra development would be needed for that.

As an alternative approach, I attempted to adjust the "cas-q.request.timeout.ms" parameter to 1ms and introduced extra load to increase the likelihood of timeouts, but no success.

Maybe there is a way to introduce some latency outside of the K8s POD, or some other methods that I did not consider.

well, this simulates a different problem, it triggers a "connection error" instead of "connection.timeout", although I have tried different variations of this scenario. Thanks for suggestion. — Igor Țurcanu
– Igor Țurcanu, Commented Apr 1, 2024 at 17:14
Can't exactly just easily build a bogus stored proc in Cassandra that would spin... Can you set up a spoof target with a simple nginx or apache (or whatever works for you) that will behave as if it were valid, at least enough to fool your pod? Send back just enough for a connection but when it gets a request, then sit and spin? — Paul Hodges
– Paul Hodges, Commented Apr 1, 2024 at 18:37

Erick Ramirez · Accepted Answer · 2024-05-13 02:31:14Z

0

One possibility is to configure the Cassandra driver in your application with a very low request timeout. This means that you don't need to have access to the Cassandra cluster and do not need to touch the Cassandra pods at all.

If you are using the Java driver for example, you can set the request timeout in application.conf to a few milliseconds:

datastax-java-driver {
  basic.request {
    timeout = 2 seconds
  }
}

For details, see Cassandra Java driver Reference Configuration. Cheers!

answered May 13, 2024 at 2:31

Erick Ramirez

16.5k2 gold badges22 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Igor Țurcanu Over a year ago

Thank you, will definitely check this out!

Erick Ramirez Over a year ago

Please keep us posted. I'm sure there'll be a few users here who would be interested of the outcome of your test. Cheers!

Collectives™ on Stack Overflow

How can I simulate a Cassandra timeout in a K8s environment?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related