Spring Rest Client read timeout not working as expected with Simple Client Http Factory

I have the code like this:

var fac = new SimpleClientHttpRequestFactory();
fac.setConnectionTimeout(Duration.ofMillis(100))
fac.setReadTimeout(Duration.ofMillis(500))

var restClient = RestClient.builder().requestFactory(fac).build();

restClient.post().uri("...").header("...", "123").body("Some text").retrive().toEntity(String.class)

The problem is I expect this method to not take more than 500+100 + some extra overhead like 5 to 10 milliseconds.

But for some reason sometime it takes up to 5 seconds. Is there any other timeout I should set? Docs isn't super clear and is there any way to have full timeout of 700. I can think of creating threads, create a future and wait upto 600 and interrupt it, but that seems overkill

Thanks