How to log request/response using java.net.http.HttpClient?

You can log request and responses by specifying
-Djdk.httpclient.HttpClient.log=requests on the Java command line.

As for testing/mocking you might want to have a look at the offline test: http://hg.openjdk.java.net/jdk/jdk/file/tip/test/jdk/java/net/httpclient/offline/

Depending on what you are looking to achieve you could use a "DelegatingHttpClient" to intercept and log requests and responses too.

Besides the Java API documentation there's also some high level documentation at http://openjdk.java.net/groups/net/httpclient/index.html

Additional note:

The jdk.httpclient.HttpClient.log property is an implementation specific property whose value is a comma separated list which can be configured on the Java command line for diagnosis/debugging purposes with the following values:

-Djdk.httpclient.HttpClient.log=
       errors,requests,headers,
       frames[:control:data:window:all],content,ssl,trace,channel,all

On our side, we did not find the logging provided by -Djdk.internal.httpclient.debug readable enough. The solution we came up with is to wrap the HttpClient with a decorator that will be able to intercept the calls and provide logging. Here how it somehow looks (should be done not only for send but sendAsync methods) :

public class HttpClientLoggingDecorator extends HttpClient {

  private static final Logger logger = Logger.getLogger(HttpClientLoggingDecorator.class.getName());

  private final HttpClient client;

  ...

  @Override
  public <T> HttpResponse<T> send(HttpRequest req, HttpResponse.BodyHandler<T> responseBodyHandler)
    throws IOException,
      InterruptedException
  {
    subscribeLoggerToRequest(req);

    HttpResponse<T> response = client.send(req, responseBodyHandler);

    logResponse(response);
    return response;
  }

  private void subscribeLoggerToRequest(HttpRequest req) {
    // define a consumer for how you want to log
    // Consumer<String> bodyConsumer = ...;
    if (req.bodyPublisher().isPresent()) {
      req.bodyPublisher().get().subscribe(new HttpBodySubscriber(bodyConsumer)));
    } else {
      bodyConsumer.accept(NO_REQUEST_BODY);
    }
  }

  private <T> void logResponse(HttpResponse<T> response) {
    // String responseLog = ...;
    logger.info(responseLog);
  }

}

And here is the HttpBodySubscriber:

public class HttpBodySubscriber implements Flow.Subscriber<ByteBuffer> {

  private static final long UNBOUNDED = Long.MAX_VALUE;

  private final Consumer<String> logger;

  public HttpBodySubscriber(Consumer<String> logger) {
    this.logger = logger;
  }

  @Override
  public void onSubscribe(Flow.Subscription subscription) {
    subscription.request(UNBOUNDED);
  }

  @Override
  public void onNext(ByteBuffer item) {
    logger.accept(new String(item.array(), StandardCharsets.UTF_8));
  }

  @Override
  public void onError(Throwable throwable) {
  }

  @Override
  public void onComplete() {
  }

}

In order to get this working with spring boot I had to do a number of things.

First, since I'm logging via log4j2 I needed log4j-jul and log4j-jpl artifacts. If you're not using spring boot additional steps may be required. note: jul is actually in the spring-boot-starter-log4j2 but jpl is not.

libs.versions.toml

[versions]
spring = "3.+"

[libraries]
spring-bom = { module = "org.springframework.boot:spring-boot-dependencies", version.ref = "spring" }
starter-log4j2 = { module = "org.springframework.boot:spring-boot-starter-log4j2" }
log4j-jpl = { module = "org.apache.logging.log4j:log4j-jpl" }
log4j-jul = { module = "org.apache.logging.log4j:log4j-jul" }

The first was I had to enable http client logging via gradle, this doesn't seem to get picked up via application.properties in gradle.

build.gradle.kts

dependencies {
  runtimeOnly(platform(libs.spring.bom))
  runtimeOnly(libs.starter.log4j2)
  testRuntimeOnly(libs.log4j.jpl)
  modules {
    module("org.springframework.boot:spring-boot-starter-logging") {
      replacedBy(
        "org.springframework.boot:spring-boot-starter-log4j2",
        "Use Log4j2 instead of Logback",
      )
    }
  }
}

tasks.withType<Test>().configureEach {
  systemProperty("jdk.httpclient.HttpClient.log", "all")
}

Then, even if your root logger is set to info.

application.properties

logging.level.jdk.httpclient = info

If you aren't using spring boot, gradle, or log4j2 you'll need to change these in different places.

note: I haven't figured out what logs frames but this was confusing

-Djdk.httpclient.HttpClient.log=
       errors,requests,headers,
       frames[:control:data:window:all],content,ssl,trace,channel,all

you can put frames,all or frames:data,all. This needs to be "first" from the looks of the code (or maybe just on a newline), but the code uses startsWith and then it splits on : or uses frames:all by default. using the square brackets like above seems to confuse the parser.

Even with with this set properly frames was never called and I don't understand when it would be enough to know why.