java.net.SocketException: Socket Closed eventhough input and output streams are closed after read, write

Question

I'm running multi threaded java application where below method will be executed by many threads(about 300) in parallel. And i see this exception only when thread count is more but not when thread count is less than 50. I read few posts in stackoverflow and code looks in complaints to what posts described. Any help would be appreciated. Thanks in advance.

Here is my method:

public static String invokeWebService(String request, String fwfmId, BatchVO batchVO){
    String responseString = "";
    //log the request string
    Logger_trace.info("Service request for input "+ fwfmId +":: "+CommonUtil.removeCrLf(request));
    try {
            URL url = new URL(batchVO.getWebServiceUrl());
            HttpURLConnection  urlConnection = (HttpURLConnection) url.openConnection();
            urlConnection.setRequestMethod("POST");
            urlConnection.setRequestProperty("Content-type", "text/xml; charset=utf-8");

            urlConnection.setDoOutput(true);
            urlConnection.setDoInput(true);
            urlConnection.setConnectTimeout(Integer.parseInt(batchVO.getConnectionTimeOut()));
            urlConnection.setReadTimeout(Integer.parseInt(batchVO.getReadTimeOut()));
            OutputStream outputStream = urlConnection.getOutputStream();
            outputStream.write(request.getBytes());
            outputStream.flush();

            //get response form input stream and convert into string
            InputStreamReader inputStreamReader = new InputStreamReader(urlConnection.getInputStream());
            BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
            String inputLine;
            StringBuffer response = new StringBuffer();

            while ((inputLine = bufferedReader.readLine()) != null) {
                response.append(inputLine);
            }

            responseString = response.toString();

            //close resources
            bufferedReader.close();
            inputStreamReader.close();
            outputStream.close(); 

            //log the response string
            logger_trace.info("Service response for input "+ fwfmId +":: "+ CommonUtil.removeCrLf(responseString));

            //Update Database columns
            updateRecordInTable(request, responseString, fwfmId, batchVO.getDataSource(), batchVO.getBatchId());    

        } catch (SocketConnectException e) {
            logger_trace.error("WebserviceHandler.invokeWebService : SocketConnectException block : "+fwfmId +" ", e);
            updateRecordInTableWithTimeout(fwfmId, batchVO.getDataSource(), batchVO.getBatchId());
        } catch (SocketTimeoutException e) {
            logger_trace.error("WebserviceHandler.invokeWebService : SocketTimeoutException block : "+ fwfmId+" ", e);
            updateRecordInTableWithTimeout(fwfmId, batchVO.getDataSource(), batchVO.getBatchId());
        } catch (MalformedURLException e) {
            logger_trace.error("WebserviceHandler.invokeWebService : MalformedURLException block : ", e);
        } catch (IOException e) {
            logger_trace.error("WebserviceHandler.invokeWebService : IOException block : ", e);
        } catch(Exception e){
            logger_trace.error("WebserviceHandler.invokeWebService : General Exception block : ", e);
        }
        return responseString;
    }

Exception:

 [ERROR] 2017-10-31 03:18:54,435 [main] TRACE -
 WebserviceHandler.invokeWebService : IOException block : 
 java.net.SocketException: Socket Closed
         at java.net.AbstractPlainSocketImpl.setOption(AbstractPlainSocketImpl.java:212)
 ~[?:1.8.0_131]
         at java.net.Socket.setTcpNoDelay(Socket.java:980) ~[?:1.8.0_131]
         at weblogic.net.http.HttpClient.openServer(HttpClient.java:411)
 ~[wlfullclient.jar:12.2.2.0.0]
         at weblogic.net.http.HttpClient.openServer(HttpClient.java:511)
 ~[wlfullclient.jar:12.2.2.0.0]
         at weblogic.net.http.HttpClient.New(HttpClient.java:313) ~[wlfullclient.jar:12.2.2.0.0]
         at weblogic.net.http.HttpURLConnection.getHttpClient(HttpURLConnection.java:314)
 ~[wlfullclient.jar:12.2.2.0.0]
         at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:760)
 ~[wlfullclient.jar:12.2.2.0.0]
         at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:41)
 ~[wlfullclient.jar:12.2.2.0.0]
         at ca.bell.webservice.WebServiceHandler.invokeWebService(WebServiceHandler.java:56)
 [MigrationTool.jar:?]
         at ca.bell.webservice.ExecuteMigrationWork.performRun(ExecuteMigrationWork.java:40)
 [MigrationTool.jar:?]
         at ca.bell.workmanagement.work.Work.call(Work.java:155) [MigrationTool.jar:?]
         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 [?:1.8.0_131]
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 [?:1.8.0_131]
         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Alexandre Dupriez · Accepted Answer · 2017-11-02 10:47:25Z

1

Interesting. Let's start with an analysis of your stack trace, starting from the closest source of the exception. From the source code of AbstractPlainSocketImpl, the first precondition enforced is:

public void setOption(int opt, Object val) throws SocketException {
  if (isClosedOrPending()) {
    throw new SocketException("Socket Closed");
  }
...

Since the scope of your HttpURLConnection is limited to the method invokeWebService, we can rule out the hypothesis of a concurrent action on the socket, and since seems fine method and hasn't closed the socket, I would believe that isClosedOrPending returns true because the socket's file descriptor is null:

public boolean isClosedOrPending() {
    /*
     * Lock on fdLock to ensure that we wait if a
     * close is in progress.
     */
    synchronized (fdLock) {
        if (closePending || (fd == null)) {
            return true;
        } else {
            return false;
        }
    }
}

What is questionable in this analysis is why the fd file descriptor is null. However, this is not really important at this stage. Should we believe in our assumptions, one conclusion we could draw is that you are creating too many connections for your system to handle.

How do we solve this? Here we need to step back a little and see if we can refactor the code to limit the number of concurrent connections. I would need much more details about your specific implementation to narrow down the design candidates which are otherwise boundless. For instance, would a connection pool be appropriate? Should we implement back-pressure and throttle the clients? Should we scale horizontally and distribute the load?

edited Nov 2, 2017 at 10:47

answered Nov 2, 2017 at 10:42

Alexandre Dupriez

3,05623 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user1510335 Over a year ago

Thanks Alex. The requirement for me is to implement multi threaded java tool which reads data from DB, construct web service request, call service and update DB with status(success/failure) based on web service response. Now, to implement this, I used ThreadPoolExecutor framework. Dynamically getting the core and max thread size from property file and creating workItems for each webservice request and call invokeWebservice() for each workItem.

Alexandre Dupriez Over a year ago

Thanks @user1510335 - I am curious to know about various metrics to help analyse the load - what is the median/peak requests per seconds (or minutes)? What is the size of your thread pool and how do you use it? Do you queue clients' requests before they are being served?

user1510335 Over a year ago

Hi Alex, peak requests per second would be approx 400 and max thread size is also set to 400. Framework(ThreadPoolExecutor) is taking care of queuing the requests if max thread count is reached and no explicit code has been written to manage threads.

Alexandre Dupriez Over a year ago

@user1510335: ok, so 400 threads opening connections is too much - a quick check would be to try reducing the max number of threads down to 50 or 100 and see what happens - and if there is too much contention for the clients?

Collectives™ on Stack Overflow

java.net.SocketException: Socket Closed eventhough input and output streams are closed after read, write

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related