java.net.SocketException: java.net.SocketException: Too many open files

Question

I have a large java application (which is used to generate some type of report) in which below class is used to create datasource.

 import org.apache.log4j.Logger;
 import org.springframework.jdbc.datasource.DriverManagerDataSource;
 import com.mysql.jdbc.Connection;
 public class DatabaseConnection
  {
     private static final Logger LOGGER = Logger.getLogger(DatabaseConnection.class.getName());
    @SuppressWarnings("deprecation")
    public static DriverManagerDataSource jdbcConnection(WebmartConfiguration webmartconnection)
    {
        DriverManagerDataSource dataSource = null;
        try
        {
            dataSource = new DriverManagerDataSource("com.mysql.jdbc.Driver", "jdbc:mysql://" + webmartconnection.getHostname() + ":" + webmartconnection.getPort() + "/" + webmartconnection.getDatabasename() + "", webmartconnection.getUsername(), webmartconnection.getPassword());
    }
    catch (Exception sqle)
    {
        LOGGER.info(sqle);
    }
    return dataSource;
 }
}

and that datasource is passed to many method that are used to execute query using JDBCTEMPLATE's query method. For some time application runs smooth and generate reports but after some time application terminated with following stacktrace.

ERROR [run has started] (DivisionThread.java:217) - Could not get JDBC Connection;      nested exception is com.mysql.jdbc.CommunicationsException:     Communications link failure due to underlying exception:    

** BEGIN NESTED EXCEPTION **    

java.net.SocketException    
MESSAGE: java.net.SocketException: Too many open files  

STACKTRACE: 

java.net.SocketException: java.net.SocketException: Too many open files 
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2641)
at com.mysql.jdbc.Connection.<init>(Connection.java:1531)
   at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
   at java.sql.DriverManager.getConnection(DriverManager.java:579)
   at java.sql.DriverManager.getConnection(DriverManager.java:190)
   at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:674)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:729)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:745)
at com.mpsinsight.reporting.dao.common.CommonUtilReport.getOutFileName(CommonUtilReport.java:1509)
at com.mpsinsight.reporting.bean.common.ActionEnum$9.getparameterType(ActionEnum.java:507)
at com.mpsinsight.reporting.main.common.JasperReport.populateParameters(JasperReport.java:142)
at com.mpsinsight.reporting.main.common.JasperReport.generateReport(JasperReport.java:61)
at com.mpsinsight.reporting.main.common.GenerateReport$4.generateReport(GenerateReport.java:123)
at com.mpsinsight.reporting.main.common.DivisionThread.run(DivisionThread.java:179)


** END NESTED EXCEPTION **

Example : Below is the one of the method of application that uses datasource.

public String getOutFileName(DriverManagerDataSource datasource, WebmartConfiguration     webmartconnection, String abbrev)
    {
    DriverManagerDataSource dmDatasource = null;
    dmDatasource = datasource;
    if (dmDatasource == null)
    {
        dmDatasource = DatabaseConnection.jdbcConnection(webmartconnection);
    }
    JdbcTemplate jdbcTemplateOb = new JdbcTemplate(dmDatasource);
    String sql = QueryList.Value("outFileNameQuery");

    LOGGER.info("abbrev :::: " + abbrev);
    LOGGER.info("outFileNameQuery :::: " + sql);

    String pathname = (String) jdbcTemplateOb.queryForObject(sql, new Object[] { abbrev }, String.class);

    LOGGER.info("pathname :::: " + pathname);
    return pathname;
}

Could someone explain what could the reason for too many open files error and how can i resolve it.

EDIT : I am working on linux environment. The above error occur only in linux environment.

You're leaking connections, and you don't need to keep recreating the DataSource either. One will do. — user207421
– user207421, Commented Dec 9, 2014 at 7:36
Where is your application deployed ? it is linux enviroment ? — Yagnesh Agola
– Yagnesh Agola, Commented Dec 9, 2014 at 8:27
@EJP i am creating connection again only if dmDatasource is null. and can you be more clear what you want to say? — Lovish
– Lovish, Commented Dec 9, 2014 at 8:30
We can't see from this code where you are creating or closing Connections at all. Only DataSources. You seem to have them confused. They aren't the same thing. — user207421
– user207421, Commented Dec 9, 2014 at 8:48

Snehal Patel · Accepted Answer · 2014-12-15 06:04:12Z

1

Check open file limit using below linux command.

ulimit -a

and you can provide open file limit in below file through root user.

/etc/security/limits.conf

For example

{username}       soft       nofile   1024
{username}       hard   nofile        65536

answered Dec 15, 2014 at 6:04

Snehal Patel

1,3302 gold badges11 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sumit Kamboj Over a year ago

only this my_username hard maxlogins 10 entry is in limits.conf file for my username. should i add two more you suggested above?

Community · Accepted Answer · 2017-05-23 12:02:24Z

1

The reason for a "too many open files" is that the application or system is reaching the limit of allowed file handles.

This problem usually occurs after several concurrent users get a connection to the Server. Java opens many files in order to read in the classes required to run your application. High volume applications can use a lot of file descriptors. This could lead to a lack of new file descriptors. Also, each new socket requires a descriptor. Clients and Servers communicate via TCP sockets. Each browser's http request consumes TCP sockets when a connection is established to a Server.

In order to prevent a file descriptor leak you should ensure that all resources like streams, database connections are explicitly closed as soon as they are no longer needed. Even Java manages resources for you: don't rely on the garbage collector to clean up your used resources. i.e. close all streams either within a finally block or use the Java 7 try with resources syntax.

The reason for the limit is that the operating system needs memory to manage manage each open file, and memory is a limited resource. Different platforms have different limits on the number of files that can be open in a single process at one time.

On Linux you can as root user change the maximum of the open files count per process (via ulimit -n) and per system (e.g. echo 800000 > /proc/sys/fs/file-max) or per user using the /etc/security/limits.conf file. For the last option you may need a special user running your application.

Before you change one or more of these values you should check whether your application has a resource leak by monitoring the open files. One possible option on Linux is to use strace.

strace -e trace=open,close,read,write,connect,accept your-command-here

You'll need to use the -o option to put strace's output somewhere other than the console, if the process can print to stderr. If your process forks, you'll also need -f or -ff

Or if you want to connect you to a running process and you know the process id you can use strace -p $MyProcess.

If you search the web you'll find a solution for monitoring the open file descriptors that matches your needs. For example monitor open process files on linux (real-time)

edited May 23, 2017 at 12:02

CommunityBot

11 silver badge

answered Dec 15, 2014 at 8:45

andih

5,6233 gold badges28 silver badges36 bronze badges

2 Comments

user207421 Over a year ago

"Java opens many files in order to read in the classes required to run your application" is true but it also closes them immediately: usually they are all in JAR files too, which reduces the load considerably. The basic issue is not the number of connections, it is the obvious resource leak that is present.

andih Over a year ago

@EJP You are right that the class loader closes (should) all resources immediately. In some special circumstances usually when the system is under heavy load it may occur that accessing a class for the first time is the reason that such an Exception occurs which usually does not appear. I also think that you are right with your assessment that there exists a resource leak. But at the moment I don't know enough about the environment, load, ... which causes the Problem

Collectives™ on Stack Overflow

java.net.SocketException: java.net.SocketException: Too many open files

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related