Too many files open error

Networking/Security Forums -> UNIX // GNU/Linux

Author: vjy PostPosted: Tue Oct 05, 2004 8:08 am    Post subject: Too many files open error
    ----
Hi,
I am using fedora with apache 2.0.48 as web server and JBoss 3.2.3 as my application server. When I deployed my application in production environment, it started giving this error and then the application server crashes.

This is my application server log:

2004-10-05 00:10:14,575 ERROR [org.jboss.web.localhost.Engine] Exception:
Error running /usr/lib/jvm/java-1.4.2-sun-1.4.2.02/bin/javac compiler
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:451)
at org.apache.tools.ant.taskdefs.compilers.JavacExternal.execute(JavacExternal.java:81)
at org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:842)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:682)
at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:317)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:370)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:473)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:190)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at com.cn.eot.ui.filters.EOTSessionChecker.doFilter(EOTSessionChecker.java:4Cool
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: java.io.IOException: Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:143)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:646)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:448)
... 54 more
--- Nested Exception ---
java.io.IOException: java.io.IOException: Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:143)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:646)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:448)
at org.apache.tools.ant.taskdefs.compilers.JavacExternal.execute(JavacExternal.java:81)
at org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:842)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:682)
at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:317)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:370)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:473)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:190)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at com.cn.eot.ui.filters.EOTSessionChecker.doFilter(EOTSessionChecker.java:4Cool
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)


----------


This is my web server log:

[Tue Oct 05 13:57:05 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:05 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:07 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:07 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:07 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:08 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:08 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:08 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:08 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:08 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:08 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:59:36 2004] [error] jk2_channel_apr_send send failed 32 Broken pipe
[Tue Oct 05 13:59:36 2004] [error] ajp13.service() Error sending initial post -3 32 0
[Tue Oct 05 13:59:42 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:59:42 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:59:42 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
---------

Please tell me what to do. Its very critical.
Thanks for any help

Author: hugoLocation: Netherlands, Europe PostPosted: Tue Oct 05, 2004 8:33 am    Post subject:
    ----
For some reason the machine seems to run out of available file handles. On Linux, this maximum can be manually overridden; the number of maximum open file is kept in /proc/sys/fs/file-max. If you do a
Code:
~# cat /proc/sys/fs/file-max

..it will give you the number it is currently configured as.

To modify the number, do the following:
Code:
~# echo "65535" > /proc/sys/fs/file-max

This would set the maximum number of open files, not-suprisingly, to 65535. Wink (Maybe the 65535 amount is a bit overkill, though.)

Hope this helps.

Author: vjy PostPosted: Tue Oct 05, 2004 8:58 am    Post subject:
    ----
Thanks hugo,

I already gave a very high value for that. So when I type

cat /proc/sys/fs/file-max
it gives the value as 131072. But in the file /etc/security/limits.conf, I have the following

#soft nofile 65536 # Commented by me today
#hard nofile 65536 # Commented by me today
soft nproc 16384
hard nproc 16384

soft nofile 20000 # Added by me today
hard nofile 20000 # Added by me today
-------------

So I increased the no to 20000. But now I still didnt get the problem, but is such a high value required? What is the reason for this happening, i.e why too many files are being opened? I feel 65536 is too large.

But after chaning the /etc/security/limits.conf file, I didnt reboot the server. Is it necessary to reboot? Please advice.

p.s also this problem doesnt come in windows environment when we tested using load.

Thanks

Author: vjy PostPosted: Wed Oct 06, 2004 6:28 am    Post subject:
    ----
Hi guys,
I have solved the problem. There is just a single thing to be added in the databse file in my application server. The line should look like this

<connection-url>jdbc:microsoft:sqlserver://x.x.x.x:1433;DatabaseName=XXX;SelectMethod=cursor</connection-url>

Previously it was like

<connection-url>jdbc:microsoft:sqlserver://x.x.x.x:1433;DatabaseName=XXX</connection-url>

That was the problem, since the connections were not being released to the database pool.
Thanks

Author: arken PostPosted: Wed Oct 06, 2004 4:17 pm    Post subject:
    ----
Thanks for posting your fix.



Networking/Security Forums -> UNIX // GNU/Linux


output generated using printer-friendly topic mod, All times are GMT + 2 Hours

Page 1 of 1

Powered by phpBB 2.0.x © 2001 phpBB Group