TechGenix and SolarWinds have partnered to provide a fully-functional, free 21-day trial version of SolarWinds ipMonitor, the WindowsNetworking.com Readers' Choice Award Winner for monitoring applications, servers, and network devices to all visitors who join Security Forums. Sign up to Security Forums and get your copy today! Existing members can pick up a copy from the Members Area.
Posted: Tue Oct 05, 2004 8:08 am Post subject: Too many files open error
Hi,
I am using fedora with apache 2.0.48 as web server and JBoss 3.2.3 as my application server. When I deployed my application in production environment, it started giving this error and then the application server crashes.
This is my application server log:
2004-10-05 00:10:14,575 ERROR [org.jboss.web.localhost.Engine] Exception:
Error running /usr/lib/jvm/java-1.4.2-sun-1.4.2.02/bin/javac compiler
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:451)
at org.apache.tools.ant.taskdefs.compilers.JavacExternal.execute(JavacExternal.java:81)
at org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:842)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:682)
at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:317)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:370)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:473)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:190)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at com.cn.eot.ui.filters.EOTSessionChecker.doFilter(EOTSessionChecker.java:4
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: java.io.IOException: Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:143)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:646)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:448)
... 54 more
--- Nested Exception ---
java.io.IOException: java.io.IOException: Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:143)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:646)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:448)
at org.apache.tools.ant.taskdefs.compilers.JavacExternal.execute(JavacExternal.java:81)
at org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:842)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:682)
at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:317)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:370)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:473)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:190)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at com.cn.eot.ui.filters.EOTSessionChecker.doFilter(EOTSessionChecker.java:4
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)
----------
This is my web server log:
[Tue Oct 05 13:57:05 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:05 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:07 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:07 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:07 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:08 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:08 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:08 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:08 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:08 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:08 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:59:36 2004] [error] jk2_channel_apr_send send failed 32 Broken pipe
[Tue Oct 05 13:59:36 2004] [error] ajp13.service() Error sending initial post -3 32 0
[Tue Oct 05 13:59:42 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:59:42 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:59:42 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
---------
Please tell me what to do. Its very critical.
Thanks for any help
Joined: 14 Jun 2003 Posts: 940 Location: Netherlands, Europe
Posted: Tue Oct 05, 2004 8:33 am Post subject:
For some reason the machine seems to run out of available file handles. On Linux, this maximum can be manually overridden; the number of maximum open file is kept in /proc/sys/fs/file-max. If you do a
Code:
~# cat /proc/sys/fs/file-max
..it will give you the number it is currently configured as.
To modify the number, do the following:
Code:
~# echo "65535" > /proc/sys/fs/file-max
This would set the maximum number of open files, not-suprisingly, to 65535. (Maybe the 65535 amount is a bit overkill, though.)
I already gave a very high value for that. So when I type
cat /proc/sys/fs/file-max
it gives the value as 131072. But in the file /etc/security/limits.conf, I have the following
#soft nofile 65536 # Commented by me today
#hard nofile 65536 # Commented by me today
soft nproc 16384
hard nproc 16384
soft nofile 20000 # Added by me today
hard nofile 20000 # Added by me today
-------------
So I increased the no to 20000. But now I still didnt get the problem, but is such a high value required? What is the reason for this happening, i.e why too many files are being opened? I feel 65536 is too large.
But after chaning the /etc/security/limits.conf file, I didnt reboot the server. Is it necessary to reboot? Please advice.
p.s also this problem doesnt come in windows environment when we tested using load.
Hi guys,
I have solved the problem. There is just a single thing to be added in the databse file in my application server. The line should look like this
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
(Maybe the 65535 amount is a bit overkill, though.)