• RSS
  • Twitter
  • FaceBook

Security Forums

Log in

FAQ | Search | Usergroups | Profile | Register | RSS | Posting Guidelines | Recent Posts

Too many files open error

Users browsing this topic:0 Security Fans, 0 Stealth Security Fans
Registered Security Fans: None
Post new topic   Reply to topic   Printer-friendly version    Networking/Security Forums Index -> UNIX // GNU/Linux

View previous topic :: View next topic  
Author Message
vjy
Just Arrived
Just Arrived


Joined: 05 May 2004
Posts: 0


Offline

PostPosted: Tue Oct 05, 2004 8:08 am    Post subject: Too many files open error Reply with quote

Hi,
I am using fedora with apache 2.0.48 as web server and JBoss 3.2.3 as my application server. When I deployed my application in production environment, it started giving this error and then the application server crashes.

This is my application server log:

2004-10-05 00:10:14,575 ERROR [org.jboss.web.localhost.Engine] Exception:
Error running /usr/lib/jvm/java-1.4.2-sun-1.4.2.02/bin/javac compiler
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:451)
at org.apache.tools.ant.taskdefs.compilers.JavacExternal.execute(JavacExternal.java:81)
at org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:842)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:682)
at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:317)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:370)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:473)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:190)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at com.cn.eot.ui.filters.EOTSessionChecker.doFilter(EOTSessionChecker.java:4Cool
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: java.io.IOException: Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:143)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:646)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:448)
... 54 more
--- Nested Exception ---
java.io.IOException: java.io.IOException: Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:143)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:646)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.executeExternalCompile(DefaultCompilerAdapter.java:448)
at org.apache.tools.ant.taskdefs.compilers.JavacExternal.execute(JavacExternal.java:81)
at org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:842)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:682)
at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:317)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:370)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:473)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:190)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at com.cn.eot.ui.filters.EOTSessionChecker.doFilter(EOTSessionChecker.java:4Cool
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)


----------


This is my web server log:

[Tue Oct 05 13:57:05 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:05 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:06 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:06 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:06 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:07 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:07 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:07 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:08 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:08 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:08 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:57:08 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:57:08 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:57:08 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
[Tue Oct 05 13:59:36 2004] [error] jk2_channel_apr_send send failed 32 Broken pipe
[Tue Oct 05 13:59:36 2004] [error] ajp13.service() Error sending initial post -3 32 0
[Tue Oct 05 13:59:42 2004] [error] channelApr.receive(): Error receiving message body -1 11
[Tue Oct 05 13:59:42 2004] [error] workerEnv.processCallbacks() Error reading reply
[Tue Oct 05 13:59:42 2004] [error] ajp13.service() ajpGetReply recoverable error 120000
---------

Please tell me what to do. Its very critical.
Thanks for any help
Back to top
View user's profile Send private message
hugo
Forum Fanatic
Forum Fanatic


Joined: 14 Jun 2003
Posts: 16777215
Location: Netherlands, Europe

Offline

PostPosted: Tue Oct 05, 2004 8:33 am    Post subject: Reply with quote

For some reason the machine seems to run out of available file handles. On Linux, this maximum can be manually overridden; the number of maximum open file is kept in /proc/sys/fs/file-max. If you do a
Code:
~# cat /proc/sys/fs/file-max

..it will give you the number it is currently configured as.

To modify the number, do the following:
Code:
~# echo "65535" > /proc/sys/fs/file-max

This would set the maximum number of open files, not-suprisingly, to 65535. Wink (Maybe the 65535 amount is a bit overkill, though.)

Hope this helps.
Back to top
View user's profile Send private message
vjy
Just Arrived
Just Arrived


Joined: 05 May 2004
Posts: 0


Offline

PostPosted: Tue Oct 05, 2004 8:58 am    Post subject: Reply with quote

Thanks hugo,

I already gave a very high value for that. So when I type

cat /proc/sys/fs/file-max
it gives the value as 131072. But in the file /etc/security/limits.conf, I have the following

#soft nofile 65536 # Commented by me today
#hard nofile 65536 # Commented by me today
soft nproc 16384
hard nproc 16384

soft nofile 20000 # Added by me today
hard nofile 20000 # Added by me today
-------------

So I increased the no to 20000. But now I still didnt get the problem, but is such a high value required? What is the reason for this happening, i.e why too many files are being opened? I feel 65536 is too large.

But after chaning the /etc/security/limits.conf file, I didnt reboot the server. Is it necessary to reboot? Please advice.

p.s also this problem doesnt come in windows environment when we tested using load.

Thanks
Back to top
View user's profile Send private message
vjy
Just Arrived
Just Arrived


Joined: 05 May 2004
Posts: 0


Offline

PostPosted: Wed Oct 06, 2004 6:28 am    Post subject: Reply with quote

Hi guys,
I have solved the problem. There is just a single thing to be added in the databse file in my application server. The line should look like this

<connection-url>jdbc:microsoft:sqlserver://x.x.x.x:1433;DatabaseName=XXX;SelectMethod=cursor</connection-url>

Previously it was like

<connection-url>jdbc:microsoft:sqlserver://x.x.x.x:1433;DatabaseName=XXX</connection-url>

That was the problem, since the connections were not being released to the database pool.
Thanks
Back to top
View user's profile Send private message
arken
Just Arrived
Just Arrived


Joined: 02 Oct 2003
Posts: 1


Offline

PostPosted: Wed Oct 06, 2004 4:17 pm    Post subject: Reply with quote

Thanks for posting your fix.
Back to top
View user's profile Send private message
Display posts from previous:   

Post new topic   Reply to topic   Printer-friendly version    Networking/Security Forums Index -> UNIX // GNU/Linux All times are GMT + 2 Hours
Page 1 of 1


 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Community Area

Log in | Register