linux problems [too many open files]

Networking/Security Forums -> UNIX // GNU/Linux

Author: chrisLocation: ~/security-forums PostPosted: Fri Nov 22, 2002 11:28 am    Post subject: linux problems [too many open files]
    ----
Ive had this problem a few times now, the box has locked out the pop3 daemon totally so noone can check mail.

Here is the tail of the exim_paniclog
Quote:

Too many open files in system

Then kern.log
Quote:

Nov 14 21:46:23 bleh kernel: VFS: file-max limit 4096 reached
Nov 15 10:11:07 bleh kernel: Unable to load interpreter /lib/ld-linux.so.2
Nov 15 10:11:09 bleh last message repeated 2 times
Nov 15 10:12:28 bleh last message repeated 8 times


When this happens even doing commands such as ls bring up the too many open files in system. Its a highish spec box which has previously been up for months with no problems, only recent additions are the latest apache and openSSL.

When this happened the wierdest thing is nothing is shown in top as looking dodgy so still havent managed to narrow this down.

Ive found similar probs through google with the max files

/proc/sys/fs/file-max

so this has been increased from 4k to 16k.

Problem is I cant see how that many files can be open on the system !!

Any help appreciated


Last edited by chris on Mon Nov 25, 2002 11:18 pm; edited 1 time in total

Author: ShaolinTigerLocation: Kuala Lumpur, Malaysia PostPosted: Fri Nov 22, 2002 11:35 am    Post subject:
    ----
Check it, it's probably gone back down again..

echo 32768 > /proc/sys/fs/file-max

Found this too

"for dynamic updates, use sysctl and put it in rc.local. If you want it to be
permanent, you can recompile the kernel (this way, you can include the
#define NR_TASKS change).

on my case, i just used the file-max and inode-max kernel settings. It would have been better if NR_TASKS was included in sysctl params Smile"

I think it's probably a limitation/bug in VFS.

Maybe someone else can enlighten you a little more.

Author: chrisLocation: ~/security-forums PostPosted: Fri Nov 22, 2002 12:12 pm    Post subject:
    ----
chris@anfield:/var/log$ cat /proc/sys/fs/file-max
16384

Sorry forgot to mention these are already added to rc.d

# Increase open file limit (from 4096 files
echo 16384 > /proc/sys/fs/file-max
# Plus more inodes
echo 65536 > /proc/sys/fs/inode-max

Since having these changed and the whole machine being restarted its still done the too many open files.

Although I do want a quick fix would also like to see how there could be this many open files at any one time, doesnt seem possible to what the box is actually doing Smile

Author: ShaolinTigerLocation: Kuala Lumpur, Malaysia PostPosted: Fri Nov 22, 2002 12:25 pm    Post subject:
    ----
lsof | more Wink

Have a look at ps, strace, tree etc

But lsof is the main one you want.

And there will be a lot of files open.

Author: chrisLocation: ~/security-forums PostPosted: Fri Nov 22, 2002 12:34 pm    Post subject:
    ----
Was looking for that type of command

lsof doesnt exist though (slackware)

Looked through processes

Nothing dodgy on there

Author: ShaolinTigerLocation: Kuala Lumpur, Malaysia PostPosted: Fri Nov 22, 2002 12:43 pm    Post subject:
    ----
Sourceforge is the saviour:

http://freshmeat.net/projects/lsof/

ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/

If you have no reverse DNS try here:

ftp://ftp.cerias.purdue.edu/pub/tools/unix/sysutils/lsof
ftp://ftp.cert.dfn.de/pub/tools/admin/lsof
ftp://ftp.cetis.hvu.nl/pub/lsof
ftp://ftp.crc.doc.ca/packages/lsof
ftp://ftp.fu-berlin.de/pub/unix/tools/lsof
ftp://ftp.sunet.se/pub/unix/admin/lsof
ftp://ftp.tau.ac.il/pub/unix/admin
ftp://ftp.tu-darmstadt.de/pub/sysadmin/lsof
ftp://ftp.tux.org/pub/sites/vic.cc.purdue.edu/tools/unix/lsof
ftp://ftp.uni-mainz.de/pub/misc/lsof
ftp://ftp.web.ad.jp/pub/UNIX/tools/lsof
ftp://gd.tuwien.ac.at/utils/admin-tools/lsof
ftp://sunsite.ualberta.ca/pub/Mirror/lsof
ftp://the.wiretapped.net/pub/security/host-security/lsof/
ftp://wuarchive.wustl.edu/packages/security/lsof

Author: chrisLocation: ~/security-forums PostPosted: Mon Nov 25, 2002 11:26 pm    Post subject:
    ----
Cheers

I scheduled a cron job to run this a few times and date the results:

Quote:

0,20,40, * * * * /home/chris/lsof/lsof > "openfiles`date '+%X'`"


Seems to be quite ..or is it ?

Quote:

root@bleh:/home/chris/lsof/ wc -l results

2809 results


Its not gone over the original 4000 odd limit yet since.

Most connections by far are taken up by gnu-pop3 followed by apache then mysqld.

1300 odd of these are pop3, anything else I can check, perhaps lowering timeouts or similar server side.



Networking/Security Forums -> UNIX // GNU/Linux


output generated using printer-friendly topic mod, All times are GMT + 2 Hours

Page 1 of 1

Powered by phpBB 2.0.x © 2001 phpBB Group