• RSS
  • Twitter
  • FaceBook

Security Forums

Log in

FAQ | Search | Usergroups | Profile | Register | RSS | Posting Guidelines | Recent Posts

linux problems [too many open files]

Users browsing this topic:0 Security Fans, 0 Stealth Security Fans
Registered Security Fans: None
Post new topic   Reply to topic   Printer-friendly version    Networking/Security Forums Index -> UNIX // GNU/Linux

View previous topic :: View next topic  
Author Message
chris
Forum Fanatic
Forum Fanatic


Joined: 18 Apr 2002
Posts: 16777201
Location: ~/security-forums

Offline

PostPosted: Fri Nov 22, 2002 11:28 am    Post subject: linux problems [too many open files] Reply with quote

Ive had this problem a few times now, the box has locked out the pop3 daemon totally so noone can check mail.

Here is the tail of the exim_paniclog
Quote:

Too many open files in system

Then kern.log
Quote:

Nov 14 21:46:23 bleh kernel: VFS: file-max limit 4096 reached
Nov 15 10:11:07 bleh kernel: Unable to load interpreter /lib/ld-linux.so.2
Nov 15 10:11:09 bleh last message repeated 2 times
Nov 15 10:12:28 bleh last message repeated 8 times


When this happens even doing commands such as ls bring up the too many open files in system. Its a highish spec box which has previously been up for months with no problems, only recent additions are the latest apache and openSSL.

When this happened the wierdest thing is nothing is shown in top as looking dodgy so still havent managed to narrow this down.

Ive found similar probs through google with the max files

/proc/sys/fs/file-max

so this has been increased from 4k to 16k.

Problem is I cant see how that many files can be open on the system !!

Any help appreciated


Last edited by chris on Mon Nov 25, 2002 11:18 pm; edited 1 time in total
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger
ShaolinTiger
Forum Fanatic
Forum Fanatic


Joined: 18 Apr 2002
Posts: 16777215
Location: Kuala Lumpur, Malaysia

Offline

PostPosted: Fri Nov 22, 2002 11:35 am    Post subject: Reply with quote

Check it, it's probably gone back down again..

echo 32768 > /proc/sys/fs/file-max

Found this too

"for dynamic updates, use sysctl and put it in rc.local. If you want it to be
permanent, you can recompile the kernel (this way, you can include the
#define NR_TASKS change).

on my case, i just used the file-max and inode-max kernel settings. It would have been better if NR_TASKS was included in sysctl params Smile"

I think it's probably a limitation/bug in VFS.

Maybe someone else can enlighten you a little more.
Back to top
View user's profile Send private message Visit poster's website
chris
Forum Fanatic
Forum Fanatic


Joined: 18 Apr 2002
Posts: 16777201
Location: ~/security-forums

Offline

PostPosted: Fri Nov 22, 2002 12:12 pm    Post subject: Reply with quote

chris@anfield:/var/log$ cat /proc/sys/fs/file-max
16384

Sorry forgot to mention these are already added to rc.d

# Increase open file limit (from 4096 files
echo 16384 > /proc/sys/fs/file-max
# Plus more inodes
echo 65536 > /proc/sys/fs/inode-max

Since having these changed and the whole machine being restarted its still done the too many open files.

Although I do want a quick fix would also like to see how there could be this many open files at any one time, doesnt seem possible to what the box is actually doing Smile
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger
ShaolinTiger
Forum Fanatic
Forum Fanatic


Joined: 18 Apr 2002
Posts: 16777215
Location: Kuala Lumpur, Malaysia

Offline

PostPosted: Fri Nov 22, 2002 12:25 pm    Post subject: Reply with quote

lsof | more Wink

Have a look at ps, strace, tree etc

But lsof is the main one you want.

And there will be a lot of files open.
Back to top
View user's profile Send private message Visit poster's website
chris
Forum Fanatic
Forum Fanatic


Joined: 18 Apr 2002
Posts: 16777201
Location: ~/security-forums

Offline

PostPosted: Fri Nov 22, 2002 12:34 pm    Post subject: Reply with quote

Was looking for that type of command

lsof doesnt exist though (slackware)

Looked through processes

Nothing dodgy on there
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger
ShaolinTiger
Forum Fanatic
Forum Fanatic


Joined: 18 Apr 2002
Posts: 16777215
Location: Kuala Lumpur, Malaysia

Offline

PostPosted: Fri Nov 22, 2002 12:43 pm    Post subject: Reply with quote

Sourceforge is the saviour:

http://freshmeat.net/projects/lsof/

ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/

If you have no reverse DNS try here:

ftp://ftp.cerias.purdue.edu/pub/tools/unix/sysutils/lsof
ftp://ftp.cert.dfn.de/pub/tools/admin/lsof
ftp://ftp.cetis.hvu.nl/pub/lsof
ftp://ftp.crc.doc.ca/packages/lsof
ftp://ftp.fu-berlin.de/pub/unix/tools/lsof
ftp://ftp.sunet.se/pub/unix/admin/lsof
ftp://ftp.tau.ac.il/pub/unix/admin
ftp://ftp.tu-darmstadt.de/pub/sysadmin/lsof
ftp://ftp.tux.org/pub/sites/vic.cc.purdue.edu/tools/unix/lsof
ftp://ftp.uni-mainz.de/pub/misc/lsof
ftp://ftp.web.ad.jp/pub/UNIX/tools/lsof
ftp://gd.tuwien.ac.at/utils/admin-tools/lsof
ftp://sunsite.ualberta.ca/pub/Mirror/lsof
ftp://the.wiretapped.net/pub/security/host-security/lsof/
ftp://wuarchive.wustl.edu/packages/security/lsof
Back to top
View user's profile Send private message Visit poster's website
chris
Forum Fanatic
Forum Fanatic


Joined: 18 Apr 2002
Posts: 16777201
Location: ~/security-forums

Offline

PostPosted: Mon Nov 25, 2002 11:26 pm    Post subject: Reply with quote

Cheers

I scheduled a cron job to run this a few times and date the results:

Quote:

0,20,40, * * * * /home/chris/lsof/lsof > "openfiles`date '+%X'`"


Seems to be quite ..or is it ?

Quote:

root@bleh:/home/chris/lsof/ wc -l results

2809 results


Its not gone over the original 4000 odd limit yet since.

Most connections by far are taken up by gnu-pop3 followed by apache then mysqld.

1300 odd of these are pop3, anything else I can check, perhaps lowering timeouts or similar server side.
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger
Display posts from previous:   

Post new topic   Reply to topic   Printer-friendly version    Networking/Security Forums Index -> UNIX // GNU/Linux All times are GMT + 2 Hours
Page 1 of 1


 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Community Area

Log in | Register