Xinetd and file descriptor limits?

[John Minnihan]

I am experiencing an error condition involving use of xinetd and apparent
file descriptor limits. To summarize, when I start xinetd and parse
numerous service files (as found in /etc/xinetd.d/), I receive the
following error in the syslog:

---<snip>---
Jan 4 14:43:03 estes xinetd[1852]: Service sserver-4121 disabled because
of lack of file descriptors
Jan 4 14:43:03 estes xinetd[1852]: xinetd Version 2.3.12 started with
libwrap loadavg options compiled in.
Jan 4 14:43:03 estes xinetd[1852]: Started working: 1016 available services
---<snip>---

I am staring numerous sservers (an ssl'ified version of pserver, Corey
Minyard's version) - in fact, over a thousand of these alone.
Corresponding lines do exist in /etc/services (one per sserver port). The
number in the service name relates to the port it listens on. I begin
assigning sserver ports at 3025, and climb from there. Ports up to 4020
listen properly. I have a minimal set of services running in the
privileged port space, among them ssh, httpd, httpsd, mysql, postgres and
portsentry.

System particulars are:

Kernel .......... 2.4.18-14 (Redhat 8 base distro)
xinetd........... 2.3.12 compiled with libwrap & loadavg
/proc/sys/fs/file-max ............. various settings above 1024
ulimit -n ........ various settings well above 1024 (once at 200000)
Web server ........ Apache 1.3.9

I have also placed in the xinted startup script two 'ulimit' statements
that are intended to increase the fd limit for the user that starts
xinetd. The statements are at the top of the script before any other
action takes place. Those statements are:

ulimit -Hn 65535
ulimit -Sn 8192

I have experienced this error on two separate servers, each running
originally identical configurations, and then tweaking on one them
versions of xinetd, the kernel and apache such that the error presents in
the very latest of all three:

xinetd (originally 2.3.10) .... 2.3.12
kernel (originally 2.4.20-8) ...2.6.0
apache (originally 1.3.9) ......2.0.40

I am running all these commands as root. My apache user is non-root. I
have also run xinetd with '-d', but haven't been able to capture the
output. It seems that the '-d' overrides atemmpts to redirect to a log.
How is the output of '-d' captured for review?

At the time the error presents, a 'lsof | wc -l' shows 5653 lines of
output. I believe this maps directly to the total of both open ports and
files, right? That is well below the value returned from 'ulimit -n', as
well as below the value in /proc/sys/fs/file-max.

I suspect I am hitting the system or user default of 1024 file
descriptors, but I am curious why that number (1024) doesn't quite match
the number of services successfully started, 1016. Are these in fact
related? Is the error message believed to be informative? I am stumped
as to what else to review. Is there another setting that controls this &
I've simply missed it?

Thanks for any assistance.

[Steve G]

Post by John Minnihan
I am experiencing an error condition involving use of xinetd
and apparent file descriptor limits.

It sounds like it.

Post by John Minnihan
I have also placed in the xinted startup script two 'ulimit'
statements that are intended to increase the fd limit for the
user that starts xinetd.

I also think there is another limit settable by sysctl. If you
get past this, I think xinetd only requests 1024 descriptors. You
would need to tweek it to set the rlimit higher for the per
process limit.

Post by John Minnihan
How is the output of '-d' captured for review?

I startup x and look at it though a terminal window after setting
the number of lines for it to buffer.

Post by John Minnihan
I suspect I am hitting the system or user default of 1024 file
descriptors, but I am curious why that number (1024) doesn't
quite match the number of services successfully started, 1016.

xinetd uses a couple for internal messaging, stdin,stdout,stderr
are all set to /dev/null and then glibc also leaks descriptors in
pmap_set, and then error logging.

Post by John Minnihan
I am stumped as to what else to review. Is there another setting
that controls this & I've simply missed it?

Take a look in init.c. Look for setfd_limit(). You may want to
tweek that function to request more descriptors.

-Steve Grubb

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus

[John Minnihan]

Thanks for the detailed reply. See inline comments/questions below.

Post by Steve G
I also think there is another limit settable by sysctl. If you
get past this, I think xinetd only requests 1024 descriptors. You
would need to tweek it to set the rlimit higher for the per
process limit.

Can you provide any guidance on tweaking the rlimit? I can grok new
things pretty well, but a starting point is always handy.

Post by Steve G

Post by John Minnihan
How is the output of '-d' captured for review?

I startup x and look at it though a terminal window after setting
the number of lines for it to buffer.

Ah, ok.

Post by Steve G

Post by John Minnihan
I suspect I am hitting the system or user default of 1024 file
descriptors, but I am curious why that number (1024) doesn't
quite match the number of services successfully started, 1016.

xinetd uses a couple for internal messaging, stdin,stdout,stderr
are all set to /dev/null and then glibc also leaks descriptors in
pmap_set, and then error logging.

So that's a reasonable number. Good.

Post by Steve G
Take a look in init.c. Look for setfd_limit(). You may want to
tweek that function to request more descriptors.

I took a look at init.c and saw the function, but must admit I haven't any
idea how to tweak it. I don't see where it gets it value(s) - from one of
the includes?

[Steve G]

Post by John Minnihan
Can you provide any guidance on tweaking the rlimit?

Only change the code inside the RLIMIT_NOFILE ifdef. In that
section of code, set the values of rl.rlim_cur & rl.rlim_max to
what you want before maxfd is initialized and call setrlimit.

-Steve Grubb

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus

[John Minnihan]

I poked around a bit and came up with a solution. Let me know if you
think this can or will cause undesirable side affects-

I added a #define FD_SETSIZE 8192 to the top of init.c and recompiled.
All services now start error-free.

Post by Steve G
Take a look in init.c. Look for setfd_limit(). You may want to
tweek that function to request more descriptors.
-Steve Grubb

[Steve G]

Post by John Minnihan
I added a #define FD_SETSIZE 8192 to the top of init.c and
recompiled. All services now start error-free.

Try reloading xinetd by sending it a SIGHUP. If that passes I
think you can use what you have.

-Steve Grubb

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus

[John Minnihan]

Behaves as expected. Thanks.

Post by Steve G

Post by John Minnihan
I added a #define FD_SETSIZE 8192 to the top of init.c and
recompiled. All services now start error-free.

Try reloading xinetd by sending it a SIGHUP. If that passes I
think you can use what you have.
-Steve Grubb
__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus
_______________________________________________
xinetd mailing list
http://www.xinetd.org/mailman/listinfo/xinetd