Return to BSD News archive
Newsgroups: comp.os.386bsd.bugs
Path: sserve!newshost.anu.edu.au!munnari.oz.au!uunet!zaphod.mps.ohio-state.edu!magnus.acs.ohio-state.edu!csn!hellgate.utah.edu!fcom.cc.utah.edu!cs.weber.edu!terry
From: terry@cs.weber.edu (A Wizard of Earth C)
Subject: Re: VM problems w/unlimited memory?
Message-ID: <1993Mar21.002245.2246@fcom.cc.utah.edu>
Sender: news@fcom.cc.utah.edu
Organization: Weber State University (Ogden, UT)
References: <1o81joINNieh@usenet.INS.CWRU.Edu> <1993Mar18.183443.6397@fcom.cc.utah.edu> <1obpil$aq7@hal.gnu.ai.mit.edu>
Date: Sun, 21 Mar 93 00:22:45 GMT
Lines: 112
In article <1obpil$aq7@hal.gnu.ai.mit.edu> mycroft@hal.gnu.ai.mit.edu (Charles Hannum) writes:
>
>In article <1993Mar18.183443.6397@fcom.cc.utah.edu> terry@cs.weber.edu
>(A Wizard of Earth C) writes:
>>
>> Yes, it will cause problems on 386bsd -- it will also, however, cause
>> problems on SVR4 and AIX, both of which dynamically allocate their
>> per-process file table.
>
>Funny. It doesn't seem to be causing any problems on *this* box
>running AIX 3.2.
Use of descriptor 2000 will result in ~64K (2000*sizeof(struct file)) being
allocated to support the descriptor table *up to and including fd 2000*.
Use of unwarranted amounts of memory on AIX triggers process kills.
>> Allocation of all memory on an AIX 3.2 box will result not in a
>> crash, but in killing the process with the largest image size (rather
>> than the most recently run process or the one doing the excessive
>> allocation).
>
>Bzzzzzt. You don't understand the algorithm it uses at all.
Well, then it's apparently *random* murder of processes by a psychotic
kernel. In any case, the lack of swap shadowing all of main memory
results "swapsize + memsize" memory being used, rather than simply the
"swapsize" memory being used on BSD systems. The result is that
processes are killed. Obvesrvation of this process led me to believe
that this happened to the largest process; if this is wrong, then so
be it; it still happens.
>> I think any time the programmer makes assumptions about the kernel
>> architecture (as one is doing when one allocates a real high fd
>> number and assumes that fd's are going to be allocated in ascending
>> order by the "open()" call such that you are guaranteed that a real
>> high choice is a safe one), one has to be aware of it.
>
>No, he's going strictly by the book. If the kernel wants to say
>`Sorry, out of memory.', that's fine, but crashing is totally
>unacceptable. Whether this would cause a performance problem for bash
>or not is bash users' concern.
Agreed on the performance problem for bash being bash's problem; disagreed
that the use of an unnecessary amount of memory on some systems due to
inapropiate assumptions about architecture will not cause problems for
other non-bash-users on the sharing the same system. Again, bash certainly
isn't the largest offender.
I have never argued that crashing *was* acceptable. Certainly, an attempt
to allcate 68 terrabytes for a single processes open file table (which is
what triggers the vm bug and therefore the crash on 386BSD) when the number
of fd's has been "unlimited" in the bash shells parent process is very
unacceptable. A correction for bash, such as using fd numbers in shell
scripts as virtual fd lookup indices rather than fd's except for 0, 1, and
2, and the transitory use of /dev/tty, would not only avoid the problems
bash will have with an ENOMEM on it's dup2 (which it would NOT handle well
as currently implemented) would *also* avoid triggering the VM problem
until it was fixed. Of course the argument against this is that some bash
user tired of his system crashing will fix the vm problem and handle the
new error return from dup2 for all of us.
>What *you're* saying is that he should take the kernel architecture
>into consideration, when in fact he's doing something perfectly valid.
I disagree; bash isn't trapping the "ENOMEM" which would result from the
"corrected" 386BSD open/dup/dup2 calls, nor is it repicking an fd lower
until job control can function properly. In any implementation, the bash
code will be required to change. Preemptively fixing bash so that it
doesn't trigger the VM problem in an architecturally sensitive way will
also ensure a 64k savings in memory size on AIX and a larger savings on
systems allowing larger numbers of fd's, as well as making the VM problem
not impact bash users while it's being worked on.
>> There doesn't seem to be a good reason for this if the shell script
>> is in core in the shell; it should be closed after reading. Since
>> reading takes place entirely before execution, there is no conflict
>> with the shell script itself. In any event, a shell script is run by
>> a sub-shell,
>
>Look up `source' in the sh(1) man page, please.
If the fd numbers used in shell scripts were used as indices to an open
file table within bash itself rather than used as raw fd's, bash would not
be impacted by a source of a script using "virtual fd 3" when bash had "real
fd 3" allocated for it's own use.
The bitching about the VM system, hacking "unlimit" to do something other
than what it says it does, and complaining "Doctor, it hurts when I do this"
do not help to resolve the problems.
Even if no one wants to admit a problem in bash with bash making an
architectural assumption about file descriptor allocation when it tries to
use the largest available fd # and a problem with bash's use of the call
getdtablesize() to determine the largest allowed fd (as opposed to using
it to determine the current value of the soft limit component of the
RLIMIT_NOFILE resource limit, which is what it's documented as doing),
they must admit that if they don't jack the limit up and run bash, then
the problem won't manifest.
Terry Lambert
terry@icarus.weber.edu
terry_lambert@novell.com
---
Any opinions in this posting are my own and not those of my present
or previous employers.
--
-------------------------------------------------------------------------------
"I have an 8 user poetic license" - me
Get the 386bsd FAQ from agate.berkeley.edu:/pub/386BSD/386bsd-0.1/unofficial
-------------------------------------------------------------------------------