Subject: Linux-Development Digest #843
From: Digestifier <Linux-Development-Request@senator-bedfellow.MIT.EDU>
To: Linux-Development@senator-bedfellow.MIT.EDU
Reply-To: Linux-Development@senator-bedfellow.MIT.EDU
Date:     Sun, 19 Jun 94 00:13:12 EDT

Linux-Development Digest #843, Volume #1         Sun, 19 Jun 94 00:13:12 EDT

Contents:
  mmap, mmalloc(?) kernel 1.0 (David Pearson)
  Re: DOSEMU and Novell (Mark A. Davis)
  Re: Yggdrasil Linux with GNU C++ ?? (Eckard Kopatzki)
  Re: Filesystem semantics protecting meta data ... and users data (Totally Lost)
  Re: Filesystem semantics protecting meta data ... and users data (Totally Lost)
  Re: Filesystem semantics protecting meta data ... and users data (Dan Swartzendruber)
  Re: Yggdrasil CD-ROM question (Adam J. Richter)
  1.1.20 pdksh & csh problems (Stephen Beaton-Snook)
  Re: 1.1.20 pdksh & csh problems (Seo Hongwon)

----------------------------------------------------------------------------

From: David@royalp.demon.co.uk (David Pearson)
Subject: mmap, mmalloc(?) kernel 1.0
Reply-To: David@royalp.demon.co.uk
Date: Sat, 18 Jun 1994 16:26:00 +0000

Hi,

I was rooting around the info pages i got from mcc, or tsx-11 (can't 
remember) & found pages on functions called mmalloc, mfree etc which
implement memory mapped-file allocation arenas. I thought WOW! linux must
have memory mapped files, but alas after more investigation I couldn't find
the headers for mmalloc, & it isnt in the lib. The (underlying?) mmap 
functions in libc don't implement writing back to a demand paged file.

Is anyone aware of plans to implement the mmalloc functions? Could anyone
point me in the right direction to find out?

Cheers
Dave
-- 
............................
: Dave Pearson             :
: david@royalp.demon.co.uk : 
:..........................: 

------------------------------

Crossposted-To: comp.os.linux.help
From: mark@taylor.infi.net (Mark A. Davis)
Subject: Re: DOSEMU and Novell
Date: Sat, 18 Jun 1994 21:09:52 GMT

>Subject: Re: DOSEMU and Novell

Novell is a *company* which sells Unix and Netware, among other things...

-- 
  /--------------------------------------------------------------------------\
  | Mark A. Davis    | Lake Taylor Hospital | Norfolk, VA (804)-461-5001x431 |
  | Sys.Administrator|  Computer Services   | mark@taylor.infi.net           |
  \--------------------------------------------------------------------------/

------------------------------

From: root@stevie.isar.muc.de (Eckard Kopatzki)
Subject: Re: Yggdrasil Linux with GNU C++ ??
Date: 18 Jun 1994 23:09:43 GMT

In article <12415113@comet.sb.sub.de>, sysop@comet.sb.sub.de writes:
> Hi,
> i have seen here in a german store a Distribution of
> Linux 1.1 from Yggdrasil named PLUG and PLAY LINUX !
> 
> I looked at the package, and ask the dealer, but it wont
> help me to find out: Is this Linux Distribution delivered with
> GNU C++ - Compiler ?
> 

The Distribution-HOWTO says:

        Yggdrasil Plug-and-Play Linux is a complete CD-ROM distribution
        of the Linux operating system. It includes a great deal of
        software---nearly every package that you would expect to find on
        a complete UNIX system is available. A complete file list is
        available via FTP from yggdrasil.com.

And on the Yggdrasil Fall '93 distribution there has been a C++ compiler.

-- 
Eckard Kopatzki           Internet eko@isar.muc.de
Therese-Giehse-Allee 53     CompuServe 100024,2175
D-81739 Muenchen, Germany     Voice +49-89-6378103

------------------------------

Crossposted-To: comp.benchmarks,comp.sys.sun.admin,comp.security.unix
From: idletime@netcom.com (Totally Lost)
Subject: Re: Filesystem semantics protecting meta data ... and users data
Date: Sat, 18 Jun 1994 23:05:23 GMT

I really want this thread to continue in comp.os.research ... I responded 
to this one here since it was a weak pot shot at the meta data issue.

In article <1994Jun18.034642.28129@rpp386>,
John F. Haugh II <jfh@rpp386.cactus.org> wrote:
>In article <idletimeCrGqzv.L9L@netcom.com> idletime@netcom.com (Totally Lost) writes:
>>I will go so far as to say that if a file contains any data other
>>than written at a crash point it will greatly increase the
>>recovery applications work for that file type. I will go on
>>to say it should be required of any filesystem to indicate
>>which files may be corrupt after a crash as a minimally
>>required function to aid the sysadm and production staff
>>to get the system consistant again short of rolling back
>>to a known checkpoint..
>
>I think that part of the reason for the rejection of your ideas may be that
>they are projecting requirements that don't really seem to exist.

I have already noted that most system admins like yourself really don't care
that users files or production data may be corrupt after a crash, that
really is somebody else problem isn't it? ... or is it?

For people that really don't give a shit about the quality of the services
they provide it's easy to dismiss data corruption posibilites at every crash.

I ask ... is this really a professional way to address data corruption?

Heck no ...

>I administer a number of systems and I don't make any efforts, beyond the
>automated FSCK, to verify that all active files at the time of the crash
>are valid.  Even in an EDP environment I've seldom seen this behavior.

FSCK does not verify all files active at the time of a crash as valid.
It only verifies that filesystem metadata is not corrupted, which
would in turn cause more even more corruption if not checked.

The scary term here is "seldom seen" from somebody that isn't even
looking. Sysadmins see it all the time, but not being trained or required
to locate the cause just dismiss it as one of those unanswered questions
in life. I think this is bull crap.

Of course, most sysadmins just love to point to the application programmers
troubles tring to recover data and suggest they should have written the program
to handle it better ... not my job ... :)

>The operations staff typically has a firm understanding as to which files
>are safe and which aren't.

I have yet to see this true for 95% of the production UNIX systems in the
field ... that have no technical person watching over the system. Even the
operations staff at unix providers I have consulted for do not attempt to
issolate and examine files written  within a sync period of a crash
and check them for corruption. This statement is largely false for the
vast majority of sites in the world. Show me otherwise ...

>As an aside, the validity of data involves
>much more than the consistency of the data from a write() call
>perspective.  The filesystem will never assure more than that level of
>validity.

I disagree strongly!

Validity needs to be addressed at three levels.

        1) the data which appears on disk is the data WAS written.

        2) the data written actually ends up on the disk.

        3) the data which appears on the disk is consistant with other data.

The point of this thread was to assure item 1. That minimal level is not
currently achieved by UNIX filesystems based upon V6/V7/SVRx/BSD.
Because of both data corruption and security related issues, atleast this
level of validity must be required of all UNIX vendors.

A side point of the thread is that 2 is also required to address
recovery issues after a crash ... you must have some way to identify
those files that may not be complete. I believe that this should also
be required of all UNIX vendors.

Lastly I also believe the 3 which reqires implementation of multifile
multiprocess commit style updates should exist. I said several times
I think this belongs in every UNIX filesystem, but I stop short of
requiring it of every vendor since it clearly is niche market, but
a significant one in sales volume.

If I missed your point please clearify.

>Furthermore, I think your assertion that an async ordered write filesystem
>is going to be faster a priori than the competitive mixed designs that
>might be imagined seems a tad bold.

GEEZ ... cann't you even do the basics?

Thats a pretty bold statement to make without presenting (or thinking about)
a simple model for it ... so let's do your work (thinking) for you ...

Current sync metadata filesystems require as much as two physical I/O's for
every block/cluster written - this yields MUCH less 50% efficency due to
rotational latency and seek losses ... 5-10% or less typical. A good I/O
system will combine the block/cluster data writes making the sync metadata I/O
the vast majority of the I/O stream - few do this though. So in a 1K filesystem
best case we have 256 sync metadata writes of 1K and a 256K data cluster
resulting in probably 260+ revs of the disk (about 4.3 secs on a typical
3600RPM disk with 32K per track)

I require the data be written before the meta data. This can be implemented
several ways:

        1) Sync data, Sync Metadata - worse than Async data, Sync Metadata
           under 5% efficendy. No sane person would do this if another
           workable choice exists. 512 revs, 8.7 secs.

        2) Async queue data within a lowest level indirect block,
           block on completion of all queue blocks; Sync Metadata
           Requires one sync write per block of indirect metadata,
           resulting in BSIZE=1k filesystem clustered writes with
           better than 75% efficency for most drives. Result vary
           depending on a number of variables but are always better than the
           previous two solutions. 5 revs for .08 secs.

Other schemes exist, including the threaded one with fully async ordered
writes which are better than the above three.

The point is that the Berkeley Sync write solution is stupidly inefficent.

Nobody implements 2 that I am aware of which meets the minimal requirement
for data integrity at a reasonable cost - because it's difficult to implement
inside exist designs. The threaded design is simpler and can be simply
extended to handle all the ordering issues besides just this case.

>One problem which comes to mind is
>that you have now created an ordering in which all metadata blocks sort
>after all data blocks.

No - you do ... see the reply to Alan at SGI and the above.

>And this ordering can create thrashing as the
>drive steps back out to where the data is stored (wherever that may be)
>and back to where the metadata is stored (pick a cylinder, any cylinder).

This will happen at some frequency regardless ... with the Berkeley
Sync Metadata solution it is very frequent ... with 2 above or threaded
ordered async writes it seldom occurs.

You clearly do not know how things currently work, given these dumb statements
why don't you stop wasting our time and do your homework before posting
in the future?

>My personal inclination is towards journalled filesystems, though the
>state of that art seems absurd in that the I/O to the journal location
>frequently appears to create its own problems.

They have their own problems ... especially with relatively poor locality.
Given your knowlege of existing filesystem internals, it's not clear
you made that decision on any real informed basis.

>At this point the suggestion that you prototype a filesystem based on
>these concepts seems in order.  While it is true that the fully functional
>design might take 3 or 4 programmer-years to implement, prototypical
>implmentations should well within reach for one or two warm bodies on a
>part time basis.

That *I* prototype it? How generous you are with other peoples time. Who do
you expect to pay my mortage and feed my kids for the year while I do this
by myself ... or to pay the salaries of a couple junior's to help for a
shorter period ... heck I don't even think I would train juniors to help with
the prototype project unless I thought I had funding to complete it.

Now if a couple Univ CSc seniors in San Francisco or San Luis Obispo
or possibly Colorado wanted to help me on speculation we could
turn it into a product I might think about it. In fact doing an external
storage system from the VNODE/FSS interface was one idea on how to
turn this into a product without a systems vendors help.

Most ideas do not need an implementation to determine worth ... just
a simple pencil and paper model is generally enough as long as the
number of variables can be contained to two or three. See above.

I quoted SCO a very cheap bid to do the remaining research and a prototype
because I *really* wanted to do the project ... the london team thought they
could do something better (at surely a much higher cost). It's almost
impossible to outbid an internal development group on a fun project. Something
nobody wants to do ... piece a cake. Quality of ideas is never an issue ...
nor is experience or anything else that really matters - the internal team
almost always wins - even when you have it done already.

>No doubt some of the ideas will yield promising results.  But without some
>evidence, your grand re-invention seems littered with potholes.

Pretty bold statement to make without a better rebuttal to my ideas. Just what
are those potholes you see? spots on your glasses? :)

Try again?

>John F. Haugh II  [ NRA-ILA ] [ Kill Barney ] !'s: ...!cs.utexas.edu!rpp386!jfh

John Bass, Sr. Engineer, DMS Design                         idletime@netcom.com
UNIX Consultant                     Development, Porting, Performance by Design

------------------------------

Crossposted-To: comp.benchmarks,comp.sys.sun.admin,comp.security.unix
From: idletime@netcom.com (Totally Lost)
Subject: Re: Filesystem semantics protecting meta data ... and users data
Date: Sat, 18 Jun 1994 23:21:04 GMT

In article <1994Jun18.024610.27790@rpp386>, a confused
John F. Haugh II <jfh@rpp386.cactus.org> wrote:
>In article <DHOLLAND.94Jun14121255@husc7.harvard.edu> dholland@husc7.harvard.edu (David Holland) writes:
>>Haven't you been following the thread? If you don't flush those zeros
>>*to the disk* before using the block, a badly-timed system crash can
>>cause the UNZEROED blcosk to appear in the new file - containing who
>>knows what kind of private data.
>
>Correct, but the block number can't be in the inode of the new file
>without the buffered block being zeroed out.  If it isn't in the
>inode, the new file can't have access to the data. 

What happens in memory doesn't matter in this discussion, because the
whole point is what is the state of the disk at any point up to a crash.

With metadata sync writes the meta data always points to trash until the
async write data block is flushed. This is a long window.

>The only remaining
>discussion is whether or not the inode is flushed before the data
>block, and that is where the issue of "badly-timed" comes into play.

With any meta data sync write filesystem the window is open nearly the
entire time the file is being written - this is not a matter of badly-timed.

>The larger, more modern systems are much more likely to have this kind
>of problem (bigger buffer caches ...) than are the systems which the
>original poster blamed for introducing this problem.

Any system with over 5 buffers will exhibt the problem, that is any UNIX
system ever shipped. True it gets a lot worse over 5 buffers, but that
has been the case for nearly 20 years.

You clearly don't understand either UNIX filesystem internals or the
technical details behind this thread. Before your next post a short
study course in UNIX internals and filesystems and rereading this
thread highly is recomended.

>-- 
>John F. Haugh II  [ NRA-ILA ] [ Kill Barney ] !'s: ...!cs.utexas.edu!rpp386!jfh

------------------------------

From: dswartz@pugsley.osf.org (Dan Swartzendruber)
Crossposted-To: comp.benchmarks,comp.sys.sun.admin,comp.security.unix
Subject: Re: Filesystem semantics protecting meta data ... and users data
Date: 18 Jun 1994 23:35:44 GMT

In article <idletimeCrM850.Dqp@netcom.com> idletime@netcom.com (Totally Lost) writes:

[deleted]

If this is all so clear-cut and obvious, how about coming up
with an actual design, and putting it out for review?  If you
really know what you're talking about, it shouldn't be that
difficult, no?  And if you actually come up with an implementation
which is clearly superior FT-wise, and performs well, you could
end up saving all these poor users who are getting file corruption,
instead of keeping news spools filled.




-- 

#include <std_disclaimer.h>

Dan S.

------------------------------

From: adam@adam.yggdrasil.com (Adam J. Richter)
Crossposted-To: comp.os.linux.help,comp.os.linux.misc
Subject: Re: Yggdrasil CD-ROM question
Date: 18 Jun 1994 18:08:03 GMT

In article <2tv1tl$8k0@perot.mtsu.edu>,
Mr. Steve Wilkinson <wally@perot.mtsu.edu> wrote:
>I have an NEC3xi with a SoundBlaster 16 SCSI-II!  Yggdrasil CD-ROM does
>not mount the CD.  

        There is an experimental aha-152x driver in the Yggdrasil
Summer 1994 kernel, and in the standard Linux kernels driver for that
matter.  This driver can talk to the Adaptec chip on the Soundblaster
SCSI board.  We list the driver as "experimental" in our product
literature because it did not work on all of the CDROM drives on which
we tested it.

        In the original 1.1 kernel distribution, the aha152x driver
did not work the with NEC CDR-84dj drive, but we were able to make it
work with that drive by increasing some of the timeouts, so your
chances of making the system work are slightly better with our kernel
than with others.  Other drives, like the Texel dm3021 still fail.
You can get a chart of the SCSI testing that we did, from
ftp.yggdrasil.com:pub/support/scsi_tests.summer94.

        See page 58 of the Plug-and-Play Linux manual for instructions
on how  to turn on SCSI drivers for BIOS-less controllers (use the
aha152x driver for your Soundblaster SCSI).

-- 
Adam J. Richter                     -      --------------   "Free software for
adam@yggdrasil.com                    \  /                   the rest of us."
4880 Stevens Creek Blvd., Suite 205    || g g d r a s i l    408-261-6630
San Jose, CA 95129-1034                ||  Computing Inc.    fax 408-261-6631

------------------------------

From: steve1@morgan.ucs.mun.ca (Stephen Beaton-Snook)
Subject: 1.1.20 pdksh & csh problems
Date: 19 Jun 1994 01:35:46 GMT

Since Ive patched up to 1.1.20 Ive been haveing problems with pdksh and csh.
bash seems to work ok.

Command line history on the pdksh seems to have broken and when i type a
command as simple as 'man stty' or some such it gets sent to the background
and I cant forground it. I can run a program like 'dip -t' but when I try
to go to terminal mode it puts dip into the background and Im back to my
command line prompt. Anyone else experience this and is there a fix?

Steve


--

 Stephen Beaton-Snook                             Memorial University of NFLD
 steve1@morgan.ucs.mun.ca                         St John's, Newfoundland
                                                  Canada.


------------------------------

From: mimi@mimine.kaist.ac.kr (Seo Hongwon)
Subject: Re: 1.1.20 pdksh & csh problems
Date: 19 Jun 1994 03:40:07 GMT

Stephen Beaton-Snook (steve1@morgan.ucs.mun.ca) wrote:
: Since Ive patched up to 1.1.20 Ive been haveing problems with pdksh and csh.
: bash seems to work ok.

: Command line history on the pdksh seems to have broken and when i type a
: command as simple as 'man stty' or some such it gets sent to the background
: and I cant forground it. I can run a program like 'dip -t' but when I try
: to go to terminal mode it puts dip into the background and Im back to my
: command line prompt. Anyone else experience this and is there a fix?

me too. how can I fix it?
plz mail to mimi@baram.kaist.ac.kr

: Steve


: --

:  Stephen Beaton-Snook                             Memorial University of NFLD
:  steve1@morgan.ucs.mun.ca                         St John's, Newfoundland
:                                                 Canada.


--
        ______________                                   ______________
       / __ \_________)           Seo Hongwon           (_________/ __ \
      /   | |___)               mimi@kaist.ac.kr              (___| |   \
     /   (\_|__)      Applied Fluid Mechanics Lab., KAIST      (__|_/)   \
    /   (_____)                 +82-42-869-5014                 (_____)   \

------------------------------


** FOR YOUR REFERENCE **

The service address, to which questions about the list itself and requests
to be added to or deleted from it should be directed, is:

    Internet: Linux-Development-Request@NEWS-DIGESTS.MIT.EDU

You can send mail to the entire list (and comp.os.linux.development) via:

    Internet: Linux-Development@NEWS-DIGESTS.MIT.EDU

Linux may be obtained via one of these FTP sites:
    nic.funet.fi				pub/OS/Linux
    tsx-11.mit.edu				pub/linux
    sunsite.unc.edu				pub/Linux

End of Linux-Development Digest
******************************
