*BSD News Article 18679


Return to BSD News archive

Path: sserve!newshost.anu.edu.au!munnari.oz.au!news.Hawaii.Edu!ames!agate!howland.reston.ans.net!xlink.net!math.fu-berlin.de!fub!unlisys!max.IN-Berlin.DE!not-for-mail
From: berry@moritz.IN-Berlin.DE (Stefan Behrens)
Newsgroups: comp.os.386bsd.questions
Subject: Re: drive light on and locked -- w/ patch
Date: 21 Jul 1993 00:19:45 +0200
Organization: Private Site in Berlin, Germany
Lines: 309
Message-ID: <22hr2f$hm@moritz.in-berlin.de>
References: <22gd69$6j3@hrd769.brooks.af.mil>
NNTP-Posting-Host: moritz.in-berlin.de
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Summary: patch for 386bsd0.1 pk0.2.4 that detects and cleares wd-ctlr lockups

In article <22gd69$6j3@hrd769.brooks.af.mil> burgess@hrd769.brooks.af.mil (Dave Burgess) writes:
>There was some discussion about three weeks ago about the system hang
>where the hard drive locks up with the the drive light lit.  Someone,
>whose name I have regretfully forgotten, posted either a description of
>a change that (s)he had made to the system that reset the drive.

I cannot remember seeing a patch for it myself, nor can egrep find one.
But I also had the problem with the locked IDE controller since I
installed 386bsd one year ago.
It's easy to recognize the situation of the locked driver. It's easy
to fix the situation. But I still don't know why this happens, why
the driver looses the interrupt from the controller, or maybe why
the controller doesn't generate the expected interrupt.
I wrote some code to catch the problem. For every block-read or write
request it will be checked whether the controller answers in time.
This is easy to do for IDE controllers and doesn't mean much overhead.


>In case I dreamed it (which has happened more than once), I propose a
>simple solution (sweets from the sweet :-).
>
>On a disk {read,select} start up an alarm that times out in three
>seconds.  On a successful operation from the disk, clear the alarm.
>When the alarm expires, that means that no disk activity has succeeded
>in the last three seconds, which would seem to me to be a good indicator
>that the drive/controller has siezed up again.

This really is what my code does. But it isn't necessary to start an
timeout for every request since IDE controller are so stupid, they
can only handle one request at once. So my code does the following:

o in general -- for every attached drive every two seconds a function
  will be called which checks for an in-work request, for which the
  controller doesn't answer
o in detail:
  - on a per drive basis an function will be called periodically and
    a timeout counter will be managed
  - when a read/write request is started, the counter is set to two
  - when the expected interrup comes in it will be cleared with zero
  - in the timeout function, when the counter is > 0, it's decremented
    o if it's decremented the first time (so it's one then) nothing is
      done
    o if it's decremented the second time (and zero then) the controller
      is timed out...
    o ...when it's timed out:
      - status/debug info will be printed
      - the failure will be logged
      - the controller will be reset in order to put it in a known state
      - the request will be restarted in a `sector by sector' way, this
        means multi-sector transfers are split up


I used code similar to this in a 386bsd0.1 pk0.2.2 environment with the
Barsoom wd-driver (used in NetBSD too) and Bruce's intr/npx/com stuff
for month.
This new patch (and first public posting) is against 386bsd0.1 pk0.2.4.
It should be a two line change to use it for NetBSD but I didn't try it.
The code for the detection of the problem is very well tested and very
old. It won't hurt systems that don't show the problems. The code for
solving the problem and for restarting the request is newer.
I have two machines up which run that code. One with only one IDE drive,
and one with two IDE drives and some SCSI devices. The first and newer
computer never had problems with locked wd controllers. But the code
doesn't hurt either. The second machine which is a server used to fail
very often because of this. Now it detects the lock, resets the drive
and restarts the request. The problem is finally solved for me.


>Note:  I am using the current-sources from sun-lamp using sup.

NetBSD uses the Barsoom wd-driver. It should be easy to change the
following patch for this driver.


>The drive does not fail like this with the 0.8
>released kernel, or from the sources that came out with the original 0.8
>release.

It happens in ``cooperation'' with some other drivers, e.g. with the
com driver or with the we-ethernet driver (for me).


Maybe someone who knows more about IDE controllers can comment on this:
in the situation where the controller gets locked, the status is
- inb(wdc+wd_status) --> WDCS_READY|WDCS_SEEKCMPLT|WDCS_DRQ
  which means the drive is ready, seek completed and the data request
  bit is set
- inb(wdc+wd_error) --> is 0
- the request is a multi sector block read request
- the action that helps for me is to restart the request with
  `du->du->dk_skip = 0;' and `du->dk_flags |= DKFL_SINGLE;'
- without enforcing the redo in single sector steps the restart didn't
  succeed


Ok, after that boring talking here's the patch against 386bsd0.1 pk0.2.4.
Try it, it won't hurt but it will save many system lockups!


*** /tmp/,RCSt1000483	Tue Jul 20 23:45:22 1993
--- wd.c	Tue Jul 20 23:33:01 1993
***************
*** 56,61 ****
--- 56,63 ----
   * 17 May 93	Rodney W. Grimes	Fixed all 1000000 to use WDCTIMEOUT,
   *					and increased to 1000000*10 for new
   *					intr-0.1 code.
+  * 15 Jul 93	Stefan Behrens		Added real timeout code to catch
+  * 					hanging controller
   */
  
  /* TODO:peel out buffer at low ipl, speed improvement */
***************
*** 150,155 ****
--- 152,174 ----
  int	wddebug;
  #endif
  
+ /*
+  * counter for lost int detection.
+  * Three values are used:
+  * 2 -- this is the initial value when the timeout is armed
+  * 1 -- timed out once, give it one more chance
+  * 0 -- timeout not armed, idle
+  * The per-drive-counter is an overkill here, for wd-controller a
+  * per-controller-counter would be enough. But this way it doesn't
+  * add new restrictions to the driver, and it's simple.
+  */
+ int	wdtimeout_counter[_NWD];
+ 
+ /*
+  * during recovery of timed out requests this counter is used
+  */
+ int	wdtimeout_retry[_NWD];
+ 
  struct	isa_driver wddriver = {
  	wdprobe, wdattach, "wd",
  };
***************
*** 160,165 ****
--- 179,185 ----
  int wdcontrol(struct buf *);
  int wdsetctlr(dev_t, struct disk *);
  int wdgetctlr(int, struct disk *);
+ int wdtimeout(caddr_t);
  
  /*
   * Probe for controller.
***************
*** 227,232 ****
--- 247,257 ----
  			du->dk_port = dvp->id_iobase;
  		}
  
+ 		wdtimeout_retry[unit] =
+ 		wdtimeout_counter[unit] = 0;	/* not armed yet */
+ 		wdtimeout((caddr_t) unit);		/* initially set timeout */
+ 
+ 
  		/* print out description of drive, suppressing multiple blanks*/
  		if(wdgetctlr(unit, du) == 0)  {
  			int i, blank;
***************
*** 402,408 ****
  			(bp->b_flags & B_READ) ? "read" : "write",
  			bp->b_bcount, blknum);
  	else
! 		printf(" %d)%x", du->dk_skip, inb(wdc+wd_altsts));
  #endif
  	addr = (int) bp->b_un.b_addr;
  	if (du->dk_skip == 0)
--- 427,433 ----
  			(bp->b_flags & B_READ) ? "read" : "write",
  			bp->b_bcount, blknum);
  	else
! 		printf(" %d)%x", du->dk_skip, inb(du->dk_port+wd_altsts));
  #endif
  	addr = (int) bp->b_un.b_addr;
  	if (du->dk_skip == 0)
***************
*** 537,543 ****
  	}
  
  	/* if this is a read operation, just go away until it's done.	*/
! 	if (bp->b_flags & B_READ) return;
  
  	/* ready to send data?	*/
  	timeout = 0;
--- 562,573 ----
  	}
  
  	/* if this is a read operation, just go away until it's done.	*/
! 	if (bp->b_flags & B_READ) {
! 		wdtimeout_counter[unit] = 2;	/* arm timeout counter */
! 		if (wdtimeout_retry[unit])
! 			printf("wd.c: retry block read\n");
! 		return;
! 	}
  
  	/* ready to send data?	*/
  	timeout = 0;
***************
*** 561,566 ****
--- 591,599 ----
  	outsw (wdc+wd_data, addr+du->dk_skip * DEV_BSIZE,
  		DEV_BSIZE/sizeof(short));
  	du->dk_bc -= DEV_BSIZE;
+ 	wdtimeout_counter[unit] = 2;	/* arm timeout counter */
+ 	if (wdtimeout_retry[unit])
+ 		printf("wd.c: retry block write\n");	/* never seen this */
  }
  
  /* Interrupt routine for the controller.  Acknowledge the interrupt, check for
***************
*** 588,593 ****
--- 621,630 ----
  	du = wddrives[wdunit(bp->b_dev)];
  	wdc = du->dk_port;
  
+ 	wdtimeout_counter[wdunit(bp->b_dev)] = 0;	/* unarm counter */
+ 	wdtimeout_retry[wdunit(bp->b_dev)] = 0; /* start from zero */
+ 
+ 
  #ifdef	WDDEBUG
  	printf("I ");
  #endif
***************
*** 1349,1352 ****
--- 1386,1463 ----
  	}
  	return(0);
  }
+ 
+ /*
+  * called periodically every two seconds for each attached drive.
+  * check if the drive didn't answer in time.
+  */
+ int
+ wdtimeout(caddr_t arg)
+ /* arg is # of unit */
+ {
+ 	int x = splbio();	/* XXX kept all the time */
+ 	register int unit = (int) arg;
+ 
+ 	if (wdtimeout_counter[unit]) { /* armed? */
+ 		if (--wdtimeout_counter[unit] == 0) { /* timed out? */
+ 			struct disk *du = wddrives[unit];
+ 			int wdc = du->dk_port;
+ 
+ 			wdtimeout_retry[unit]++;
+ 
+ 			/* log failure */
+ 			printf("wd.c: wd%d timed out, retry #%d\n",
+ 			       unit, wdtimeout_retry[unit]);
+ 
+ 			/* reset ctlr and redo request */
+ 			switch (wdtimeout_retry[unit]) {
+ 			case 1: /* for me one retry is enough :-) */
+ 			case 2:
+ 			case 3:
+ 				/*
+ 				 * print some status info
+ 				 *
+ 				 * The values I see for my system are:
+ 				 * status=58, error=0
+ 				 * That means status is:
+ 				 * WDCS_READY		Selected drive is ready
+ 				 * WDCS_SEEKCMPLT	Seek complete 
+ 				 * WDCS_DRQ		Data request bit.
+ 				 * Does anyone know a reason for the timeout?
+ 				 */
+ 				printf("wd.c: wd%d status %x, error %x\n",
+ 				       unit, inb(wdc+wd_status),
+ 				       inb(wdc+wd_error));
+ 				/*
+ 				 * reset the device, give it a known state
+ 				 */
+ 				outb(wdc+wd_ctlr, (WDCTL_RST|WDCTL_IDS));
+ 				DELAY(1000);
+ 				outb(wdc+wd_ctlr, WDCTL_IDS);
+ 				DELAY(1000);
+ 				(void) inb(wdc+wd_error);	/* XXX! */
+ 				outb(wdc+wd_ctlr, WDCTL_4BIT);
+ 				/*
+ 				 * we'll redo the xfer sector by sector.
+ 				 * This is the trick that helps here!
+ 				 */
+ 				du->dk_skip = 0;	/* start at #0 again */
+ 				du->dk_flags |= DKFL_SINGLE;	/* slow down */
+ 				break;
+ 			default: /* give up -- never happened for me */
+ 				panic("cannot solve problem with hanging wd ctrl");
+ 				break;
+ 			}
+ 
+ 			/* restart request */
+ 			wdstart();
+ 		}
+ 	}
+ 
+ 	/* plan next timeout */
+ 	timeout(wdtimeout, unit, 200);
+ 	splx(x);
+ 	return (0);
+ }
+ 
  #endif
-- 
Stefan (berry@max.IN-Berlin.DE)