IP: Y2K Horror Stories a Foretaste of Breakdowns to Come

[Vladimir Z. Nuri]

From: believer at telepath.com
Subject: IP: Y2K Horror Stories a Foretaste of Breakdowns to Come
Date: Wed, 23 Sep 1998 23:08:03 -0500
To: ignition-point at majordomo.pobox.com

Source:  http://www.garynorth.com/y2k/detail_.cfm/2665

Category: Noncompliant_Chips
Date: 1998-09-23 11:10:48
Subject: Horror Stories That Are a Foretaste of Breakdowns to Come
Link:  http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000AUk

Comment: 

This document speaks for itself. Mutiply this through every institution
that uses embedded systems. 

There is a lesson here. Any time anyone tells you, "This will work. No
problem," ask him to demonstrate how it will work with no problem. 

Companies are just like governments. They stonewall. They pretend. They
lie. Mainly, when caught, they lie. "I did not have noncompliance with that
product!" 

This was especially interesting: 

"Nearly a month after I sent email to two different year 2000 addresses at
the company, I received a phone call from the company. The caller described
herself as a liaison with technical support and admitted to having no
technical knowledge of the instrument. She said that their legal department
would only allow verbal communication in this matter. She was prohibited
from replying by email or written communication. 

"She said the embedded systems 'could be a problem' if new firmware wasn't
installed. When questioned, she admitted that new firmware was not
available and she did not know when it would be." 

Substitute the term "Western civilization" for "new firmware." 

* * * * * * * * * * * 

I am an analytical chemist specializing in inductively coupled plasma
atomic emission spectroscopy (ICP-AES). ICP-AES is a method used to
determine trace amounts of metals in solutions in the part per million and
part per billion range. 

I am responsible for two spectrometers each of which cost approximately
$130,000. Both instruments contain embedded microprocessor systems that are
in turn controlled by external computers. 

The first uses a PDP-11/53 for the external computer. It is running the
instrument control software under RT-11 and TSX-Plus. No hardware real time
clock exists in the system. This version of RT-11 only accepts 1979 to
1999 as valid years for the system clock. There is no 2000 or 00. If left
powered up, the system clock rolls over from 99 to 100. The file system is
not designed to handle dates outside of the 79 to 99 range. A compliant
version of RT-11 has just become available, but the manufacturer of the
instrument says this won't help with problems in the instrument control
software. We have been told that the software will "crash and burn" when hit
with year 2000.  

The instrument was manufactured in 1988 and the software is no longer
supported by the company. In fact, the company no longer employs anyone who
worked on the Fortran source. The company's solution is to replace the
PDP-11 with a PC and purchase a $6000 software package that runs under
Windows. If we did this, we would no longer be able to use the quality
control and data transfer programs we have written for the current system.
We would also have several weeks of down time while entering our current
analytical methods into the new system. My experience with a beta version
of the PC software leads me to believe productivity would suffer in the end.  

The second instrument was controlled with a DEC 466 (486-66) running
instrument control software under SCO-UNIX System V/386 Release 3.2. This
system became very confused after the real time clock (RTC) was set to
12/31/99. 

Allowing the clock to roll over with the power off gave a RTC date of 1900.
On boot up, Unix said the system year was 2000. However, the instrument
software reported the current date as 01/01/10 and the RTC was reset to 1970. 

With the RTC set to 2000, the instrument software still reported the date
as 01/01/10, but now the RTC was reset to 2070. On the next boot up, Unix
would report the system date as 1970. At any time, typing in 000101 in the
yymmdd field on boot up resulted in the RTC being reset to 1970. SCO has a
patch for the operating system, but again this would not solve problems in
the instrument control software. 

In this case, we have replaced the 486 with a Y2k compliant Pentium 300 and
installed instrument control software that runs under Windows 95. The
manufacturer no longer supports the software that ran under Unix. The
instrument is two years old. 

Inside the instrument are two embedded 68000 microprocessor systems with
real time clocks. Access to the clocks requires a program normally supplied
only to service technicians. Execution of that program requires a
password. The real time clocks can run without external power for up to ten
years on internal lithium batteries. 

One function of the clocks is to insure the proper start-up sequence is
followed after shutdowns of different lengths (hot or cold start).
Extensive damage could be caused by an error in the control of heating,
cooling, or
gas flow systems. The clocks are also used in the monitoring of safety
interlocks and sensors. Both clocks rollover from 1999 to 1900 and can not
be set to 2000. The clocks also count off the days of the week. I am
reluctant to extensively test the system myself because of the possibility
of damage. 

When I first talked to technical support about my concerns, the technician
pulled out a copy of the year 2000 compliance certificate for the new
software and read it over. He then stated: "This certification only covers the
software - not the *instrument*." This instrument and software is being
sold as the company's year 2000 solution to replace almost all earlier
systems.  

My next contact was with a different technician who was in the lab to
repair an electronic problem in the instrument. He told me: "Don't worry,
we use four digit dates." I asked him to demonstrate setting the clocks
ahead. He ran the program on the main computer and it allowed him to type
the year 2000 on the entry screen. He then executed the command to send the
date to the embedded system. There were no error messages. He gave me a
look that implied "I told you so". I then asked him to read back the
current time from the clock he just set. He was visibly startled when he
saw 1900 appear on the screen. He said he would have one of their experts
call me.  

A few days later I heard from the "expert" who tried to reassure me with
phrases like "they're just timers" and "the instrument doesn't _need_ to
know what day it is". However, since he didn't seem to have a mastery of
what the clocks actually do, I wasn't reassured. For example: I can put the
instrument into standby (sleep mode) over the weekend with instructions to
be ready to operate at 8:00 AM Monday. This will conserve gases (nitrogen
and argon) and reduce power needs. 73 minutes before the assigned ready
time, the embedded systems will query the main computer for start-up
instructions. If the main system is running, it takes control of the
sequence. If the main system is off, the embedded systems will look for it
for 10 minutes and then independently begin the start-up sequence in the
instrument EPROM. The "expert" didn't seem to have even this depth of
knowledge of the system.  

Nearly a month after I sent email to two different year 2000 addresses at
the company, I received a phone call from the company. The caller described
herself as a liaison with technical support and admitted to having no
technical knowledge of the instrument. She said that their legal department
would only allow verbal communication in this matter. She was prohibited
from replying by email or written communication. 

She said the embedded systems "could be a problem" if new firmware wasn't
installed. When questioned, she admitted that new firmware was not
available and she did not know when it would be. 

She also said that the clocks were used for the "wake up" procedure and not
used for anything else. Since she had no knowledge of the instrument, there
was no point in asking about system error (SYSERROR) message 75 described
in the appendix of the manual: 

75 Real-Time Clock. Call Service. 

The real-time clock, used to monitor the status of interlocks (and
sensors), is not functioning. This will not shut down the system; however,
a XXXXXXXX service engineer should be contacted. 

"This will not shut down the system" means I can start up a sustained
10,000 degree plasma in a confined space with uncertainty as to whether the
interlocks and sensors will detect a malfunction. 

It's my belief that in thinking about embedded systems, this company has
been in "sleep mode". I may have been the first to sound a "wake up" alarm
about this instrument. Now they have to muster the people and resources to
re-write the programs stored in the EPROM's, produce new chips, and install
them in hundreds of instruments around the world. I should also mention
that this is only one instrument in a large product line that has other Y2k
problems. Will they get them all done? On time? I don't think they know. 

 Link: http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000AUk 
----------------------
NOTE: In accordance with Title 17 U.S.C. section 107, this material is
distributed without profit or payment to those who have expressed a prior
interest in receiving this information for non-profit research and
educational purposes only. For more information go to:
http://www.law.cornell.edu/uscode/17/107.shtml
-----------------------






**********************************************
To subscribe or unsubscribe, email:
     majordomo at majordomo.pobox.com
with the message:
     (un)subscribe ignition-point email at address
**********************************************
www.telepath.com/believer
**********************************************