From: believer@telepath.com Subject: IP: Y2K Horror Stories a Foretaste of Breakdowns to Come Date: Wed, 23 Sep 1998 23:08:03 -0500 To: ignition-point@majordomo.pobox.com Source: http://www.garynorth.com/y2k/detail_.cfm/2665 Category: Noncompliant_Chips Date: 1998-09-23 11:10:48 Subject: Horror Stories That Are a Foretaste of Breakdowns to Come Link: http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000AUk Comment: This document speaks for itself. Mutiply this through every institution that uses embedded systems. There is a lesson here. Any time anyone tells you, "This will work. No problem," ask him to demonstrate how it will work with no problem. Companies are just like governments. They stonewall. They pretend. They lie. Mainly, when caught, they lie. "I did not have noncompliance with that product!" This was especially interesting: "Nearly a month after I sent email to two different year 2000 addresses at the company, I received a phone call from the company. The caller described herself as a liaison with technical support and admitted to having no technical knowledge of the instrument. She said that their legal department would only allow verbal communication in this matter. She was prohibited from replying by email or written communication. "She said the embedded systems 'could be a problem' if new firmware wasn't installed. When questioned, she admitted that new firmware was not available and she did not know when it would be." Substitute the term "Western civilization" for "new firmware." * * * * * * * * * * * I am an analytical chemist specializing in inductively coupled plasma atomic emission spectroscopy (ICP-AES). ICP-AES is a method used to determine trace amounts of metals in solutions in the part per million and part per billion range. I am responsible for two spectrometers each of which cost approximately $130,000. Both instruments contain embedded microprocessor systems that are in turn controlled by external computers. The first uses a PDP-11/53 for the external computer. It is running the instrument control software under RT-11 and TSX-Plus. No hardware real time clock exists in the system. This version of RT-11 only accepts 1979 to 1999 as valid years for the system clock. There is no 2000 or 00. If left powered up, the system clock rolls over from 99 to 100. The file system is not designed to handle dates outside of the 79 to 99 range. A compliant version of RT-11 has just become available, but the manufacturer of the instrument says this won't help with problems in the instrument control software. We have been told that the software will "crash and burn" when hit with year 2000. The instrument was manufactured in 1988 and the software is no longer supported by the company. In fact, the company no longer employs anyone who worked on the Fortran source. The company's solution is to replace the PDP-11 with a PC and purchase a $6000 software package that runs under Windows. If we did this, we would no longer be able to use the quality control and data transfer programs we have written for the current system. We would also have several weeks of down time while entering our current analytical methods into the new system. My experience with a beta version of the PC software leads me to believe productivity would suffer in the end. The second instrument was controlled with a DEC 466 (486-66) running instrument control software under SCO-UNIX System V/386 Release 3.2. This system became very confused after the real time clock (RTC) was set to 12/31/99. Allowing the clock to roll over with the power off gave a RTC date of 1900. On boot up, Unix said the system year was 2000. However, the instrument software reported the current date as 01/01/10 and the RTC was reset to 1970. With the RTC set to 2000, the instrument software still reported the date as 01/01/10, but now the RTC was reset to 2070. On the next boot up, Unix would report the system date as 1970. At any time, typing in 000101 in the yymmdd field on boot up resulted in the RTC being reset to 1970. SCO has a patch for the operating system, but again this would not solve problems in the instrument control software. In this case, we have replaced the 486 with a Y2k compliant Pentium 300 and installed instrument control software that runs under Windows 95. The manufacturer no longer supports the software that ran under Unix. The instrument is two years old. Inside the instrument are two embedded 68000 microprocessor systems with real time clocks. Access to the clocks requires a program normally supplied only to service technicians. Execution of that program requires a password. The real time clocks can run without external power for up to ten years on internal lithium batteries. One function of the clocks is to insure the proper start-up sequence is followed after shutdowns of different lengths (hot or cold start). Extensive damage could be caused by an error in the control of heating, cooling, or gas flow systems. The clocks are also used in the monitoring of safety interlocks and sensors. Both clocks rollover from 1999 to 1900 and can not be set to 2000. The clocks also count off the days of the week. I am reluctant to extensively test the system myself because of the possibility of damage. When I first talked to technical support about my concerns, the technician pulled out a copy of the year 2000 compliance certificate for the new software and read it over. He then stated: "This certification only covers the software - not the *instrument*." This instrument and software is being sold as the company's year 2000 solution to replace almost all earlier systems. My next contact was with a different technician who was in the lab to repair an electronic problem in the instrument. He told me: "Don't worry, we use four digit dates." I asked him to demonstrate setting the clocks ahead. He ran the program on the main computer and it allowed him to type the year 2000 on the entry screen. He then executed the command to send the date to the embedded system. There were no error messages. He gave me a look that implied "I told you so". I then asked him to read back the current time from the clock he just set. He was visibly startled when he saw 1900 appear on the screen. He said he would have one of their experts call me. A few days later I heard from the "expert" who tried to reassure me with phrases like "they're just timers" and "the instrument doesn't _need_ to know what day it is". However, since he didn't seem to have a mastery of what the clocks actually do, I wasn't reassured. For example: I can put the instrument into standby (sleep mode) over the weekend with instructions to be ready to operate at 8:00 AM Monday. This will conserve gases (nitrogen and argon) and reduce power needs. 73 minutes before the assigned ready time, the embedded systems will query the main computer for start-up instructions. If the main system is running, it takes control of the sequence. If the main system is off, the embedded systems will look for it for 10 minutes and then independently begin the start-up sequence in the instrument EPROM. The "expert" didn't seem to have even this depth of knowledge of the system. Nearly a month after I sent email to two different year 2000 addresses at the company, I received a phone call from the company. The caller described herself as a liaison with technical support and admitted to having no technical knowledge of the instrument. She said that their legal department would only allow verbal communication in this matter. She was prohibited from replying by email or written communication. She said the embedded systems "could be a problem" if new firmware wasn't installed. When questioned, she admitted that new firmware was not available and she did not know when it would be. She also said that the clocks were used for the "wake up" procedure and not used for anything else. Since she had no knowledge of the instrument, there was no point in asking about system error (SYSERROR) message 75 described in the appendix of the manual: 75 Real-Time Clock. Call Service. The real-time clock, used to monitor the status of interlocks (and sensors), is not functioning. This will not shut down the system; however, a XXXXXXXX service engineer should be contacted. "This will not shut down the system" means I can start up a sustained 10,000 degree plasma in a confined space with uncertainty as to whether the interlocks and sensors will detect a malfunction. It's my belief that in thinking about embedded systems, this company has been in "sleep mode". I may have been the first to sound a "wake up" alarm about this instrument. Now they have to muster the people and resources to re-write the programs stored in the EPROM's, produce new chips, and install them in hundreds of instruments around the world. I should also mention that this is only one instrument in a large product line that has other Y2k problems. Will they get them all done? On time? I don't think they know. Link: http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000AUk ---------------------- NOTE: In accordance with Title 17 U.S.C. section 107, this material is distributed without profit or payment to those who have expressed a prior interest in receiving this information for non-profit research and educational purposes only. For more information go to: http://www.law.cornell.edu/uscode/17/107.shtml ----------------------- ********************************************** To subscribe or unsubscribe, email: majordomo@majordomo.pobox.com with the message: (un)subscribe ignition-point email@address ********************************************** www.telepath.com/believer **********************************************