IP: Looming Y2k Crisis

From: "Douglas E. Renz" <dougr@zlink.net> Subject: IP: Looming Y2k Crisis Date: Thu, 10 Sep 1998 15:53:12 -0400 To: "'Chris Ball'" <1stlight@mail.tds.net>, "'En_Agape'" <en_agape@prodigy.net>, "'Kim/Jr'" <jrandkim@juno.com>, "'Kristen'" <kristen_renz@juno.com>, "'Steve Cortese'" <Sccortese@juno.com>, "'Tom Young'" <tayoung@cottagesoft.com> 36 Seemingly Innocuous Questions That Pry the Truth Out of Managers and Programmers Regarding the Year 2000 (Y2K) Problem and Their Organization's Looming Crisis (c) Gary North, 1997 http://www.garynorth.com The author hereby gives permission to any reporter, columnist, or journalist to e-mail a copy of this report to any colleague in the profession. INTRODUCTION The Year 2000 (Y2K) Problem is real. If it weren't, Chase Manhattan Bank would not have budgeted between $200 million and $250 million to solve it. Nobody knows what the fallout will be. My Web site offers summaries and links to documents posted all over the Web. But there are so many potential falling dominoes that nobody can predict most of what's going to happen. We will be hit from all sides without warning. Things that nobody is thinking about today will be disrupted. For example, the water-control locks on the Panama Canal are controlled by a mainframe computer system. Jimmy Carter and Congress signed away the Canal back in 1977. It goes to Panama on January 1, 2000. Will the U.S. government fund this Y2K repair? (Ha!) If not, what happens to the canal in 2000? What happens to shipping? The government is not discussing this. Dominoes. Nothing but dominoes. Maybe you can find some interesting ones locally. I have set up a closed discussion forum on my Web site for reporters who are working on Y2K assignments. You can post messages to each other. Maybe it will help. If you find an interviewing strategy that works better than what I suggest here, let me know. Meanwhile. . . . I. EXPECT TO BE STONEWALLED OR MISLED This reporting assignment is easier for women. The company's PR department will assume that a woman doesn't know anything technical. The manager or PR flak may be less guarded. If you're a woman who majored in English lit, let the public relations people know this early. For general background, read the cover story of NEWSWEEK (June 2). This is an accurate account. If anything, it is overly subdued. You need to know what is at stake internationally before you begin research on how the Millennium Bug will affect your community. My report is no cure-all. A well-informed manager is going to be paranoid about a story on Y2K and his organization. You have to hope you wind up interviewing a manager or spokesman who really doesn't understand the problem. The closer you get to the truth, the more that a careful manager will raise his "deflector shields." "We cannot divulge any proprietary information." "Our contractor is in charge of this project." "No, he cannot speak with you about this." "Our vendors are taking care of this." "No, we cannot supply their names." "We are on schedule: late 1998." When you hear any of these, be assured: you're onto something important. To get the full story, you must now earn your paycheck. You can invoke the time-honored, "We want you to have an opportunity to tell your side of the story." You may choose to hand him a photocopy of the NEWSWEEK article. "This paints a pretty glum picture. I had hoped to find out if things really are as bad as this report indicates. If your organization is on top of this problem, I would like to be able to report this to our readers." You can gently make it clear that a refusal to answer questions will make the company's situation look bad -- maybe worse than it really is. If you can't get access to a programmer, keep asking the spokesman questions, even if you get "no comment" to all of them. Write each refusal down in your notebook. This will make him nervous. The most important Y2K-threatened organizations in your community are the public utilities: water/sewers, natural gas, and above all, electricity (the national power grid is at risk). You could do a powerful series of articles on just these. They can't stonewall you as easily as a private firm can. They are politically controlled. You can go to members of the public utilities commission, tell them they you're being stonewalled, and ask for help. Ultimately, public utilities commission members will become personally liable if the system goes down and they knew about it in advance. You can even print out two or three of the documents I've posted on my Web site under "Litigation." Hand these print-outs to one or more of the PUC's members. My site's address is: http://www.garynorth.com A member of a public utility commission is at great personal risk if he lies or deliberately misleads the public, and then a disaster takes place. Call it "Chernobyl syndrome." Ask one or more of them to run interference for you. You want the interview. If you absolutely can't get access to a programmer at a public utility, print out a copy of this report and hand it to every member of the public utilities commission. Tell them they had better get these questions answered for themselves, and soon. I know of only one detailed local Y2K study that is easily available: the report on North Platte, Nebraska, written by three Creighton University economics professors. I have posted it on my site under "Government." A front-page story on Dick Brich, the programmer in charge of the county's computer system, appeared in the May 1, 1997 issue of USA TODAY. To understand the programming problem, you need a standard for comparison. Here is a good one. Social Security has had 400 programmers working since 1991 to fix its system, which has 30 million lines of code. The SSA discovered the problem in 1989. As of 1997, the system is still noncompliant. (See my Web site's category, "Government.") Use Social Security your benchmark. II. WHAT IS THE THREAT? You should assume that the spokesman or manager who has consented to be interviewed has no idea of what is involved technically in a Y2K repair. He does not know how long it will take. Neither does anyone else. Nothing on this vast scale has ever been attempted in the Information Technology world. The company probably has an official deadline: December 31, 1998. This has been forced on management by the threat of post-1999 litigation: a due diligence date, just in case the repair fails. At least a year of testing is necessary to verify that the repair is bug-free or at least manageable. The question is: Manageable by how many people? Kathy Adams of Social Security says that a 1% error rate is too high. With either 43 million or 50 million monthly checks -- she invoked both figures on separate occasions -- either 430,000 or 500,000 phone calls would hit the branch offices within days. It would overload the SSA's system, she says. (Her estimate is just for month one. What if these complaints all not taken care of, and another 1% error rate happens again next month? SSA will get a cascading effect -- noise, confusion, and telephone busy signals. Shutdown!) Medicare expects to pay a billion claims in 2000. What percentage of errors would overwhelm the system, assuming the system doesn't shut down on Jan. 1, 2000? But the manager who is speaking with you -- think of him as Dilbert's boss's boss -- has not thought about any of this. So, he has no idea what constitutes a failed repair, short of an absolute shutdown. His job is to paint a happy-face picture for you and all other inquirers. If the company is facing a disaster, he doesn't want you to find out, assuming that anyone in the company has warned him, which is doubtful. Everyone in the company has an incentive to lie. Dilbert is paid to write code. He does not make waves. Meanwhile, his boss is threatened with getting fired if he tells senior management, "I can't get this project finished on schedule. The system will crash." The boss says his team in on schedule. Who can check up on him to find out if he's behind schedule? Nobody in management. He collects his pay until December 31, 1999, at which time he turns into Maxwell Smart: "Sorry about that." Senior managers desperately want to believe the IT department, so they don't ask further questions. They hope that all their competitors are in the same condition, which is surely the case. This is why bond-rating services have refused to downgrade any organization's bonds because of Y2K risks. They're all equally at risk. III. ASSESSING THE ORGANIZATION'S VULNERABILITY Here are the Six Fundamental Questions: 1. "In what ways is your system is dependent on dates?" (It's is a soft-core version of this: "If you fail to fix this, what happens?") 2. "When did the company find out about Y2K?" 3. "At what stage is the repair: inventory, assessment, code-fixing, testing [ha!]?" 4. "How many lines of code are in the system?" 5. "How many programmers are working on it?" 6. "What is your deadline to begin testing?" It is a standard estimate that a skilled programmer with a date-search software tool ("silver bullet") can correct about 100,000 lines of code a year. This number is crucial to your final story. If you can find out how many lines of code the organization must correct and the number of programmers working full-time on the repair, you can estimate whether there is any chance they will get the problem fixed. By this I mean the local computer. This does not solve the ultimate problem: integrating a repaired computer with other computers, which, if not repaired, or repaired with a different approach, will send bad data into the first computer, ruining the repair. (There is no agreed-upon standard for Y2K repairs; every programming team is on its own, all over the world.) Be generous. Let's say that the programmers are all hot-shots. Let's say they can fix on average 10,000 lines a month. Multiply this by the number of months left until the universally agreed-upon date for the beginning of testing: December 31, 1998. Then multiply this number by the number of programmers. This will tell you if the outfit has a chance of meeting the deadline for testing. If the programmers miss the deadline -- and in 85% of all large-scale code revision projects, they do -- then this outfit the equivalent of the Titanic. Do you think that a program that's at least 500 times more complex than your word processor can be re-coded by a committee and not suffer a crash or a major glitch the first time it's run? They had better test it. What happens if they don't have time? Second, what happens if the test crashes the new system? The Jan. 1, 2000 deadline is a brick wall. Normally, 40% of a Y2K repair budget must be allocated for testing. This means 40% of the time budget unless proven otherwise. It may be higher in complex systems -- above 100 million lines of code. The spokesman may tell you the truth about the Y2K discovery date if he's proud of it: 1995 or early 1996. Keep thinking "Social Security." They found out in 1989, got started on the repair in 1991, and it's still not compliant. "Which stage?" He may not know at which stage their repair team is. If he does know, he may not want to tell you. Maybe the programmer in charge will, if he doesn't think it's privileged information. So, your goal is to get the spokesman to turn you over to the senior programmer -- or, better yet for information purposes, a subordinate programmer, who probably has no idea of the sensitivity of the information he possesses. IV. DETOUR: "PACKAGED SOFTWARE SOLUTION" I don't count these among the 36 questions. The spokesman may tell you that they will soon buy a packaged system. If you were conducting this interview in 1992, this might make sense. Today, it's too late. Packaged systems for highly complex, highly customized software are very expensive to buy and very time-consuming to implement. A packaged system is no panacea today. The problem is this: How can they port their old data and operations to the new, date-compliant software? This job takes years. (Some consultants say that it's now too late to repair code in the old systems. I'm of this opinion. But is there enough time to design and implement a replacement system? Management hasn't a clue.) He may tell you that they are switching to a client/server system. Ask how much the old system cost. Unfortunately, he may have no idea. The new system had better cost at least five times as much, given the difference in prices (inflation). If the new figure isn't at least five times higher than the old one, the company doesn't understand the magnitude of the complexity of the old system and the difficulty of switching. (On this point, I was advised by Cory Hamasaki, a full-time Y2K programmer. For his voluminous Web postings, as well as his humor, search "Hamasaki" on www.dejanews.com ) [Side Note: in a July 2, 1997, letter to me, Hamasaki reported on the U.S. government's present Y2K status. He lives near Washington. "Everywhere I checked last year, they said that they were either working on it or had solved the problem. This year, the same places are saying that it'll be tough, but they're having meetings and are doing their planning. What will they say next year? I expect that they will finally be past denial, and the screaming and scape goating will begin."] If he says they plan to outsource the job to India or Ireland, ask: "Do you have a contract with the software repair company yet?" These companies are now booked up or are close to it. The shortage of mainframe programmers is real, though not yet an insurmountable problem. It will become insurmountable in 1999. A WASHINGTON POST story predicts a U.S. shortage of 500,000 to 700,000. Another estimate places England's at 300,000. (I have posted this information on my Web site under "Too Late?") If the company has "outsourced" the project, request an interview with the contracting firm's project manager. If he says no, then you're onto something. But if you can't get additional information, the interview will be much harder. You will have to interview the person inside the local outfit who serves as the technical liaison. V. HOW TO GET ACCESS TO A PROGRAMMER After you ask the spokesman about the date they discovered the problem, ask him a technical question that seems to be important to you, but which is actually designed to get him to let you get your interview with a programmer. I suggest this question: 7. "Which method of dealing with the problem have you chosen: encapsulation, windowing, or expansion?" He won't know. It's a purely technical question. Nobody with decision-making capacity in management would know the answer. So, his ego isn't involved if he says he doesn't know the answer. You must now press politely but firmly him allow you to interview a programmer. What if he asks why you want to know? Answer: "So that I can estimate how their repair will integrate with the repairs made by other firms in your industry." You do, in fact, need to know this. This, in fact, is the crucial Y2K issue. Any repair of one system that fails to integrate with the industry's other newly repaired systems will either: (1) kick the local computer out of the system or (2) bring down the whole system. If, for example, the banks don't get together on their fixes, you and I will not be getting paid for our brilliant writing skills in 2000 and beyond. (Sad, but true.) If the spokesman thinks you're interested mainly in techie-type information, he may leave you alone with a programmer. That's your goal. If he insists on being in attendance, you must ask questions that seem to be merely technical, but which will get the programmer to tell you facts that will enable you to estimate whether the organization is going to make it. VI. YOU AND THE PROGRAMMER The lower on the hierarchy he is, the less he knows about what constitutes sensitive information. Even if the PR flak knows, he may be hesitant to tell the guy to shut up. He may not want to appear to be stonewalling. The more information you can get early, the less vulnerable you'll be to a "no further questions" announcement. That's why your early questions should be more technical. They will seem innocuous. Ask the programmer about windowing, expansion, and encapsulation. The guy will know. Ask him why they adopted their approach. The answer may or may not be comprehensible. Let him talk. It loosens his tongue. (OK, OK: loosens HER tongue. This is no time to worry about political correctness.) He's in his element. 8. Ask to see some of the code. It will be gobbledegook to you. But ask to see it. 9. Ask the guy to tell you what he has to do to change a date. Sit next to him at his screen and watch how he does it. This will give you an education. Also, it puts him in his element. He's more relaxed. He's in charge. Meanwhile, the spokesman is as clueless as you are. If possible, conduct the rest of the interview while the programmer is staring at his screen. It will give him confidence. Look at his screen or your notebook as much as you can. A lot of eye contact may make him nervous. You're after information about the number of lines of code in the system. Ask him: 10. "Would you have adopted either of the other approaches if you had been facing a system with fewer lines of code?" Let him talk code. He feels more secure. Then ask: 11. "How many lines of code are in the system?" If the spokesman doesn't intervene here, you've got a story. If it's 5 million lines or more, the outfit is in big trouble. It's a huge, complex system. Now you want to find out how many programmers are working on it full-time. Ask (12). He might even tell you. Next: 13. "When did you begin the actual code repair?" If he says that they're still in the assessment or inventory stage, this outfit is dead. Management just doesn't know it yet. Any programming team that is not actively repairing code today will be incapable of finishing the job by 2000. You will want to visit my Web site for confirmation. Read the California White Paper. Once you have the Six Fundamental Questions answered, you can begin to evaluate the company's actual condition. The spokesman may not know how to put all this information together. In fact, he probably doesn't know. He may not even see what you're up to when you ask about which stage the repair is in. If he does, he may invoke "proprietary information." Keep going: you're now after "no comments." VII. ASSESSING THE TASK'S COMPLEXITY 14. "I've heard that there are more languages embedded in a system than just COBOL. How many have you come across? Some systems will have 20 other languages. Assembler is one of the monsters. Find out if any of the system is written in Assembler. (Most of the IRS's system is in Assembler. The IRS told Congress in June of 1997 that it needs $258 million in 1998 to repair its system. The IRS may not get this money, given its admission in January, 1997, of its 11-year, $4 billion failure to consolidate its system into one. Interesting?) Then ask: 15. "How difficult will it be to keep all of the languages and programs in this system to remain integrated after you've made the necessary date changes in the code? One major problem is the compiler: a piece of software that fits all parts of the system together. The compiler that the original programmer used to build the system was not designed to handle 4-digit dates. It may not have been updated -- in fact, probably has not been updated. The company that made it may be out of business. So, if he mentions "compiler," pursue the matter, but don't mention it first. Remember, you're ignorant. Now go for the jugular. You must play the "please help me to understand all this" role. You just don't understand. You had better hope that the flak doesn't either, if he's still sitting in the room. What you're after here is information regarding the vulnerability of this organization. Which kind of dates can blow up the system and why? He won't tell you if he thinks you're looking for bombs, but he'll tell a whole lot if he thinks he's your new-found mentor. If you're looking at the screen or a print-out of indecipherable code, look helpless. Always look helpless. You may even ask the spokesman to sit closer. Let this become a joint-learning experience. 16. "You use dates in your computer software and in old data records. How do you or your software tools find them?" He probably won't volunteer that there are hidden dates in the system's programs. Dates are hidden in subroutines that his "silver bullet" can't find. Finding these is a very difficult, time-consuming task. Ask: 17. "Can the tool you use automatically find every date in the code?" It can't. There is no such thing as a true silver bullet. This is why the task is so painstaking. 18. "How do you spot dates if they're not visible?" Find out. He may tell you about the screwball names, such as some old girlfriend's first name, that the original programmer used to identify his clever, undocumented code. I emphasize "undocumented." Ask: 19. "Do you have the original documentation in front of you when you do the search?" If he says the company doesn't have it (which is probably the case), you've hit pay dirt. Don't let your face show it. Say, "Boy, that must make your work hard." It does. Let him whine about it a bit. Next: 20. "I'm having trouble understanding all this. Am I interpreting this correctly? Is the problem that all of these dates are stored as mm/dd/yy or in any other combination that has yy instead of yyyy?" This is indeed the problem. The fate of the world's economy hinges on the solution to it. Then ask: 21. "If your dates are not already in the yyyy format, do you have to go in and add the extra 2 digits, line by line, through the entire system, including all of the old data files?" Let him explain this to you. Then. . . . 22. "Is it true that when you rewrite a single line of code, this can have unpredictable repercussions in other parts of the system?" Here is the terrifying truth -- a truth that literally threatens the survival of our economy. If a programmer makes a date change in one line of code, this can have unforeseen repercussions -- always bad -- anywhere else in the system. This is why final testing is crucial. It is also why final testing will crash many systems. Then the team will have to start over: search all of the lines of code for non-date anomalies (no date- locating silver bullet tool can help here), rewrite the affected code, and test again. The system may crash again. It probably will crash again. Time will be running out. All this testing, fixing, and re-testing must be done in 1999. This assumes that firms actually meet the formal deadline for testing: December 31, 1998. This is why Chase Manhattan Bank (200m lines of code), Citicorp (400m lines) and AT&T (500m lines) have problems. So do you and I if we expect the economy to stay afloat after Dec. 31, 1999. 23. "If one change can crash the system, do you have to check every line of code in every program to insure that if a date is being used, it can accept the new date format?" He had better say yes. If he says his "silver bullet" tool can help him do this, fine, but he is still limited to about 100,000 lines of code a year. Ask him how many lines of code he can fix in a year (24). Now, you must take him into new worlds where no man has gone before. 25. "How will you test the system after all of the dates have been changed? He had better tell you by parallel testing: running data into the original program and the revised one simultaneously, to see if the revised system crashes. 26. "How long will it take to test all this?" If he estimates anything under six months, use this for comparison purposes with other local systems. The larger the program, the longer the testing period should be. Now, for the killer, the one that is unanswerable: 27. "Won't parallel [mirror] testing require two times your computer capacity and staff?" If the spokesman lets him answer this, he has made a big mistake. The rule for mainframes is "24 x 7 x 365": all year long. These expensive machines are run all day at 90% capacity except for routine maintenance in non- prime time. Here is the Achilles Heel for all of the Y2K repair hopes: there will not be enough excess computer capacity -- memory, data tapes, and staff -- to run full parallel testing if every system that must be fixed is brought to the testing phase. On the other hand, if there is no shortage in capacity, then it's because very few firms have reached the testing phase. (This is my bet.) Finally, if systems aren't tested in 1999, most of them will crash or act up to the point of uselessness in 2000. 28. "Does your firm plan to rent computer time in order to run the tests?" If he says no, ask how they can do in-house. If he says yes, ask this question: 29. "Where will you rent the excess capacity in 1999 if other organizations are also looking to rent time on mainframes?" By now, the guy knows that you know they can't fix it, test it, and implement it by 2000. You've got your story. Nevertheless, keep on going if you're allowed to. 30. "Companies rely on outside vendors for some of their programs. Have your vendors supplied you with Y2K-compliant updates? 31. "Most mainframes are not Y2K compliant, and currently no PC is. Will you be replacing all of your computer hardware as well as your computer software?" 32. "Some of the firms that you interact may not make the deadline. What steps will you have to take to insure data you get from other computers is also Y2K compliant?" VIII. BACK TO MANAGEMENT Now you can go talk to the spokesman back in his office. If he let you get this far, he doesn't understand what you've got. Move away from discussing his firm. Discuss the industry. This lets him relax. 33. "About how many suppliers does the typical organization in this industry rely on?" 34. "Is your firm typical, approximately?" 35. "Has the industry discussed contingency plans if some of its major suppliers should fail to meet the Year 2000 deadline?" Last question (36): "Where can I get copies of anything that the industry has released on Y2K?" If there isn't anything, this industry is headed for a disaster. CONCLUSION As the year 2000 approaches, this story is going to move toward the front page of every newspaper. As yet, stories on local companies have been mostly tapioca pudding. Reporters are accepting at face value the happy- face, "we're on top of this" PR interviews offered by local managers. The articles dutifully report the story: "Yes, there's a problem, but it's being dealt with by the [ ] company." Fact: the only problem being dealt with successfully the company is the reporter problem. If a local public utility goes down and stays down on January 1, 2000, the days of wine and roses will end in your community. The larger your community, the greater the problem. It is fair to give people a warning if they are really at risk. But if they are at risk, nobody in authority at the local public utility will want to admit this. Deferral has become management's job number-one. This strategy works because nobody in the media blows the whistle. Reporters (as with almost everyone else) are in denial about Y2K. Deferral and denial are the Tweedledum and Tweddledee of the Year 2000 story. * * * * * * * * * * * * ABOUT THE AUTHOR Ph.D., history, U. of California, Riverside. Contributor, <The Year 2000 Problem: Strategies and Solutions from the Fortune 100> (1997), edited by Leon Kappelman.  ********************************************** To subscribe or unsubscribe, email: majordomo@majordomo.pobox.com with the message: (un)subscribe ignition-point email@address ********************************************** www.telepath.com/believer **********************************************