Tracking Internet Infrastructure

From: IN%"rre@weber.ucsd.edu" 14-APR-1996 07:00:37.86 From: Phil Agre <pagre@weber.ucsd.edu> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= This message was forwarded through the Red Rock Eater News Service (RRE). Send any replies to the original author, listed in the From: field below. You are welcome to send the message along to others but please do not use the "redirect" command. For information on RRE, including instructions for (un)subscribing, send an empty message to rre-help@weber.ucsd.edu =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Date: Fri, 12 Apr 1996 22:35:54 -0400 (EDT) From: Gordon Cook <gcook@tigger.jvnc.net> Tracking Internet Infrastructure: A Handbook on Business, Technology & Structural Issues Reshaping the Landscape of the Commercial Internet An Anthology of Recent Articles from The COOK Report The Internet has undergone huge changes in the year since the NSFnet backbone service was turned off. It has become a much larger, more stratified, and more expensive entity within which to operate. Since last September we have published a series of articles examining these changes in depth. We have concluded that it might be useful to gather them together into a single volume which is titled Tracking Internet Infrastructure: an organized, indexed Handbook on Internet Infrastructure Issues. This Handbook covers the following critical range of issues: * interviews with key industry players about viable Internet business models, * the hierarchical organization of ISPs through CIDR and routing pressures, * to the viability of the NAPS; * to renumbering and ownership of IP addresses; * to strains on backbone routers and backbone network redesign with switched cores; * to industry views on quality of service issues; to issues of settlements and route charging; * to issues of bandwidth availability in network design; and to ATM as technology savior or dinosaur. The rest of this message contains: (1) Our introduction to the Handbook (2) The Handbook table of contents (3) Description of the Handbook's audience (4) Price and ordering information 1. Introduction A Summary of the Operational Environment Power Consolidates at the Top of a Hierarchical Internet Less than three years ago, at about one twentieth its current size, the topology of the Internet was relatively flat. Service providers could attach to each other via the NSFnet backbone or CIX router and, for the most part, they could consider themselves plugged directly into the Internet with no one up stream of them. This meant that no one was in a position to dictate to them a multiplicity of rules, regulations and costs as part of providing Internet service. One paid the leased line costs for one's own backbone and, in the case of research and education, was connected to the NSFnet backbone for free. The R&E networks paid ANS a transit fee for their commercial customers. If the service provider were commercial, it joined the Commercial Internet Exchange and interconnected at the CIX router for the princely sum of $45,000 a year ($10,000 membership fee, $5,000 port fee and about $20,000 for T-1 line to the CIX. Or one was a downstream customer of Sprint and relied on Sprint to deliver one's packets to the CIX router without having to pay the CIX membership fee. Over the period of about a year Sprint, by allowing ISPs to resell connectivity, and by giving all ISPs downstream of it connectivity to the rest of the Internet Universe, created a situation where customers of Sprint received most of the benefits given those ISPs that connected directly to the CIX router. Some ISPs were shocked in mid summer of 1994 when the CIX proposed that those packets of SprintUs resellers who didn't pay the CIX fee would be blocked. Since then, as the Internet has grown by more than an order of magnitude in size, the importance of the NSFnet and CIX interconnects has either disappeared or faded. A very hierarchical Internet has emerged. One can have several levels of upstream service providers. Such service provision ends at a traffic exchange point known either as a NAP or a MAE. Address space in the form of IP numbers is no longer handed out the interNIC to all ISPs. To get address space direct from the interNIC, you have to do one of two things: (1) show that you have no one up stream. The only way to do that is to directly connect to a NAP or MAE and preferably to more than one. This level of the hierarchy is reached by perhaps 40 of about 2,400 ISPs nation wide. Or (2) an ISP can multi-home (take a connection from two different backbone providers). Perhaps four to five hundred ISPs are multi-homed. But even a multi-homed ISP is unlikely to get interNIC address space, unless it can demonstrate a rapid rate of growth. Connecting at a NAP or MAE, in an effort to put one's own operating environment under one's direct control, is very expensive. Unlike the $45,000 CIX fee the minimum annual cost is $100,000 and up. Once there, providers have to pay additional sums of money to those still higher in the hierarchy to see that their packets are delivered. These sums of money are known as transit fees. Multi-homing is obviously much less expensive. But even at traffic exchange points (NAPs or MAEs) there are additional hierarchies. Some find others who will peer with them. That is to say, they will engage in cost-free transit for a certain percentage of those attached. Those at the peak of the hierarchy are the six service providers who are believed to engage in cost free peering and transit with each other - MCI, SPRINT, UUNET, ANS, PSI and AGIS. These six operate the default free core of the American Internet. Exchange of routing information among them is supposed to be complete, so that none need say: if your routers don't have address XYZ, send packets by default to the next large central player, in the hope that his routers will know. Virtually everyone else is, in one sense or another, a customer of these top six. BBN is a special case. MCI carries all of BBN's routes. Consequently in this sense BBN which is as large as UUNET and certainly much larger than AGIS, is a customer of MCI. We expect this to change by year's end when AT&T fully deploys the network for its partnership with BBN. [Editor's Note: when we fact checked this assertion with sources at BBN we were told that BBN is putting its own national T-3 backbone in place, that significant parts are now operational, that it is at MAE East, MAE West and the Sprint NAP, that it peers with PSI, UUNET, ANS and others, and that its transit relationship with MCI is to reach MCI customers and any Internet sites not directly reachable via its peers.] Those who are able to buy transit at a NAP gain by this action gain the ability to ensure it to their downstream (ISP) customers. In other words transit rights are generally resellable. In this sense, those who purchase transit at the NAPs or MAEs are as fully connected to the core Internet as the big six - with one critical exception. They are renting their core connectivity for a hefty monthly price and without the protection of a long term lease. Readers however should avoid generalizations. Transit agreements are very private, never talked about openly and vary widely on a case-by-case basis. IP Numbers and Other Indicators of Hierarchy Another way of describing the top of the hierarchy is to point out that all six get their IP numbers directly from the interNIC and hand out numbers lower on down in the IP hierarchy to those connected to them. But this hierarchy is not uniformly rigid. The rule of thumb of your upstream provider as a source of IP numbers has some notable exceptions. For example: of the remaining approximately 35 providers which are directly connected to one or more of the major NAPs or MAEs, most get some of their IP numbers direct from the interNIC while others are derived from being attached at some point or in some way to one or more of the central seven. Finally, as we indicated above, the several hundred ISPs who are multi-homed also get their numbers from the interNIC, as may ISPs who can demonstrate extremely rapid downstream growth. An ISP without IP addresses is worthless, since the only thing making it possible to connect an individual to the Internet is the IP number that tells others how to find him. The hierarchy of the Internet is now such that about 80% of service providers must get their addresses directly from their backbone vendors who are often also their competitors. These address blocks are referred to as CIDR blocks. We have recently written extensively about them. Until the fall of 1995 large ISPs were able to connect to one or two NAPs and sometimes negotiate cost free peering. As an option they could then buy transit to the other five from one of the big six. During the past 90 days this has changed. Reports reaching us indicate that cost free peering is available only to those connecting at T-3 speed to three or more NAPS a task that will cost an ISP well over $300,000 a year. But even those who do this will find that there is nothing that will force everyone at each NAP to which they connect to peer with them. If one or more of the majors refuses to peer with a newcomer, that newcomer will have to buy transit to that major from one of those with which it peers or be in the awkward position of being unable to reach a significant part of the Internet. For most of 1995 an ISP connecting to a NAP could by transit for $5 to 15 thousand dollars a month from one of the seven. But reports reaching us now indicate that the six have effectively eliminated the purchase of transit from them as an option for domestic ISPs. These developments effectively shut off connection to a major NAP as a means for an ISP to operate from the top of the Internet hierarchy. The only ISPs that may even attempt to do so now are those with upwards of $500,000 a year to spend on the adventure. In the meantime the NAPs themselves are developing a hierarchy. MAEs either are opening or have opened in Los Angeles, Texas, Chicago and New York. Without the presence of at least the big six at these NAPs, all they are good for is exchanging local traffic among local ISPs and keeping loads on long haul backbones down. The Issue of IP Portability When a customer changes phone service from AT&T to MCI, or visa versa, that customer does not have to change phone numbers. Although the portability of IP address assignments from CIDR blocks has been discouraged it has never been prohibited. But last month a new internet draft (draft-ietf-cidrd-addr-ownership-07.txt) by Yakov Rekhter and Tony Li of Cisco was published. This draft created much debate in mid February when it was put forth by the CIDRD Working Group for elevation to best current practice. Such elevation would put IETF approval behind the practice of a service provider insisting on the return of address space when a customer left. If such a customer were to go to a different service provider for a new connection, every one attached to that customer's network would be forced to renumber their own networks. For a network of any size renumbering would be expensive and, if that network were involved in anything approaching a mission critical application, would become unthinkable. This would very likely mean that any customers buying leased lines to connect a network larger than a few dozen hosts in size will find themselves well-advised to purchase only from an ISP directly attached to one or more of the MAEs/NAPs -- and, therefore, in direct control of its address space. Furthermore, the safest and most conservative action with be to connect to the six providers who are part of the default free backbone. Certainly we suspect that the auto industry would tell its suppliers not to connect outside the direct NAP connected top tier. While there are technical reasons for this policy (fear of the collapse of internet routing if it is not carried out), it is ironic that such policy would greatly accelerate the Internet's stratification into a business service and a consumer service, for those who are there to explore, just for the fun of it. There is also an anti-competitive aspect to implementing such policy, in that large organization customers, which embed non-transferable IP addresses into their network hosts are really locking themselves to a single provider. Should a provider's service becomes "less than optimal," we are sure that providers are aware that the cost associated with renumbering in order to change vendors, limits their customer's options. While network address translation devices (NATs) do exist and will give some customers an alternative, they are by no means regarded as a perfect answer to these difficult problems. It is beginning to appear that, the more the Internet increases in size, the faster that power flows upwards into the hands of a few who, since they are both operators and rule makers for the commercial Internet, would find themselves singled out for accusations of blatant conflict of interest in most other situations. Under these conditions where the fox is essentially in charge of the hen house. Given the nature of a large portion of the customer base (ie large industry and educational networks), we wonder how long customers will suffer these burdens without demanding regulatory relief. 2. Contents: Tracking Internet Infrastructure: Editor's Introduction A Summary of the Operational Environment p. 1 Part One: Internet Business Models Some Large Providers Seek Forum to Push for Internet Service Model Change (Sept. 95) p. 8 Interview with Vint Cerf: Discussion Needed of Benefits Derived from Backbone Resources (Sept.95) p. 12 PSI Satisfied with Cooperative Best Effort Internet Business Model Interview with Bill Schrader(Oct. 95) p. 17 Thoughts on Internet Business Models by Sean Doran (Oct. 95) p. 20 Zero Sum Internet Business Models Vie with Internet Cooperative Culture (Oct. 95) p. 24 Routing Arbiter & Charging for Routing Announcements: Potential Operational and Financial Impacts Assessed (Jan. 96) p. 33 Part Two: Internet Architecture Change & Network Stratification Evolution in CIDR Rules In 1995 Makes Most IP Numbers non Transportable (Sept. 95) p. 43 Constraints of Growth: Provider Based CIDR Likely to Impede Smaller Players Interview with Dave Crocker and Noel Chiappa (Nov. 95) p. 46 Pace of Internet Stratification Increases -- IETF Internet Draft Suggests That Customer Network Renumbering Be Accepted As "Best Current Practice" (Mar. 96) p. 54 Part Three: Backbone Routing Versus Switching Continued Exponential Growth Stresses Internet Backbone Routing Infrastructure (Dec. 95) p. 62 Part Four: Institutions - Sprint; IETF, ISOC, and the NAPs SprintLink Experiencing Employee Attrition - Executives Slow to Provide Staffing Resources Needed for Continued Major Growth (Sept. 95) p. 81 National Science Foundation Domain Name Charges Financial Implications for Network Solutions NSF Rationale Behind Actions Interview with Don Mitchell (Oct. 95) p. 85 Internet Society: Role of Charter Members a Contentious Issue (Nov. 95) p. 87 Transition Pains at the Internet Society (Feb. 96) p. 88 Interview with Paul Mockapetris Who Considers Future of IETF & Finds Software Patents a Growing Obstacle (Jan. 96) p. 89 Interview with Tony Rutkowski Who Finds Internet International Coordinating Group Desirable (Mar. 96) p. 93 No Room at Sprint's Pennsauken NAP (Jan. 96) p. 96 Part Five: Quality of Service Automotive Industry Will Seek Internet Service Provider Certification (Feb. 96) p. 102 Steve Wolff Sees Convergence Between Internet and Telephony (Feb. 96) p. 107 Part Six: ATM and the Technology of Bandwidth on Demand Can Bandwidth Supply Keep Pace With Demand? ATM to the Rescue? An Introduction to a Series of Articles on Role of ATM in the Internet (Mar. 96) p. 111 ATM: Grand Unifying Technology or Brain-Damaged Transport Product? (April 96) p. 114 Bandwidth & Resource Reservation as Factors in Ones Network Provisioning Philosophy -- Can Bandwidth Ever Be Too Cheap to Meter? (April 96) p. 117 Interview with Bellcore's Dave Sincoskie Who Discusses the Internet Future of ATM & Outlines BellcoreUs Interest in Building Network of Interconnected ATM NAPs (March 96) p. 120 InternetMCI Bets its Future on ATM Data Services Marketing & Data Services Engineering Vice Presidents Explain MCI Strategy. Interview with Stephen von Rump and Steve Tabaska (April 96) p. 124 Interview with BBN's John Curran: Has the Internet Derailed ATM? (May 96) p. 132 Index p. 140 3. The Audience for the Handbook Within the national Internet service provider community, Tracking Internet Infrastructure is intended to educate strategists with the complexities facing their engineering and operations staff. Among smaller ISPs it should serve as a tool to bring owner-operators, who are busy 18 hours a day ordering lines, installing them and servicing their customers, up to speed on the changes going on in the environment in which they must operate. LECs and other phone companies will find it useful. Finally familiarity with the issues discussed within the Handbook will provide corporate MIS people with a valuable knowledge base from which to negotiate with their present or future internet service providers. However, since these infrastructure issues are also critical to the continued growth and success of the industry, this Handbook is expected to be a tool for use by those in the banking and investment community. If those in the financial community understand the changing technical and power relationships in the industry, they will be able to improve the quality of their investment decision making. It should also be useful to corporate strategic planners who will be advising their companies' decision making in vertical industry applications. 4. This handbook may be purchased in several ways: A. Single Copy GBC Bound, double sided xeroxed. $275.00 B. Site license: Set of single sided original 600 dpi laser written pages suitable for purchasing organization to reproduce as many copies as it wishes for its employees only. $750 C. A current subscriber without a site license may upgrade to a site license and pay an additional price of $275 to receive the report with full site license privileges. D. A current subscriber with a $650 site license or higher may purchase the report with full site license privileges for $275. To order contact Gordon Cook by phone (609) 882-2572 or email: cook@cookreport.com. ********************************************************************* Gordon Cook, Editor & Publisher Subscriptions: Individ-ascii $85 The COOK Report on Internet Individ. hard copy $150 431 Greenway Ave, Ewing, NJ 08618 USA Small Corp & Gov't $200 (609) 882-2572 Corporate $350 Internet: cook@cookreport.com Corporate Site Lic. $650 http://pobox.com/cook/ for new COOK Report Glossary of Internet terms *********************************************************************