[Nsi-wg] [NSI imp] Some perspective on the discovery service and topology

[John MacAuley]

> - It is not explained why it is better to have a flooding mechanism to
>  distribute documents instead of distributing only the link to discovery
>  services and subscribing directly from the source. This is despite the latter
>  being significantly simpler, while providing the exact same functionality
>  and with the same amount of messages send in the system.

Sorry about that, but I though it was a well known in the team.  I will add a section to more fully describe the issue of sparse control plane connectivity and tree based routing/path finding versus a fully connected meshed set of control plane controllers.

> - The flooding mechanism distributes the exact number of messages / documents
>  by pushing them around. It is not more efficient, it just adds more with
>  lifetime, expiration, etc.


Whether doing a pull or push is really a discussion on convenience versus quicker distribution when a change occurs.  A distributed flood can be achieved with everyone doing a pull as well.  The key is the concept of sparse control plane interconnection, but an aggregator still needing to get access to document generated by uPAs not directly connected.  In these situations, the aggregator needs to get access to the documents through its peer aggregator NSA.

There are also a couple key points:

- Only aggregators need to cache and flood other NSA documents since they represent the control plane aggregation points within the network.  Signalling must traverse them to reach children, and therefore, a peer aggregator NSA must know there are children behind the aggregator and associated control plane connectivity.

- uPA only need to distribute their own documents to directly connected NSA (uRA, Aggregators).

- uRA only need to read documents, and even this is optional.  A uRA can have all configuration locally provisioned and not need to discover the directly connected NSA.  I could decide to only discover the directly connected NSA for interface versions.  It could also chose to discover topologies if it would like to do path finding itself.


> - The only way to got more effient distribution is to aggregate information
>  somehow, but that is - AFAICT - not done.

Aggregation is an optimization needed to solve large scalability issues in the face of large data sets.  If all we are propagating is inter-domain port information, and services offered, then the information for 10k networks is not an unmanageable issue.  Obviously, this will be an nice problem to have.

> - Also, I have no idea what it is this thing is trying to do/solve.

See point on tree pathfinding in a distributed environment.

> Signing and distributing the documents like this, still doesn't solve the issue
> of transitive trust. Say a setup like this:
> 
> A - B - C
> 
> A trusts B. B trusts C. A does NOT trust C. A getting information from C,
> signed by C, does actually do anything, because cannot verify it. What is weird
> is that if A requests a connection service from A to C, through B, it HAS to
> trust B, and implicately C, to provide the service. This issue comes up
> practially everywhere when building networks. You have to trust your provider
> to deliver your data, and trust your peers to not only announce the right
> reachability information, but also to deliver data to end users.

If you don't trust a network why would you route through it?  Seem counterintuitive that you would trust you important service to go through a network you have no desire to trust.  I will leave the discussion up to those who are worrying about the trust issue.  The Discovery Protocol will work without trust.

> Signing documents doesn't solve the issue of transitive trust.

I will leave that to smarter people who can argue the topic.

> To transit into the second major issue: Distributing information like this
> combined with NML means one can acquire network topology descriptions. However
> in our current system (and way of thinking) we only export one topology per
> network. The discovery service system further enhances this view of the world
> by distributing the single description further. However network don't really
> work like that. They have different policies towards different other networks.

Yes, you are correct in that only a single topology per networkId can be propagated, however, multiple networks for an organization can be distributed from the same NSA just so long as each one has a unique network identifier.  In the NSI discussions Jerry specifically argued on behalf of NORDUnet that you absolutely cannot advertise different views of a single topology to different NSA/clients.  This is why he spent a lot of time working through a network abstraction using NML modelling to present a segmented NORDUnet topology. Please discuss it with him and come to resolution on the NORDUnet requirement.

> In our current model, describing topology doesn't tell me how I can use it.
> 
> This makes path finding rather difficult. The reasons for different policies to
> different networks is usally political and economical. Right now we have a
> model that deals mainly with technological aspects of networking, but we cannot
> ignore the other aspects of it.


Inder explained to me the BGP-like concepts you are trying to apply to your NSI deployment.  It is an interesting problem, but it does present a few issues within the NSI model, the foremost of which I believe is it will only work in a chain-based signalling model.  Let me see if I can clearly articulate my understanding and concerns:

1. Each NSA only communicates with its directly connected peer NSA, where control plane peering follows data plane peering.

2. In this model you only need NSA discovery and topology documents from you directly connected peers.  NSA Discovery gives you the adjacent networkId and routing vectors for your peers, and Topology gives you the adjacent port information for the SDP between you and your peer networks.

3. Each peer NSA can be given a different view of routing vectors (topology) to its peers based on routing policies.  For example, A is connected to B who is connected to C and D.  B can tell A it is connected to C, but not D if it has a policy prohibiting routing between A and D.

4. You can also selectively filter out specific routing vectors for networks other than directly connected peers.  for example, if C is connected to Z and you do not what any traffic to/from Z traversing your network you can chose not to advertise these adjacencies.

Based on points 1-4 here are my observations:

1. This model only works if topology is not distributed beyond the direct neighbours.  If it was to be distributed then the true network connectivity could be determined and the routing vectors ignored.

2. This model only works in a chain-based solution where the restricted hop-by-hop routing decisions are made using the advertised routing vectors.

3. This type of filtered route policy cannot be 100% enforced in a tree model.  If someone did have a global view of topology, or decided to send a single connection request to a network (with source and destination STP on the network), there is no way to tell if the single connection should be allowed or denied since their is no additional connection context.  Policy can only be enforced if it dictates that two edge ports cannot be connected.

I like the fact we are now starting to discuss policy since we have spent so much time brushing it aside.  The only thing we need make sure is that any solution also supports tree based routing and connection setup.  This is the distinguishing feature of NSI.  If we devolve to just chain-based signalling then NSI is not the optimal solution for this, and we should be using a proven, industry accepted protocol solution.

> Anyway, this is just the tip of the iceberg. In short: The discovery service
> doesn't solve any problem AFAICT. In fact, it just reinforces some dogmas which
> I think are wrong.

Well, it does solve EXACTLY what it set out to do.  Now whether to entire NSI protocol is wrong as you seem to be suggesting is a different story.

John

On 2014-03-03, at 7:03 AM, Henrik Thostrup Jensen <htj at nordu.net> wrote:

> Hi
> 
> As promised, here is my concerns over the proposed discovery service, a long
> with concers over our current topology model / way of thinking about topology.
> 
> Discovery service:
> 
> - Essentially a mechanism to publish (and subscribe to) documents, with the
>  documents emitted through a flooding mechanism.
>  - Documents are signed by their creater.
> 
> - It is not explained why it is better to have a flooding mechanism to
>  distribute documents instead of distributing only the link to discovery
>  services and subscribing directly from the source. This is despite the latter
>  being significantly simpler, while providing the exact same functionality
>  and with the same amount of messages send in the system.
> 
> - The flooding mechanism distributes the exact number of messages / documents
>  by pushing them around. It is not more efficient, it just adds more with
>  lifetime, expiration, etc.
> 
> - The only way to got more effient distribution is to aggregate information
>  somehow, but that is - AFAICT - not done.
> 
> - Also, I have no idea what it is this thing is trying to do/solve.
> 
> Signing and distributing the documents like this, still doesn't solve the issue
> of transitive trust. Say a setup like this:
> 
> A - B - C
> 
> A trusts B. B trusts C. A does NOT trust C. A getting information from C,
> signed by C, does actually do anything, because cannot verify it. What is weird
> is that if A requests a connection service from A to C, through B, it HAS to
> trust B, and implicately C, to provide the service. This issue comes up
> practially everywhere when building networks. You have to trust your provider
> to deliver your data, and trust your peers to not only announce the right
> reachability information, but also to deliver data to end users.
> 
> I'll just repeat it...
> 
> Signing documents doesn't solve the issue of transitive trust.
> 
> 
> To transit into the second major issue: Distributing information like this
> combined with NML means one can acquire network topology descriptions. However
> in our current system (and way of thinking) we only export one topology per
> network. The discovery service system further enhances this view of the world
> by distributing the single description further. However network don't really
> work like that. They have different policies towards different other networks.
> 
> I'll use the nordunet network to show how these policies work. You can see an
> overview of it here:
> http://stats.nordu.net/stat-q/load-map/ndn-map,,traffic,peak
> 
> We do several things with our network:
> - Provide generic connectivity to our few owners/customers (the NRENs of the
>  five nordic countries)
> - Peer with a number of networks, mainly at NETNOD, AMSIX, LINX, DECIX, along
>  with several Equinix sites and CPS in the US.
> - We also provide US transit to PIONIER and SURFnet (which we have private
>  links to)
> - We also have links to GEANT, these are used for transit towards europeans
>  NRENs
> - We also have services towards RBnet and RUNNET, but I am actually not sure
>  what we provide to them.
> 
> As you can imagine this is a bit of mish-mash with a lot of policies for who
> can use the network in what manner. Here are some things we DO NOT allow:
> 
> - SURFnet & PIONIER to send data to GEANT through us
> - Allow transit between GEANT and our peerings
> 
> Both of these lists can be made very long. Most of this is handled with BGP
> communities (I assume). Essentially NORDUnet is a transit network (I guess all
> network are that in some sense), but how you can use our infrastructure depends
> on the agreement you have with us.
> 
> I can fairly easily create a single topology of our network, but that topology
> doesn't tell anyone how they can use it. For instance our links over the
> Atlantic are rather heavily loaded, but additional capacity over the Atlantic
> costs a significant amount of money (the ANA-100 circuit is attempt of trying
> to reduce costs over the Atlantic for NRENs), so we are in general not going to
> allow connections over those link except for special arrangements.
> 
> Our initial NSI peering are likely to be DeIC, GEANT, SURFnet, and possibly
> SUNET (Onsala). What I'd like to announce wrt. reachability is something like
> this:
> 
> DeIC:    SURFnet, GEANT, SUNET
> SUNET:   SURFNET, GEANT, DeIC
> SURFnet: DeiC, SUNET
> GEANT:   DeIC, SUNET
> 
> Furthermore we have a potential user through DeIC that may want to reach SLAC
> which means transatlantic link, so we might announce a different reachability
> towards DeIC. Our agreement with SURFnet and GEANT might also differ in how
> other networks can reach us through them, and what they can announce other
> networks.
> 
> But this is not really possible currently. I can start to serve different
> topologies depending on request IP, but that is a hack. So essentially, the
> problem boils down to this:
> 
> In our current model, describing topology doesn't tell me how I can use it.
> 
> This makes path finding rather difficult. The reasons for different policies to
> different networks is usally political and economical. Right now we have a
> model that deals mainly with technological aspects of networking, but we cannot
> ignore the other aspects of it.
> 
> Furthermore, with networks the size/layout of NORDUnet (and GEANT, ESNet, and
> more) it becomes rather difficult to perform effecient path finding when having opaque
> networks. I've been toying with idea of being able to split network into
> sub-parts (nordunet would probably be nordic, eu, and us), somewhat inspired by
> some ideas from Jerry, but I haven't gotten around to do something concrete,
> and I am not entirely sure how useful it is.
> 
> Anyway, this is just the tip of the iceberg. In short: The discovery service
> doesn't solve any problem AFAICT. In fact, it just reinforces some dogmas which
> I think are wrong.
> 
> 
>    Best regards, Henrik
> 
> Henrik Thostrup Jensen <htj at nordu.net>
> Software Developer, NORDUnet
>