iqnets: opportunistic XYZ, e.g. "begin xmit"
Every round trip between 2 peer nodes incurs 2x the average link latency. Google did substantial design and testing for various TCP HTTP(s) speedups. One of the concepts ISTR bandied about was opportunistic data sending - as long as the server is happy with the initial packet(s) sent by the client, and the client has included its initial request(s) into those initial packets, then the server begins to transmit data to the client, without waiting for ACKs of ACKs and further requests. The concept, at least as I understand it, is to, where possible, include initial data/ link set up/ etc requests, as early as feasible in the p2p link establishment packets, to minimize or eliminate round trips and thereby drastically reduce at least initial latency, and possibly also additional latency. For example think of a web page which contains hrefs to content, which the client only knows exist when it receives the base web page, so then of course must otherwise make a new request for the content identified - if the server knows what the client will need to ask for, it can automatically/ opportunistically send that data, without waiting for the obvious request. There may be peer node negotiation for different types of opportunism, e.g.: - on first ever new node contact, nodes may choose to auto (opportunisticly) hand out X cache undertaking/ promise - nodes may request "may I opportunisticly assume I may begin a "bulk fill" link with you, if I need (or am requested) to do so? - certain types of opportunism may be enabled by default, unless overridden by conf - other types of opportunism may only (by default) occur by nego - we ought be diligent in attempting to identify all possible types of opportunism that overlay nets may utilise/ provide for - we that we may opportunisticly optimize for as much opportunism as possible - everything possible to minimize number of round trips - algos which scale linearly (or better) with time, rather than b/w, and rather than exponentially in any unit A thought giving rise to the above is that notwithstanding our best efforts to stay "optimally" below link b/w and other limits, for many reasons any of these attempts will, at different times, fail. A link at a b/w of say 2MiB/s may oscillate or otherwise vary over time, up to 2MiB/s. When we are xmitting at say 1.95MiB/s, and phys nodes out of our control (such as an ISP's router) shape downwards, even temporarily, UDP packets get dropped. In many, most or all cases to start with, we will not have QoS at the (from our overlay net point of view) physical network layer. - This means we may have minimal to no control over which UDP packets get dropped. - This QoS issue is a fundamental area we need to expand our understanding of. - Longer term, iqnets shall be a motivator to end users, ISPs and GT-* folks, to push QoS down to the link layer. - When we do achieve internet wide QoS contracts at the network layer, a privacy issue (depending on your threat model) will be which QoS modes to utilize - e.g. you may be better off using "bulk fill", rather than "telephone audio" class QoS, in order to better hide your important phone call. - notwitstanding use cases and threat models, we don't fail to maximise provision for QoS, just because some use cases won't use it
- When we do achieve internet wide QoS contracts at the network layer, a privacy issue (depending on your threat model) will be which QoS modes to utilize - e.g. you may be better off using "bulk fill", rather than "telephone audio" class QoS, in order to better hide your important phone call.
One way to use bulk fill for real time data, is for links (i.e. peer nodes), to simply "maintain excess headroom during requisite (phone call) time". This implies the need to hide a node's (downwards) phys link utilization: - either all nodes always reserve a relevant phys link %, e.g.: - 2%, or 10KiB/s, whichever is greater, - unless total phys link is less than 30 KiB/s, in which case this node must essentially act as a client only node (a comparatively unsafe option (presumably)) Protocol for bulk-fill "telephone audio" link nego: Step 1: Nodes A and N agree intention to make "max secure" phone call: - node A originates the phone call request, contacting N: - phone call request - asap - using only bulk fill link QoS class - only via "trusted" middle node(s) - A "trusts" B to some degree - N replies to A with "ACK, please set up the links" Node A now attempts to nego "headroom links", to minimize packet drop outs and thereby maximize audio quality of the AN phone link: Step 2: Node A and middle node B, nego "headroom" links AB, and BN: - A requests of B to "reserve excess headroom for real time b/w W, of intended duration ~T, beginning "asap". - B checks its current link undertakings (bulk, r/t, total b/w vs b/w availability etc), and offers to A something like: - I can ACK your request not before 13 minutes, (presumably due to current link contracts); I will hold open this offer for you, for 10 seconds, i.e. I will not enter new link contracts before $NOW + 10s. - A considers this, and since B is the only node A is presently willing to entrust with such a request, A re negos with B: - A first calculates two random time periods, to be buffer time before and after its phone call with N, say: 347 seconds and 13 seconds; we note that statistically significant (in a cryptographic sense) random extensions of such time windows, is something that needs careful mathematical analysis by someone competent in the field - usually, we reduce, not increase, randomness when we do such things (math don't care how good our intention is); and for "moar headroom" windows to be useful, such windows need to not only be random in respect of an actual phone call, but also random in respect of "no phone call at this time period, but we reserved headroom anyway", so that headroom reservations all appear normal and more importantly, completely random; "Achieving randomness in practice, is not trivial." - A to B: Please reserve headroom for me, and a link for me, as follows: - begin time $NOW + 10 mins - b/w 7KiB/s (effectively an audio phone call) - duration 347s + 30 minutes + 13s - we note that human phone calls can vary wildly in their actual duration, as compared with expected duration when a user first attempts a phone call - a half hour call may end up being under 20 seconds, or over an hour and a half, etc - maximising hiding of high value phone calls, means the users (the people in the phone call/ conference), MUST be aware of the "max security" window within which they are operating, and that the call quality may reduce after that time window. - Node B: - accepts this request - sends an ACK to A - sends an ACK/ intent to connect, to N - N sends an ACK back to B (optional, and possibly not done, just "virtually ACKed" ie. assumed - we assume A did its job properly and first nego'ed with N, and we don't introduce unnecessary additional ACKs without reason.) - it may be that A should ACK to N Step 3: - at agreed time, A phones N An open question TODO: if all nodes in iqnets are bound to implement random "headroom" windows, at random times, and for random durations, can the actual headroom be measured and/ or tested by peer nodes? - if so, we would have a mechanism to empirically test and therefore utilize untrusted nodes, albeit at entirely random (unpredictable times), to make high value phone calls using "bulk fill contracts", without the untrusted nodes knowing that this is what we are doing! - this would be a very desirable property for any overlay network - but we must think like the government stalkers (who are out to illegally monitor us), and who have very deep pockets, and who run an abundance of trojan nodes: - such tojan nodes will say to their peer nodes that they are undertaking "headroom" contracts at random times for random durations, and yet may be doing no such thing at all, in order to firetruck us over a barrell Next we consider that our effort to push QoS down to the network physical layer and up through the entire stack, may well ultimately result in much greater ability for us all to maximal utilize global network b/w, at the same time as reducing packet loss to an absolute minimum. This would be an absolute win for everyone, including ISPs and GT* backhauls. - bittorrent peers know exactly how much they request of one another, and can therefore readily use nego net "NegoNet, for n_ggers who can't configure their torrent client!" - phone calls are an instant win - nego b/w, choose optimal codec for agreed b/w - web servers could rate limit per nego'ed link, per client etc
On Thu, Oct 31, 2019 at 02:36:27PM +1100, Zenaan Harkness wrote:
- When we do achieve internet wide QoS contracts at the network layer, a privacy issue (depending on your threat model) will be which QoS modes to utilize - e.g. you may be better off using "bulk fill", rather than "telephone audio" class QoS, in order to better hide your important phone call.
One way to use bulk fill for real time data, is for links (i.e. peer nodes), to simply "maintain excess headroom during requisite (phone call) time".
This implies the need to hide a node's (downwards) phys link utilization:
- either all nodes always reserve a relevant phys link %, e.g.: - 2%, or 10KiB/s, whichever is greater, - unless total phys link is less than 30 KiB/s, in which case this node must essentially act as a client only node (a comparatively unsafe option (presumably))
- or, we institute a randomization protocol, so that at one time or another links are "normally, but randomly" headroom shaped down a bit Again, achieving actual randomness, ain't easy. And we must (!) modulo against user requirements as well, every time we plan to institute some randomization scheme. Further, for the use case of phone calls, we are talking randomized appearance of "headroom shaped routes", and -not- randomized appearance of "headroom shaped links" - these are similar, but quite different, things. Phone calls require routes. If all "usable" links are independent, (link set) intersections constituting usable links would be greatly diminished - ahh, probably, since perhaps not - if we consider links between known friends, where such links (when they randomly appear) can be used as single hop p2p link for phone calls. Maths, especially statistical, set and algebraic math, can be pretty interesting since it can be so useful.
On Thu, Oct 31, 2019 at 03:24:50PM +1100, Zenaan Harkness wrote:
On Thu, Oct 31, 2019 at 02:36:27PM +1100, Zenaan Harkness wrote:
- When we do achieve internet wide QoS contracts at the network layer, a privacy issue (depending on your threat model) will be which QoS modes to utilize - e.g. you may be better off using "bulk fill", rather than "telephone audio" class QoS, in order to better hide your important phone call.
One way to use bulk fill for real time data, is for links (i.e. peer nodes), to simply "maintain excess headroom during requisite (phone call) time".
This implies the need to hide a node's (downwards) phys link utilization:
- either all nodes always reserve a relevant phys link %, e.g.: - 2%, or 10KiB/s, whichever is greater, - unless total phys link is less than 30 KiB/s, in which case this node must essentially act as a client only node (a comparatively unsafe option (presumably))
- or, we institute a randomization protocol, so that at one time or another links are "normally, but randomly" headroom shaped down a bit
Again, achieving actual randomness, ain't easy.
And we must (!) modulo against user requirements as well, every time we plan to institute some randomization scheme.
Further, for the use case of phone calls, we are talking randomized appearance of "headroom shaped routes", and -not- randomized appearance of "headroom shaped links" - these are similar, but quite different, things.
Phone calls require routes.
If all "usable" links are independent, (link set) intersections constituting usable links would be greatly diminished - ahh,
s/links/routes/
probably, since perhaps not - if we consider links between known friends, where such links (when they randomly appear) can be used as single hop p2p link for phone calls.
Maths, especially statistical, set and algebraic math, can be pretty interesting since it can be so useful.
- B checks its current link undertakings (bulk, r/t, total b/w vs b/w availability etc), and offers to A something like:
- I can ACK your request not before 13 minutes, (presumably due to current link contracts);
I will hold open this offer for you, for 10 seconds, i.e. I will not enter new link contracts before $NOW + 10s.
Such protocols lead to information exposure (i.e. privacy bugs), since if stalker node S "the snake" makes a request of node B during this 10s window, and gets an "unusual response latency", then S can gather than B may be holding back additional contracts due to the above phone call "headroom link" proto. This may not be a lot of information, and may ultimately not be useful ("exploitable"), but we must keep such "protocol level" exposures in mind. If there are various reasons, including say "arbitrarily random" request-response latency occurrences, we may either mitigate, or completely eliminate, such information exposures.
Step 2:
Node A and middle node B, nego "headroom" links AB, and BN:
- A requests of B to "reserve excess headroom for real time b/w W, of intended duration ~T, beginning "asap".
- B checks its current link undertakings (bulk, r/t, total b/w vs b/w availability etc), and offers to A something like:
- I can ACK your request not before 13 minutes, (presumably due to current link contracts);
I will hold open this offer for you, for 10 seconds, i.e. I will not enter new link contracts before $NOW + 10s.
A consequential issue arising in relation to "network layer contracts", is one or another resource starvation attack: - if our protocols are sufficiently weak (attackable), then - stalker nodes S1, S2 ... Sn can "gang up" against node B - e.g., nodes Ss may repeatedly (say in random round robin fashion) make a certain resource request of node B, in order to starve either or both of node B, and node B's friend nodes, from that resource If node B naievely/ eagerly ACKs such contract requests, without suitable mitigation protocols then both B and B's friend nodes, will be starved of that resource. "Unconstrained eager contract agreement, is a resource starvation attack exposure point." Example resource starvation mitigation protocols: - headroom reservation, with the headroom available to preferred nodes only; watch out for information leakage here - local node and friend node preferencing; - time limiting contract ACKS, so that maximum latency "until next availability window" is reduced; but as usual, be very cautious about information leakage here Depending on protocols, we may steer strongly or weakly to different virtual network topologies, and the difference between these topos must be considered at some point: - strongly preferenced/ maintained F2F links, vs - weak F2F preferencing - information leakage every time a resource is "repurposed", unless there are protocol mitigations (primarily randomization of one form or another) - so randomized shifting between friend and anon nodes may bode well for global network flexibility, whilst maintaining reasonable preferred node resource availability, whilst hiding actual usage of said resources by said preferred nodes - ultimately we are balancing between various, sometimes competing, sometimes complementary, goals: - privacy - anonymity - latency - resource usage (are we on average actually making use of offered resources) - efficiency (what are the overheads to deliver the resource) - connectivity (to whom (which nodes) is a resource available) - availability (when is a resource available) By handling billions of nodes at core and not as an afterthought, we may be able to achieve desirable combinations of the above characteristics in relation to various resources. Again, network resources include: - b/w (bandwidth) - b/w over duration - latency - latency over duration - data storage - data storage over duration - link count - link count over duration Are we yet missing a basic "network level" resource? PS: when we use the term "latency" as a noun referring to a resource, we mean "desirable latency" which is usually low latency.
Again, network resources include:
- b/w (bandwidth)
- b/w over duration
- latency
- latency over duration
- data storage
- data storage over duration
- link count
- link count over duration
Are we yet missing a basic "network level" resource?
("over duration" means "availability", so s/.../availability/)
Again, network resources include:
- b/w (bandwidth)
- b/w availability
- latency
- latency availability
- data storage
- data storage availability
- link count
- link count availability
Are we yet missing a basic "network level" resource?
The lone neurone strikes again: A typical network use pattern is: - connect - begin download So link reservation windows where the pertinent parameter is time T, may be complemented with "data xmit" total, either at link setup, and/ or "shortly" after link has been set up. So this protocol might be e.g. - connect, default time reservation e.g. 2 minutes - begin download - (e.g. HTTP) headers identify the download as being say 160MiB - link/switch request to modify "T=2 minutes" to "xmit=170MiB" - ack or nack from peer nodes as they are willing/able
On Wed, Dec 04, 2019 at 12:23:57PM +1100, Zenaan Harkness wrote:
Again, network resources include:
- b/w (bandwidth)
- b/w availability
- latency
- latency availability
- data storage
- data storage availability
- link count
- link count availability
Are we yet missing a basic "network level" resource?
The lone neurone strikes again:
A typical network use pattern is:
- connect - begin download
So link reservation windows where the pertinent parameter is time T, may be complemented with "data xmit" total, either at link setup, and/ or "shortly" after link has been set up.
So this protocol might be e.g.
- connect, default time reservation e.g. 2 minutes - begin download - (e.g. HTTP) headers identify the download as being say 160MiB - link/switch request to modify "T=2 minutes" to "xmit=170MiB" - ack or nack from peer nodes as they are willing/able
Of course, X bps for T period, is a similar way to say transfer Y MiB "at your convenience" and nego bandwidth. Remember that (total) xfer = bandwidth x time and so of course time = xfer / bandwidth and similarly bandwidth = xfer / time , and so pick the forumla you need for the inputs you have, to determine the request/ consequence (e.g. what to nego).
An open question TODO: if all nodes in iqnets are bound to implement random "headroom" windows, at random times, and for random durations, can the actual headroom be measured and/ or tested by peer nodes?
Possibly "dropped packet count per unit time" is the only measure available? In any case this metric is a core metric - effectively aggregating end user node, ISP, and GT-* "dropped packets" performance. IQNets should inherently improve the global "dropped packets" performance, not the least reason being that we shall, at core, optimize on this specific metric (amongst others). It may well be that handling this metric "very well", may satisfy the bulk of our QoS, and node "peer node performance monitoring" needs! When the fundamental definition of a "good" peer is one which is able to minimize dropped (and therefore needing to be re-sent) packets, modulo "is able to send packets within specified latency", it seems we may have the bulk of what we need for a (proper) functional overlay network.
- if so, we would have a mechanism to empirically test and therefore utilize untrusted nodes, albeit at entirely random (unpredictable times), to make high value phone calls using "bulk fill contracts", without the untrusted nodes knowing that this is what we are doing!
- this would be a very desirable property for any overlay network
- but we must think like the government stalkers (who are out to illegally monitor us), and who have very deep pockets, and who run an abundance of trojan nodes:
- such trojan nodes will say to their peer nodes that they are undertaking "headroom" contracts at random times for random durations, and yet may be doing no such thing at all, in order to firetruck us over a barrell
participants (1)
-
Zenaan Harkness