[SAGA-RG] SAGA Message API Extension

[John Shalf]

On Jan 18, 2007, at 10:36 AM, Andrei Hutanu wrote:

> Hi Andre,
>
> Thanks for writing this up. I think it looks very good but I
> still have just a few quick comments to the latest version..
>
> 1) Motivation - can make a bit stronger perhaps :
> a)to break out of the stream semantics (byte-oriented, strict  
> ordering and reliability)
> and b)to remove the burden from the application programmer of dealing
> with arbitrary-sized entities such as Ethernet packet size.
>
> 2) I see ordering is enforced, could that be an option?

I think ordering is *not* enforced, but I do wonder if it should be  
an option or a channel property (certainly semireliable will likely  
result in some reording whereas a TCP channel would enforce ordering  
of the messages for instance).

This is a controversial topic in the HPC message passing community  
(whether msg. ordering is a good or bad-thing to enforce in at the  
hardware level).

> 3) community : it would be nice if some of the active transport  
> protocol developers would
> give their input. I would think the UDT team : Robert Grossman and  
> Yunhong Gu and the EVL team :
> Venkatram Vishwanath should take a look at this. Perhaps the XIO  
> team as well.
> They may all be already involved but just in case.

That's actually a pretty good suggestion.  Perhaps its not worth  
holding up the current draft submission, but getting the lower level  
transport poeple on board is pretty powerful.

> Andrei
>
> Andre Merzky wrote:
>
>> Quoting [John Shalf] (Jan 17 2007):
>>
>>> I agree.  I think the reliability semantics should be a property  
>>> of  the channel (and hence underlying protocol) and not something  
>>> to  switch at runtime.  However, I think the clients that use  
>>> the  connection should be able to query what the channel  
>>> properties are  (or define the minimum requirements and throw an  
>>> exception if they  cannot be met).  This is just a little  
>>> introspection (not really a  deep or fancy coding underneath).
>>>
>>
>> I agree - that can safely be done with a read-only
>> attribute.
>>
>>
>>
>>>> I wanted to avoid a 'reliability' attribute on the endpoint,
>>>> as I don't see a reasonable easy way to switch reliability
>>>> in mid stream.  It is most likely that changes in
>>>> reliability policy would require different protocols (only
>>>> few protocols will be able to serve both ends), and would
>>>> hence need a new connection setup anyway.  Not sure if that
>>>> is reasonable though.
>>>>
>>>> Anyway, that would leave us with specification of
>>>> reliability on connection setup (i.e. endpoint construction)
>>>> - right now that is done via the scheme part of the URL.
>>>> Would a flag be more appropriate/explicit?  Probably...
>>>>
>>>>
>>>>
>>>>> 		b) what about if the message arrives more than once? (eg.
>>>>> 		redundant  copies of messages can occur in practice for a 		 
>>>>> number of
>>>>> unreliable  or semi-reliable messaging protocols).
>>>>>
>>>> Good point.  IMHO we should specify that messages arrive 'AT
>>>> MOST ONCE' (unreliable) or EXACTLY ONCE (reliable).  I don't
>>>> think that we need to support additional modes, nor the
>>>> complete spectrum of (un/)reliable transmission modes.  Or?
>>>>
>>> I cannot remember the document, but there was a spec for  
>>> defining  reliability of another kind of messaging bus.  It had
>>> 	unreliable: message may or may not arrive (typical UDP).
>>> 	semi-reliable: message must arrive (will be retransmitted if  
>>> not  acked).  Source keeps resending until it sees the ack.  It  
>>> is  possible that the ack is lost, in which case, the message may  
>>> arrive  at the client twice (so this is a Message will arrive at  
>>> least once,  and possibly more than once.)  This protocol is  
>>> useful since the  message destination does not need to maintain  
>>> complex message  state... it just acks messages that arrive (very  
>>> simple).
>>> 	reliable: the message must arrive and it will only arrive once.
>>>
>>
>> I added to the spec:
>>
>>   The available realiability levels are:
>>
>>   \up
>>   \begin{tabbing}
>>    XXXXXXXXXX          \= \kill
>>    |Unreliable|:       \> messages MAY (or may not) reach the
>>                           remote clients.\\[0.3em]
>>
>>    |Atomic|:           \> |Unreliable|, but a message  
>> received                           by one client is\ 
>> \                        \> guaranteed to (MUST) arrive
>>                           at all clients.\\[0.3em]
>>
>>    |SemiReliable|:     \> messages are guaranteed to  
>> (MUST)                           arrive at all\\
>>                        \> clients, but may arrive more than once.\\ 
>> [0.3em]
>>
>>    |Reliable|:         \> all messages are guaranteed to  
>> (MUST)                           arrive at\\
>>                        \> all clients.\\[0.3em]
>>   \end{tabbing}
>>
>> That is similar to your list, but adds the 'atomic' one.
>> Not sure if that is useful (sounds like) or easily
>> implementable (??), but I found it in a list of relibility
>> modes for message transmissions which looked sensible, and
>> seems to be close to what you propose.
>>
>> [ about point-to-point ]
>>
>>
>>> That does address one of the cases (not exactly the one I was   
>>> thinking of).
>>> Also should allow -1 to specify *any-number-of-clients* at the   
>>> underlying protocol's discretion.
>>>
>>
>> Yes, definitely, thats in the spec already.
>>
>>
>>
>>> But now for a semantic thicket....
>>>
>>> Case 1: I connect to a named destination and I am joined  
>>> together  with everyone else who has connected to that port.  In  
>>> this case, any  message I send will be broadcast to everyone else  
>>> and vice verse  (message bus).  This is the equivalent of a  
>>> publish-subscribe service.
>>> Case 2: I connect to a port and I own the service.  This appears  
>>> to  be the case you are setting up above as it only allows one  
>>> client to  join the service.
>>> Case 3: I connect to the port and for a sub-process that does  
>>> not  share the messages with the other clients that have  
>>> connected to that  port (like an HTTP server).  I don't quite see  
>>> how that kind of point- to-point message service is supported.
>>>
>>> I think we need an attribute for the message port that says  
>>> whether  it is a bus or a point-to-point.  In addition, I like  
>>> the idea of  setting the message queue length (as you have  
>>> above).  I was not  thinking of that, but I can see the value of  
>>> creating a first-come- first-serve message port as well. (if that  
>>> is not too complicated).
>>>
>>
>> Got you (I think) :-)  Yes, that makes sense.  I added a
>> 'topology' enum and attribute, which is handled similarly to
>> the 'reliability' property (static over connection/endpoint
>> lifetime, inspection via ReadOnly attribute).
>>
>>
>> New draft is attached, with new sections marked.  It would
>> be great if you could review the paragraphs about connection
>> topology.
>>
>> Thanks!
>>
>>  Andre.
>>
>>