[SAGA-RG] SAGA Message API Extension

['Andre Merzky']

Hi Hartmut,

I certainly do understand your unease with implementation
managed memory, and, yes, its the only place in the spec
where we have that at the moment...

The main (and actually only) reason for this is the ability
to avoid memcopies.  If you think of large messages, you
definitely would like to have a way to avoid the copy from
implementation memory (where it was received, and is
buffered anyway) into application memory.  

For streams, that problem is not that bad, as you can delay
the low level read until the stream.read() method is really
called, and then you can read directly into the application
level memory.  For messages that does not work, as you do
not know the size of the message beforehand.  If you call
test() (or check() now), the message must (at least partly)
already recide in the application memory.  

Do you see any other possibility for avoiding the additional
copy from impl. memory to application memory?

About your second comment, the resuse of the msg (or
message) object in other parts of the API: I am not sure
about this, really.  The main difference to stream and file
byte buffers is the opacity of the message: for files and
streams, you can write an int, five floats, another int etc.
There are no contraints on when the data are transferred
etc, and if, you flush the file.

For the message API, the message is to be transferred as an
entity, but can have internal structure.  So you want to
pack an int, five floats, and another int in a message, and
then ship that as an opaque entity.  w/o message object that
is awkward.  OTOH, a message object for that would be
overkill for the file and stream API I think.

However, I think we CAN reuse the message abstraction in
several places, e.g. in the advert service, or, as we
discussed in other mails, to create specific message types
for certain application domains, or with typed data support
etc.

Well, these are my 2 cent... :-)

Cheers, Andre.



Quoting [Hartmut Kaiser] (Jan 20 2007):
> 
> Hi Andre, 
> 
> > here is the updated draft of the SAGA Message API, in 
> > preparation for OGF-19.  It is also available in CVS, as usual.
> > 
> > We would be happy to get feedback on the list of course, so 
> > don't feel oblidged to hold back your comments for OGF-19
> > :-)
> 
> Sorry, it took me some time to digg through the new spec... I had to start
> implementing the stuff to have comments (I'm perhaps too old to do
> programming other than in an intuitive way :-P).
> 
> The comments I have relate to the API part of the spec. 
> 
> Frankly, I don't like the design of the 'msg' object, I even think it's not
> viable to make it a first class SAGA object (derived from saga::object) - is
> there a use case requiring that?
> 
> The msg object in its current design mixes two different paradigms into one
> object (remember: every kind of bool in an API is a strong hint for a flawed
> design):
> a) saga::msg can be a wrapper object for application memory (size > 0) and
> b) saga::msg can be a object managing memory on behalf of the SAGA
> implementation (size == -1)
> I strongly believe that we don't need b) and even if we do, it should be
> implemented in a separate abstraction. 
> 
> Why do I think we don't need b)? 
> 
> Instead of relying on the SAGA implementation to manage the memory for the
> application (BTW: this is the only instance in the whole spec, where we
> require the implementation to do this), an application always may write
> (syntax is C++):
> 
>     endpoint ep(...);
>     ep.connect(...);
>     int size = 0;
>     if (ep.test(..., &size)) {
>         void *data = new unsigned char[size];
>         ep.recv(data, size);
> 
> Or even better:
> 
>         ep.recv(buffer(data, size));
>         ...
>         delete data;
>     }
> 
> i.e. The application always can handle the memory itself, without having to
> go through too much overhead.
> 
> This brings me to the second comment: by implementing the msg/buffer object
> solely for the purpose of wrapping up application memory this msg/buffer
> could be reused for other packages as well: file::read, file::write,
> stream::read, stream::write, rpc::parameter::buffer - to name a few. 
> 
> BTW: I silently introduced a const_buffer and mutable_buffer objects,
> combined with a buffer() generator function in the C++ implementation of the
> saga::file package to improve memory handling in the API.
> 
> Regards Hartmut
> 
> 
> 
> 



-- 
"So much time, so little to do..."  -- Garfield