[saga-rg] SAGA strawman API: comments to Stream functions
Thomas Eickermann
th.eickermann at fz-juelich.de
Tue Dec 13 11:02:17 CST 2005
Hi Andre,
sorry for responding so slowly, but times are tough as usual in
December ...
I finally surrender to your arguments. I was not aware that SAGA
will provide callbacks (via the Monitoring interface ?). Using such
callbacks, you can probably solve the problem that i have raised
in most cases. The rest would fall under the 80/20 rule.
Thanks for the discussion,
Thomas
Andre Merzky wrote:
> Hi Thomas,
>
>
> Quoting [Thomas Eickermann] (Dec 02 2005):
>
>>Hi Andre,
>>
>>thanks for your answer. I agree with your comments that access
>>to raw descriptors is somewhat problematic.
>>Nevertheless i regard them as quite important for implementing
>>real-life interactive applications. Two examples:
>>
>>1. AVS/Express allows to trigger a module if an event occurs at a
>>descriptor. I do not know of any other way to implement stream
>>communication within AVS/Express and i imagine, that the situation
>>is quite similar for other tools like OpenExplorer.
>>
>>2. If you use some GUI toolkit (e.g. Qt), you can also register callbacks
>>for events on descriptors. If such a descriptor is not available, the
>>only thing you can probably do is to use timers and polling. There are
>>obviously good reasons to avoid that, if possible.
>
>
> I agree about the importance.
>
>
>
>>See further comments inline.
>>
>>Best regards,
>>Thomas
>>
>>
>>Andre Merzky wrote:
>>
>>>Hi Thomas,
>>>
>>>sorry for the late answer... *sigh*
>>>
>>>
>>>Quoting [Thomas Eickermann] (Nov 10 2005):
>>>
>>>
>>>>* in both use-cases there is an interactive visualisation/steering
>>>>application sitting at one end of the stream. Such applications
>>>>typically use some kind of GUI-toolkit with operates in an
>>>>event-callback mode or similar, where callbacks can be registered
>>>>for events occuring at socket/pipe/...-descriptors. On the other hand, in
>>>>many implementations, a Stream will somehow use a socket, pipe or
>>>>something else with a descriptor. Therefore we need a function that
>>>>returns
>>>>the descriptor associated to a stream:
>>>> fd = descriptor(Stream);
>>>
>>>
>>>This is complicated, as we do not know _if_ there will be a
>>>native descriptor available, and if operations on that will
>>>interfere with the implementation.
>>>
>>
>>availability:
>>right, i was somewhat unprecise. What i meant was the native
>>respresentation of a socket in the given language (a descriptor
>>in C, an IO::Socket obejct in Perl and probably there is something
>>like that in most languages.
>
>
> I was unprecise as well I guess: you are right, there'll be
> a native representation in most languages, not doubt.
>
> What I mean with availability is best shown in Globus-XIO:
> I simply do not know _how_ I could obtain the native
> descriptor from a globus-XIO based implementation: there is
> no globus-xio call exposing the descriptor.
>
>
>
>>>E.g. the implementation might be based on Globus-XIO or on
>>>Reliable UDP. For the first I do not know if a descriptor
>>>is availbale via the API, for the second there are two
>>>descriptors (one tcp, one udp) the application would need to
>>>handle.
>>
>>To handle the second case, 'descriptor' would have to return
>>an array of descriptors. I don't see that in the first case
>>an implementation could return something better than a NULL-array.
>
>
> It is dependend on the middleware then: the application
> needs to be aware of the fact that saga implementation
> returns a single fd, but saga implementation returns 2 fd's,
> with specific handling requirements.
>
> I am not saying that this is impossible. Just would like to
> be sure that this is the best way to go...
>
>
>
>>>Also, the implementation might transparently add a protocol
>>>(say checksumming), and direct tempering with the desfriptor
>>>might very well break that protocol.
>>
>>I agree, the only thing the user is allowed to do with the
>>descriptor, is to call 'select'. Read, write, seek, close
>>would most likely break the protocol.
>>
>>However, this cannot be enforced, and i am aware that this
>>conflicts the principle of hiding the implementation from
>>the user (to minimize the risk that the user breaks something).
>
>
> I checked with QT, and I think there is a way to handle
> SAGA events in QT:
>
> ------------------------------------------------------------
> // this be a callback for a saga stream object, which gets
> // fired on new data arriving etc.
> void my_saga_file::saga_cb (void)
> {
> emit update_gui ();
> }
>
> my_gui g;
> my_saga s;
>
> QObject::connect (&s, SIGNAL (update_gui (void)),
> &g, SLOT (saga_cb (void)));
> ------------------------------------------------------------
> (The code above is probably not really correct, but you get
> the image...)
>
> my_saga::saga_cb would need to registerd to saga as callback
> for events on a stream. Then, if such an event occurs, the
> callback would trigger (emit) a QT event (update_gui) which
> would be received by your gui class, and cause, well, the
> gui to update etc.
>
> The QT event mechanism would allow to use any data available
> in the callback (e.g. string read from the stream) to be
> signalled to the gui loop as well.
>
> So, would that fullfill your requirements?
>
> Am am not so sure about AVS and other packages. I did not
> work with AVS for quite some time, but IIRC, the events have
> been coupled much tighter to selects on descriptors, with no
> alternative rounds? However, for all other packages I know
> (not many), a trivial (e.g. 3 line) translation from saga
> callbacks to 'native' events seems to exist.
>
>
> Cheers, Andre.
>
>
>
>>>So, we might be able to add the API function, but it is
>>>difficult to define what it should return in the cases
>>>outlined above...
>>>
>>>
>>>
>>>
>>>>However, users have to be aware, that several Streams
>>>>may share the same decriptor (e.g. in VISIT, if several
>>>>Streams are tunneled through a single connection of a
>>>>different protocol like ssh).
>>>
>>>
>>>
>>>>* a general remark on Streams: i like the idea of keeping the Stream-API
>>>>close
>>>>to the BSD socket API. However, as Andrei mentioned, we also often have
>>>>to deal with more message- or block-oriented communication patterns.
>>>>In the internal API the we use in the VISIT toolkit, we have therefore
>>>>added a timeout parameter to all functions and slightly changed the
>>>>semantics.
>>>>For example
>>>>
>>>>nread = read(stream, buffer, size, timeout);
>>>>
>>>>works as: try to read size bytes from stream, return if either:
>>>>- size bytes have been read, or
>>>>- timeout seconds have passed (wait forever, if timeout < 0), or
>>>>- an error has occured
>>>>the function returns the number of bytes read or -1 in case of an error.
>>>>
>>>>This differs from the normal BSD-read, which blocks until *some* data is
>>>>available
>>>>and return that data (but nor more than size bytes).
>>>>
>>>>We found that more convenient to implement synchronous message-oriented
>>>>applications
>>>>and 'normal' Stream communication with a single and straight-forward API.
>>>
>>>
>>>Thanks for this feedback! You may have seen the mail
>>>exchange to that topic. If you are interested, could you
>>>give us your feedback to the proposal written down at:
>>>
>>> http://wiki.cct.lsu.edu/saga/space/SAGA+API/Messages
>>>
>>>Best regards,
>>>
>>> Andre.
>>>
>>>
>>>
>>>
>>>>Best regards,
>>>>Thomas
>>>>
>>>>
>>>>Andrei Hutanu wrote:
>>>>
>>>>
>>>>>Hi all,
>>>>>
>>>>>Here is my feedback regarding the Viz-LSU use case. In some cases
>>>>>these requirements might be outside the scope of SAGA, I would like to
>>>>>know if that is the case though ..
>>>>>
>>>>>*Block-based data transmission is not covered by the current API
>>>>>*Resource discovery is not covered by the current API
>>>>>*Job submission to multiple resources (co-scheduling) is not covered by
>>>>>the current API
>>>>>*Simple job submission seems to be covered by the API. Here is a list of
>>>>>things
>>>>>that are not covered (because outside the 80-20 rule?) and
>>>>>there doesn't seem to be a "generic" attribute in the JobDefinition
>>>>>class where these attributes
>>>>>could be specified if the underlying scheduler happens to support them.
>>>>>** Logical file requirement (the jobs needs to run on a machine where an
>>>>>instance of this particular logical file exists)
>>>>>** Graphics requirements
>>>>>** Networking requirements (network interface, bandwidth to ..)
>>>>>** Performance-based descriptions : GFlops, memory bandwidth ..
>>>>>
>>>>>Andrei
>>>>>
>>>>>Thilo Kielmann wrote:
>>>>>
>>>>>
>>>>>
>>>>>>Dear all,
>>>>>>
>>>>>>the SAGA-RG has its use case document up for public comment, ending
>>>>>>oct 30.
>>>>>>
>>>>>>So far, there are 0 comments! :-(
>>>>>>
>>>>>>I hereby urge everybody to have a look and comment, even if the
>>>>>>comment is just
>>>>>>trivially in favour...
>>>>>>
>>>>>>
>>>>>>Thanks for your help,
>>>>>>
>>>>>>
>>>>>>Thilo
>>>>>>
------------------------------------------------------------
Dr. Thomas Eickermann
Zentralinstitut fuer Angewandte Mathematik
Forschungszentrum Juelich GmbH
D-52425 Juelich
Phone: +49 2461 61-6596
Email: Th.Eickermann at fz-juelich.de Fax: +49 2461 61-6656
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3760 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.ogf.org/pipermail/saga-rg/attachments/20051213/410331c6/attachment-0003.bin
More information about the saga-rg
mailing list