[saga-rg] SAGA strawman API: comments to Stream functions
Andre Merzky
andre at merzky.net
Tue Dec 13 11:25:49 CST 2005
:-D
Thanks, Andre.
Quoting [Thomas Eickermann] (Dec 13 2005):
>
> Hi Andre,
>
> sorry for responding so slowly, but times are tough as usual in
> December ...
>
> I finally surrender to your arguments. I was not aware that SAGA
> will provide callbacks (via the Monitoring interface ?). Using such
> callbacks, you can probably solve the problem that i have raised
> in most cases. The rest would fall under the 80/20 rule.
>
> Thanks for the discussion,
> Thomas
>
>
> Andre Merzky wrote:
> >Hi Thomas,
> >
> >
> >Quoting [Thomas Eickermann] (Dec 02 2005):
> >
> >>Hi Andre,
> >>
> >>thanks for your answer. I agree with your comments that access
> >>to raw descriptors is somewhat problematic.
> >>Nevertheless i regard them as quite important for implementing
> >>real-life interactive applications. Two examples:
> >>
> >>1. AVS/Express allows to trigger a module if an event occurs at a
> >>descriptor. I do not know of any other way to implement stream
> >>communication within AVS/Express and i imagine, that the situation
> >>is quite similar for other tools like OpenExplorer.
> >>
> >>2. If you use some GUI toolkit (e.g. Qt), you can also register callbacks
> >>for events on descriptors. If such a descriptor is not available, the
> >>only thing you can probably do is to use timers and polling. There are
> >>obviously good reasons to avoid that, if possible.
> >
> >
> >I agree about the importance.
> >
> >
> >
> >>See further comments inline.
> >>
> >>Best regards,
> >>Thomas
> >>
> >>
> >>Andre Merzky wrote:
> >>
> >>>Hi Thomas,
> >>>
> >>>sorry for the late answer... *sigh*
> >>>
> >>>
> >>>Quoting [Thomas Eickermann] (Nov 10 2005):
> >>>
> >>>
> >>>>* in both use-cases there is an interactive visualisation/steering
> >>>>application sitting at one end of the stream. Such applications
> >>>>typically use some kind of GUI-toolkit with operates in an
> >>>>event-callback mode or similar, where callbacks can be registered
> >>>>for events occuring at socket/pipe/...-descriptors. On the other hand,
> >>>>in
> >>>>many implementations, a Stream will somehow use a socket, pipe or
> >>>>something else with a descriptor. Therefore we need a function that
> >>>>returns
> >>>>the descriptor associated to a stream:
> >>>> fd = descriptor(Stream);
> >>>
> >>>
> >>>This is complicated, as we do not know _if_ there will be a
> >>>native descriptor available, and if operations on that will
> >>>interfere with the implementation.
> >>>
> >>
> >>availability:
> >>right, i was somewhat unprecise. What i meant was the native
> >>respresentation of a socket in the given language (a descriptor
> >>in C, an IO::Socket obejct in Perl and probably there is something
> >>like that in most languages.
> >
> >
> >I was unprecise as well I guess: you are right, there'll be
> >a native representation in most languages, not doubt.
> >
> >What I mean with availability is best shown in Globus-XIO:
> >I simply do not know _how_ I could obtain the native
> >descriptor from a globus-XIO based implementation: there is
> >no globus-xio call exposing the descriptor.
> >
> >
> >
> >>>E.g. the implementation might be based on Globus-XIO or on
> >>>Reliable UDP. For the first I do not know if a descriptor
> >>>is availbale via the API, for the second there are two
> >>>descriptors (one tcp, one udp) the application would need to
> >>>handle.
> >>
> >>To handle the second case, 'descriptor' would have to return
> >>an array of descriptors. I don't see that in the first case
> >>an implementation could return something better than a NULL-array.
> >
> >
> >It is dependend on the middleware then: the application
> >needs to be aware of the fact that saga implementation
> >returns a single fd, but saga implementation returns 2 fd's,
> >with specific handling requirements.
> >
> >I am not saying that this is impossible. Just would like to
> >be sure that this is the best way to go...
> >
> >
> >
> >>>Also, the implementation might transparently add a protocol
> >>>(say checksumming), and direct tempering with the desfriptor
> >>>might very well break that protocol.
> >>
> >>I agree, the only thing the user is allowed to do with the
> >>descriptor, is to call 'select'. Read, write, seek, close
> >>would most likely break the protocol.
> >>
> >>However, this cannot be enforced, and i am aware that this
> >>conflicts the principle of hiding the implementation from
> >>the user (to minimize the risk that the user breaks something).
> >
> >
> >I checked with QT, and I think there is a way to handle
> >SAGA events in QT:
> >
> >------------------------------------------------------------
> > // this be a callback for a saga stream object, which gets
> > // fired on new data arriving etc.
> > void my_saga_file::saga_cb (void)
> > {
> > emit update_gui ();
> > }
> >
> > my_gui g;
> > my_saga s;
> >
> > QObject::connect (&s, SIGNAL (update_gui (void)),
> > &g, SLOT (saga_cb (void)));
> >------------------------------------------------------------
> >(The code above is probably not really correct, but you get
> > the image...)
> >
> >my_saga::saga_cb would need to registerd to saga as callback
> >for events on a stream. Then, if such an event occurs, the
> >callback would trigger (emit) a QT event (update_gui) which
> >would be received by your gui class, and cause, well, the
> >gui to update etc.
> >
> >The QT event mechanism would allow to use any data available
> >in the callback (e.g. string read from the stream) to be
> >signalled to the gui loop as well.
> >
> >So, would that fullfill your requirements?
> >
> >Am am not so sure about AVS and other packages. I did not
> >work with AVS for quite some time, but IIRC, the events have
> >been coupled much tighter to selects on descriptors, with no
> >alternative rounds? However, for all other packages I know
> >(not many), a trivial (e.g. 3 line) translation from saga
> >callbacks to 'native' events seems to exist.
> >
> >
> >Cheers, Andre.
> >
> >
> >
> >>>So, we might be able to add the API function, but it is
> >>>difficult to define what it should return in the cases
> >>>outlined above...
> >>>
> >>>
> >>>
> >>>
> >>>>However, users have to be aware, that several Streams
> >>>>may share the same decriptor (e.g. in VISIT, if several
> >>>>Streams are tunneled through a single connection of a
> >>>>different protocol like ssh).
> >>>
> >>>
> >>>
> >>>>* a general remark on Streams: i like the idea of keeping the
> >>>>Stream-API close
> >>>>to the BSD socket API. However, as Andrei mentioned, we also often have
> >>>>to deal with more message- or block-oriented communication patterns.
> >>>>In the internal API the we use in the VISIT toolkit, we have therefore
> >>>>added a timeout parameter to all functions and slightly changed the
> >>>>semantics.
> >>>>For example
> >>>>
> >>>>nread = read(stream, buffer, size, timeout);
> >>>>
> >>>>works as: try to read size bytes from stream, return if either:
> >>>>- size bytes have been read, or
> >>>>- timeout seconds have passed (wait forever, if timeout < 0), or
> >>>>- an error has occured
> >>>>the function returns the number of bytes read or -1 in case of an error.
> >>>>
> >>>>This differs from the normal BSD-read, which blocks until *some* data
> >>>>is available
> >>>>and return that data (but nor more than size bytes).
> >>>>
> >>>>We found that more convenient to implement synchronous message-oriented
> >>>>applications
> >>>>and 'normal' Stream communication with a single and straight-forward
> >>>>API.
> >>>
> >>>
> >>>Thanks for this feedback! You may have seen the mail
> >>>exchange to that topic. If you are interested, could you
> >>>give us your feedback to the proposal written down at:
> >>>
> >>>http://wiki.cct.lsu.edu/saga/space/SAGA+API/Messages
> >>>
> >>>Best regards,
> >>>
> >>>Andre.
> >>>
> >>>
> >>>
> >>>
> >>>>Best regards,
> >>>>Thomas
> >>>>
> >>>>
> >>>>Andrei Hutanu wrote:
> >>>>
> >>>>
> >>>>>Hi all,
> >>>>>
> >>>>>Here is my feedback regarding the Viz-LSU use case. In some cases
> >>>>>these requirements might be outside the scope of SAGA, I would like to
> >>>>>know if that is the case though ..
> >>>>>
> >>>>>*Block-based data transmission is not covered by the current API
> >>>>>*Resource discovery is not covered by the current API
> >>>>>*Job submission to multiple resources (co-scheduling) is not covered
> >>>>>by the current API
> >>>>>*Simple job submission seems to be covered by the API. Here is a list
> >>>>>of things
> >>>>>that are not covered (because outside the 80-20 rule?) and
> >>>>>there doesn't seem to be a "generic" attribute in the JobDefinition
> >>>>>class where these attributes
> >>>>>could be specified if the underlying scheduler happens to support them.
> >>>>>** Logical file requirement (the jobs needs to run on a machine where
> >>>>>an instance of this particular logical file exists)
> >>>>>** Graphics requirements
> >>>>>** Networking requirements (network interface, bandwidth to ..)
> >>>>>** Performance-based descriptions : GFlops, memory bandwidth ..
> >>>>>
> >>>>>Andrei
> >>>>>
> >>>>>Thilo Kielmann wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Dear all,
> >>>>>>
> >>>>>>the SAGA-RG has its use case document up for public comment, ending
> >>>>>>oct 30.
> >>>>>>
> >>>>>>So far, there are 0 comments! :-(
> >>>>>>
> >>>>>>I hereby urge everybody to have a look and comment, even if the
> >>>>>>comment is just
> >>>>>>trivially in favour...
> >>>>>>
> >>>>>>
> >>>>>>Thanks for your help,
> >>>>>>
> >>>>>>
> >>>>>>Thilo
> >>>>>>
>
> ------------------------------------------------------------
> Dr. Thomas Eickermann
> Zentralinstitut fuer Angewandte Mathematik
> Forschungszentrum Juelich GmbH
> D-52425 Juelich
> Phone: +49 2461 61-6596
> Email: Th.Eickermann at fz-juelich.de Fax: +49 2461 61-6656
>
--
+-----------------------------------------------------------------+
| Andre Merzky | phon: +31 - 20 - 598 - 7759 |
| Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 |
| Dept. of Computer Science | mail: merzky at cs.vu.nl |
| De Boelelaan 1083a | www: http://www.merzky.net |
| 1081 HV Amsterdam, Netherlands | |
+-----------------------------------------------------------------+
More information about the saga-rg
mailing list