[drmaa-wg] C Binding

[Rajic, Hrabri]

I concur with the attitude to use the GridForge tools to our advantage
and record all the suggestions that will improve the spec and also
implementation usability.
What is less obvious is the way we could propagate some of the needed
changes to the spec in a way that would be acceptable to the GFSG and
all the interested parties.   We had few discussions about this in the
past and we will continue having them till we find the best way to
handle the issue.
It looks like a good agenda item for the next DRMAA call.
I would not mind if there is an alternative C bindings spec that has all
deemed necessary/useful stuff in it, a GFD-E doc, but would also like to
advocate having a DRMM v1.0 conformant C bindings spec and the spec
conforming implementation as well.  

Wasn't I already accused of being quite successful in finding ways that
need more work:-).

    -Hrabri
 

-----Original Message-----
From: owner-drmaa-wg at ggf.org [mailto:owner-drmaa-wg at ggf.org] On Behalf
Of Andreas Haas
Sent: Thursday, January 13, 2005 6:58 AM
To: DRMAA Working Group
Subject: Re: [drmaa-wg] C Binding

On Thu, 13 Jan 2005, Daniel Templeton wrote:

> > In a previous e-mail, Daniel Templeton wrote:
> >> I've started to work on the C binding doc.
> >> There's something I've been dying to do since I started with DRMAA,
> >> and now that I have my hands on the C spec, I think it's a good
> >> time to do it.
> >
> > Alarming attitude.
>
> That's me.  I'm a wild man.  I also rip the tags off of matresses.
>
> > It is my understanding that the C spec is being revised with the
> > intent of making it a stand-alone document.   Although I would
> > expect this to entail vast changes to the document, the technical
> > content should be unchanged.

I fully second Dan's attitude with that respect. Our whole work
wouldn't be worth the effort if we ignore the overall objective of
adoption. Thus it is even our duty to incorporate changes to language
specs that have proven to hinder adaption. In case technical changes
are required to pursue this it can be handled trough the GFD-E
document we have to deliver anyways.

> > This will make revision comparisons very difficult.

This is true. But luckily GGF provides Gridforge issue tracker for
handling that complexity in a way transparent for anyone. We got
Gridforge access at no charge and it has turned out to be a powerful
and flexible tool.

> >
> > Please continue to propose changes during your editing cycle
> > so they can be discussed and potentially incorporated into
> > the spec after the changes to make it a stand-alone ducument.
>
> A very reasonable suggestion.  Sure.

Absolutely. There is already a number of doc change requests in
DRMAA Gridforge database that should be revised due to their
relevance for the C binding document.

One of the items that certainly is of relevance is the lack of
diagnosis information in case drmaa_wifaborted() indicates a job
failure

   http://forge.gridforum.org/tracker/?aid=309

isn't it fair when integrators are asking to have an interface
allowing them to forward diagnosis information known by the DRM
to their end user apps? Those who are in contact with customers in
their everydays job know providing programmatic access to error
diagnosis is key with reducing the number of cases where the end
users of those apps have to bother their administrators.

>
> >> That is namely to fix the drmaa_attr_*_t structures.
> >> They are currently unusable.  In order to do anything useful
> >> with them, we need either a way to get the count of the elements
> >> in the structure or to reset the cursor to the beginning of the
> >> list or both.
> >>
> >>
> >> Here's are the 6 new functions I propose:
> >>
> >> int drmaa_get_num_attr_names(drmaa_attr_names_t* values, int*
count);
> >> int drmaa_get_num_attr_values(drmaa_attr_values_t* values, int*
count);
> >> int drmaa_get_num_attr_ids(drmaa_attr_ids_t* values, int* count);
> >> int drmaa_reset_attr_names(drmaa_attr_names_t* values);
> >> int drmaa_reset_attr_values(drmaa_attr_values_t* values);
> >> int drmaa_reset_attr_ids(drmaa_attr_ids_t* values);
> >>
> >> I strongly recommend we at least add one set or the other.  My
> >> preference would be both, but I think the first set is the more
> >> important.
> >
> > The drmaa_get_next_* routines provide a mechanism to obtain
> > every element in the data structure (once).
> > I would expect many, if not most, callers of these routines
> > to insert the elements into a data structure which is suitable
> > for their usage.
> > I am of the opinion that the existing routines are sufficient
> > for incrementally constructed data structures.
> > Although, if the caller is building something like an array,
> > there is likely to be a resizing and/or memory copying overhead.
> >
> >> Does anyone have anything against adding these functions?
> >
> > I don't think the convenience of the proposed functions
> > is sufficient to require their implementation.
> >
> > The agreed upon interface for the drmaa_get_next_* routines
> > accept a buffer for the primary output value.
> >  For example:
> >    int drmaa_get_next_attr_name(drmaa_attr_names_t* values,
> >                                 char *value, size_t value_len);
> > Thus, the drmaa implementation is not required to retain
> > information once it has been provided to the application.
> >
> > If it is important to iterate over values more than once,
> > then presumably the DRMAA implementation should retain the
> > entire set of values.  I'd prefer to let the application
> > decide when to retain values.
>
> So, you're belief is that an implementation may only fetch values as
> required so it doesn't have to store them all in memory?  I can see
> where that would be useful, but I don't see where that applies to
DRMAA.
>   The things you get back as one of these structs are the job ids from
a
> drmaa_run_bulk_jobs() call, attribute names from
> drmaa_get_attribute_names(), and values from
> drmaa_get_vector_attribute_value().  In all of these cases, the
> implementation has to store the full list of values internally anyway,
> unless it writes everything into a database so that it can read it out
> incrementally, and even then it still at some point had to have the
> whole thing in memory so it could write it out in the first place
> (except maybe the job ids...).  To me, this is a purely academic
> argument against a practical consideration.
>
> I am in the peculiar position of both developing an implementation and
> using the implementation in applications.  I have now written two
"real"
> DRMAA apps, and both times I ran into a problem with not knowing how
> many elements were being returned to me.  In the first app, I cheated
> and loaded the header that defined the structs so I could do the math
> myself.  In the second app, I just declared a really big array, which
is
> both a memory waste and dangerous.  The correct answer would be to use
a
> char** and re-malloc it if I get too many results, but that particular
> idiom is very error prone.  I don't see why a developer should be
forced
> to do dangerous and redundant things if it's avoidable.  Besides,
> knowing how easy it would be to for the implementation to return the
> number of elements, it just seems sadistic to make developers jump
> through hoops for what should a simple operation.  C is complicated
> enough without adding thorny interfaces.
>
> To be fair, the number of ids returned from drmaa_run_bulk_jobs() can
be
> calculated from start, end, and incr, but I see it as needlessly
> dangerous to make the developer do math to guess what the API is going
> to do, especially when that math is related to memory allocation.  The
> less thinking the developer has to do, the fewer array overflows he's
> going to have.
>
> This would all be different if we were talking about Java or C++ where
> there's a linked list implementation already there, just waiting to be
> used, but we're not.  This is C, and when you're developing with
sticks
> and stones, even the slightest gimmee can mean a great deal.
Remember,
> adoption is important to us.

I second this. Writing DRMAA apps without this is cumbersome even though
it is no impossible.

Regards,
Andreas