[DRMAA-WG] Treat job templates as value types ?
Daniel Templeton
daniel.templeton at oracle.com
Mon Mar 29 09:07:32 CDT 2010
The reason we have vendor extensions is that it's much cleaner than the
native specification, and it gives the DRMAA implementation the
opportunity to understand the meaning of the job template without having
to understand how to parse the native specification. In the SGE
implementation, the native specification is passed directly to SGE.
DRMAA knows nothing about what's in it. (Hence it's not sanity checked.)
I could live with the idea of making the job template a value type and
adding in C an undefined struct type field (e.g. struct
drmaa2_vendor_attributes_t) that vendors are allowed to define
themselves, with an empty struct being a valid definition.
How would this approach affect the other non-OO languages, like Perl?
Without thinking about it too hard, I think Perl would actually behaving
like an OO language in this case. Are there other non-OO languages we'd
need to worry about? I doubt it, but we need to be careful before
closing the door on any implementations.
I'm still chewing on whether it's really OK for the job template to
allow bogus values, expecting that the run() method will point out the
problems.
We had a discussion about allowing DRMAA to filter the list of jobs to
be returned from a search. If the job template were a value type, it
could be used as the filter. Fill in the fields that interest you, and
leave the rest empty. You'll get all the jobs that have those values
set in those fields. And the easiest answer would be not to allow
wildcarding. See the Jini discovery service for a complete
implementation of this approach.
Daniel
On 03/26/10 07:23, Mariusz Mamoński wrote:
> Hi,
>
> I have tried to collect all the problems, suggestions, and also my
> experience with different APIs and created a scratch page on my wiki:
> "How the JobTamplate could look if.. ") with several different
> possible options:
>
> http://fury.man.poznan.pl/~mmamonski/wiki/index.php/DRMAAv2/JobTemplate_-_Bindings
>
> Please review every of the 4 options and provide your comments! We can
> always combine some ideas.
>
> After writing this summary i asked one dummy question to myself: "What
> is the benefit of having vendor specific attributes if we already have
> nativeOptions (nativeSpecification)"?
>
> Cheers,
>
> On 24 March 2010 14:28, Andre Merzky<andre at merzky.net> wrote:
>> Quoting [Daniel Templeton] (Mar 24 2010):
>>>
>>> Andre,
>>>
>>> Don't worry about being biased. I'm unabashedly SGE-biased. :)
>>
>> :-)
>>
>>
>>> Unfortunately, though, what you proposed is what we're already
>>> discussing. It's the problem, not the solution. We're trying to have a
>>> consistent API across OO and non-OO languages. The issue goes away if
>>> we just add "except in C" to the description.
>>
>> Hmm, I thought the problem was:
>>
>>>>>>>> just only one concern: what if one of the DRM vendor wish to
>>>>>>>> provide some JobTemplate attributes additional to those
>>>>>>>> specified in DRMAA (as i remember SGE was an example)? I
>>>>>>>> expect having different struct definition (different
>>>>>>>> drmaa2.h ) leads to serious problems. I'm not saying no
>>>>>>>> (actually having a self allocated struct in C is more
>>>>>>>> convenient than using getter/setters - e.g. error handling),
>>>>>>>> but this, for me, requires addressing the extension
>>>>>>>> methodology on the IDL level.
>>
>> The SAGA scheme would still map to C in a very simple way
>>
>> // create job template
>> drmaa_job_template_t jt = drmaa_job_template_create ();
>>
>> // set a predefined DRMAA attribute
>> // these calls would always succeed, basically.
>> drmaa_job_template_set_attribute (jt, DRMAA_JOB_TEMPLATE_EXECUTABLE, "/bin/date");
>> drmaa_job_template_set_attribute (jt, "Executable", "/bin/date");
>> drmaa_job_template_set_executable (jt, "/bin/date");
>>
>> // set a vendor specific attribute
>> // this call would return an error if the attribute is not
>> // supported
>> drmaa_job_template_set_attribute (jt, "SGE:Executable-Type", "UniversalBinary");
>>
>> // free job template
>> drmaa_job_template_destroy (jt);
>>
>> So, no special rule for C, and the same drmaa2.h for all vendors.
>>
>> Well, the special rule for C is to have a Create and Destrpy method
>> - but you have that for all classes/types, job template is not
>> special in that respect...
>>
>> Cheers, Andre.
>>
>>
>>> Andre does bring up a valid point, though, which is that we really could
>>> just special-case the non-OO languages, just like we did in v1. Or,
>>> like I said before, we could leave the template management to the
>>> implementation, scoped to the session, with an overflow value of being
>>> allowed to free templates as needed.
>>>
>>> Daniel
>>>
>>> On 03/24/10 06:03, Andre Merzky wrote:
>>>> my biased point of view: use the saga model:
>>>>
>>>> {
>>>> // create job template on the stack
>>>> saga::job::description jd;
>>>>
>>>> // 3 equivalent ways for predefined attribute
>>>> jd.set_attribute (saga::job::description_executable, "/bin/date");
>>>> jd.set_attribute ("Executable", "/bin/date");
>>>> jd.set_executable ("/bin/date");
>>>>
>>>> // vendor-defined attributes:
>>>> jd.set_attribute ("SGE:Executable-Type", "UniversalBinary");
>>>>
>>>> // job template gets deleted from the stack when leaving the scope
>>>> }
>>>>
>>>> Memory management of the hash table is done internally. The
>>>> implementation also needs to maintain a list of predefined
>>>> attributes - but that is anyway the case I think. C language
>>>> bindings could of course still fall back to the explicit
>>>> create/delete methods.
>>>>
>>>> Again, sorry for being biased...
>>>>
>>>> Best, Andre.
>>>>
>>>>
>>>> Quoting [Daniel Templeton] (Mar 23 2010):
>>>>>
>>>>> It's not a bad idea. I would really like to find a way to make it work,
>>>>> because it would make life quite a bit simpler. Is there a way to make
>>>>> it work?
>>>>>
>>>>> Daniel
>>>>>
>>>>> On 03/23/10 13:56, Peter Tröger wrote:
>>>>>> Ok, it was a bad idea. Thanks for commenting.
>>>>>>
>>>>>> /Peter.
>>>>>>
>>>>>> Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
>>>>>>
>>>>>>> And that may be the original reason why the SGE C binding uses that
>>>>>>> pseudo-OO structure. It is more or less a hash map, allowing
>>>>>>> arbitrary
>>>>>>> name-value pairs to be stored, facilitating vendor-defined job
>>>>>>> template
>>>>>>> attributes. If we can't find a way around that issue, it's a
>>>>>>> show-stopper. We have to allow for vendor-defined attributes.
>>>>>>>
>>>>>>> Daniel
>>>>>>>
>>>>>>> On 03/22/10 07:29, Mariusz Mamo??ski wrote:
>>>>>>>> Hi Peter, all,
>>>>>>>>
>>>>>>>> 2010/3/22 Peter Tröger<peter at troeger.eu>:
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> we have an ongoing discussion about the possible removal of
>>>>>>>>> "createJobTemplate" and "deleteJobTemplate". The last proposal was
>>>>>>>>> to move this functions to the language bindings that need it. This
>>>>>>>>> is currently only C - all other languages have their own way of
>>>>>>>>> performing instantiation and termination explicitly.
>>>>>>>>>
>>>>>>>>> After some more thinking, I think I got the true underlying issue.
>>>>>>>>> So far, we are treating job templates as instances with state and
>>>>>>>>> behavior - objects, in most languages. The only reason for doing
>>>>>>>>> this (so far) is the ability to throw errors on attribute access,
>>>>>>>>> since we need that for DRMAA's understanding of optional job
>>>>>>>>> template attributes. However, from all other perspectives, job
>>>>>>>>> templates are just value types. If you got one, you fill it, and
>>>>>>>>> then you pass the thing as a whole. In a RPC scenario, you would
>>>>>>>>> also expect to send filled job templates as a whole, similar to a
>>>>>>>>> filled JSDL document.
>>>>>>>>>
>>>>>>>>> So if we change that, what would that mean:
>>>>>>>>>
>>>>>>>>> C: Generic getter / setter functions for job template attributes
>>>>>>>>> would go away. Instead, you would create / delete JobTemplate
>>>>>>>>> structs directly. runJob() would take a pointer to this struct.
>>>>>>>>> Explicit removal is no longer needed, since the stack is cleared
>>>>>>>>> automatically.
>>>>>>>>>
>>>>>>>>> C# : JobTemplate class would become JobTemplate struct.
>>>>>>>>>
>>>>>>>>> Java / Python: JobTemplate class remains JobTemplate class, since
>>>>>>>>> they have no struct concept. Creation and destruction can be
>>>>>>>>> managed with OO mechanisms.
>>>>>>>>>
>>>>>>>>> Another effect would be a change in the semantics of optional
>>>>>>>>> attributes. We already demand the syntactical inclusion of
>>>>>>>>> optional attribute names in the class definition. In C, the usage
>>>>>>>>> of a non-implemented optional attribute names gives a specialized
>>>>>>>>> error. With the change, we would have struct members that you are
>>>>>>>>> not allowed to fill in with some DRMAA libraries. But this would
>>>>>>>>> be detected only at submission time, since struct changes do not
>>>>>>>>> involve library code.
>>>>>>>>>
>>>>>>>> just only one concern: what if one of the DRM vendor wish to provide
>>>>>>>> some JobTemplate attributes additional to those specified in DRMAA
>>>>>>>> (as
>>>>>>>> i remember SGE was an example)? I expect having different struct
>>>>>>>> definition (different drmaa2.h ) leads to serious problems. I'm not
>>>>>>>> saying no (actually having a self allocated struct in C is more
>>>>>>>> convenient than using getter/setters - e.g. error handling), but
>>>>>>>> this,
>>>>>>>> for me, requires addressing the extension methodology on the IDL
>>>>>>>> level.
>>>>>>>>
>>>>>>>>> And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at
>>>>>>>>> all then, regardless of the language.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Peter.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> drmaa-wg mailing list
>>>>>>>>> drmaa-wg at ogf.org
>>>>>>>>> http://www.ogf.org/mailman/listinfo/drmaa-wg
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>> --
>>>>>>> drmaa-wg mailing list
>>>>>>> drmaa-wg at ogf.org
>>>>>>> http://www.ogf.org/mailman/listinfo/drmaa-wg
>>>>>>
>>>>>> --
>>>>>> drmaa-wg mailing list
>>>>>> drmaa-wg at ogf.org
>>>>>> http://www.ogf.org/mailman/listinfo/drmaa-wg
>>>
>>
>>
>>
>> --
>> Nothing is ever easy.
>> --
>> drmaa-wg mailing list
>> drmaa-wg at ogf.org
>> http://www.ogf.org/mailman/listinfo/drmaa-wg
>>
>
>
>
More information about the drmaa-wg
mailing list