[DRMAA-WG] What information should JobInfo provide ?

[Daniel Gruber]

As you indicated, Peter, it is not as easy as it sounds to add all job template attributes
to the job info object. As mentioned before I think in most systems you can change the
submission attributes after submission (in Grid Engine we have qalter, JSV, job classes,
sge_request files, the -@ parameter, path aliasing) therefore we should be conservative and not 
specify that the submission parameters in job info needs to match 100% the job template.

Andre: You don’t need to keep the job template at this point in time anyhow. There is a 
get job template method specified on the job object itself.

When it comes to the Univa Grid Engine implementation, I’m storing the job template
in the library when a submission is done. When you leave the application and do a
re-connect the behavior is unspecified (meaning there is no job template available
anymore). While there is now a way in Univa Grid Engine 8.2 to get the information
out of qstat (you can see the qsub request line in qstat and qacct) I would not rely on
this to be available on all systems. Even when a 3rd party DRMAA2 implementation 
relay on parsing this out, this is highly error prone (as new parameters and syntax
changes can come in, which makes an implementation unusable). 

Also, we need to consider that the job can be either submitted from the drmaa2 app
(where we have the job template) and in a monitoring session the job can be 
from any source (where we don’t have the job template). Having here a bunch of 
attributes which finally are optional and not available makes the interface not 
attractive. The job info object is extensible hence each implementation can decided
what to add and what not as the last way out to get something like a job name
into job info.

Just one example to make it more clearer:

You can submit a job with an certain output path in job template but on 
the machine where it runs the output path could be translated in something
different, probably because of a different mount point. In UGE we have 
something called path aliasing, which is done by the system transparently
to the user. Now which output path would be correct?
When the job template output path is right, then the question 
is if it is catchable by all systems in a general way with job status calls?
When the real output path is right then the question is do we get that one
for all systems? I don’t think we can specify which one is the right on 
DRMAA level hence we would need to make all of them optional or implementation
specific. Hence the implementation specific approach seems more 
natural for me.

Due to the fact that we have already a method for returning the job template,
and in order to have small consistent set of attributes in job info, I 
would really not go the way to do such a major change.

Why I added this *job name* request in the job info object is because
the job name (rather then other attributes) is probably the most 
natural attribute of a job. In Univa Grid Engine you can do a lot of 
useful things with it: defining job dependencies (or multi-job dependencies),
using is for a grouped qstat etc. 

Short:
- I just want to add a job name in job info since it is one of the most
basic job info attributes
- I do not want to specify whether the job name matches the job name
of the job template or not. I would only specify that this is the job name
the system considers as the job name - and as a fall back to have it
available in all implementations it could be also just the job id. 
- I do not want to add all job template attributes to the job info object
since most of them we don’t need for filtering in real live  and due to
the fact that they would be optional anyhow. Hence implementation
specific extensions is the way to go IMHO.
- I can also live with the fact that job name is implementation specific
but I really see the job name very basic and really worthy for job 
filtering (especially when you want to group jobs like in UGE case).

Sorry the delayed answer

Cheers

Daniel

Am 14.09.2014 um 20:53 schrieb Peter Tröger <peter at troeger.eu>:

> I agree to Bruno and Andre that the advantages are worth it. Performance considerations should be done in the language bindings and / or the implementations, the root spec is the wrong place for that. I therefore vote for the simple addition all job templates attributes to the JobInfo structure.
> 
> An interesting side aspect are implementation-specific job template attributes. I would prefer to leave it to the implementors if they want to support them also in the JobInfo structure.
> 
> If Daniel agrees to all of this, especially from his implementation viewpoint in Grid Engine 8.2, then I would update the root spec and the C binding accordingly.
> 
> Best regards,
> Peter.
> 
> 
> Am 12.09.2014 um 22:18 schrieb Andre Merzky <andre at merzky.net>:
> 
>> I think the job template information are easy to add, and make app
>> writing easier (I don't need to keep the old submitted job template
>> around...).  Only drawback I see is increase in memory footprint when
>> used to report on many jobs -- but when used as a filtering
>> instruction, this should not matter.
>> 
>> 
>> My $0,02, Andre.
>> 
>> On Wed, Sep 10, 2014 at 7:48 PM, Peter Tröger <peter at troeger.eu> wrote:
>>> Dear all,
>>> 
>>> at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template:
>>> 
>>> http://redmine.ogf.org/issues/102
>>> 
>>> This leads to the situation that you cannot filter for it when doing monitoring.
>>> 
>>> Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application.
>>> 
>>> For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
>>> 
>>> Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
>>> 
>>> All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name.
>>> 
>>> What do you think ?
>>> 
>>> Best regards,
>>> Peter.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> drmaa-wg mailing list
>>> drmaa-wg at ogf.org
>>> https://www.ogf.org/mailman/listinfo/drmaa-wg
>> 
>> 
>> 
>> -- 
>> It was a sad and disappointing day when I discovered that my Universal
>> Remote Control did not, in fact, control the Universe.  (Not even
>> remotely.)
>