[SAGA-RG] Fwd (andre at merzky.net): Re: JSDL - SAGA
Pascal Kleijer
k-pasukaru at ap.jp.nec.com
Mon Feb 19 18:13:20 CST 2007
Hi all,
I first sought about a solution like Hartmut, but SAGA is targeting Grid
newbies, so JSDL might be out of the 80/20 rule scope.
I am quit happy with the current set the JD provides; now would I use a
Job Submission Framework (JSF) for NAREGI, the current SAGA won't work
perfectly. NAREGI has extended the scope to support MPI jobs and I need
to support that. I can always stick to my SAGA-style implementation for
the moment.
To allow power users to still make use of the SAGA adding a simple JSDL
property that over rules all others and is passed directly through to
the back-end would be fine. The API documentation should clearly tell
that if the JSDL tag is used it supersedes all other settings.
--
Best regards,
Pascal Kleijer
----------------------------------------------------------------
HPC Marketing Promotion Division, NEC Corporation
1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan.
Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382
Andre Merzky wrote:
> Ha, actually, that is what I expected this discussion would
> lead to :-P
>
> You write:
>> - JSDL is an OGF Standard now and there is no way of ignoring it in SAGA
>
> and Pascal:
>> The necessity to support JSDL 1.0.x is a necessity, since
>> it is an OFG standard.
>
> I think we need to step back a little. Yes, JSDL is an OGF
> standard - but so is OGSA, OGSA Security, OGSA-DAIS.
>
> Yes, we should support JSDL, and in particular be
> implementable on top of JSDL. We also should support OGSA,
> and in particular be implementable on top of OGSA. Same
> with DAIS.
>
> That does NOT mean that we need an API to write JSDL
> documents, nor an API to write OGSA Base NOtification
> documents, nor an API to write DAIS Collection Access XML
> docs.
>
> SAGA is NOT a general purpose API for existing OGF
> standards. It is _designed_ to be incomplete.
>
>
> Quoting [Hartmut Kaiser] (Feb 19 2007):
>> Hi all,
>>
>> After thinking about this issue a bit more I would like to make a even more
>> 'radical' suggestion.
>>
>> Let's recap:
>> - JSDL is an OGF Standard now and there is no way of ignoring it in SAGA
>> - the current SAGA JD is more constrained in terms of scope if compared to
>> JSDL
>> - there will be a lot of middleware supporting JSDL in the future
>> - there wll be a lot of tooling support to create JSDL in the future
>>
>> A viable solution to this in SAGA could be (duck):
>>
>> - redo the SAGA job_service and job interfaces so that these work solely
>> based on JSDL strings
>
> Further, the saga job description has a (small) number of
> keys which are not supported by JSDL, and will not be in the
> near future. Those are scheduling related: Queue,
> JobStartTime etc.
>
>
>> - supply a SAGA JD component providing a _simple_ interface for JSDL
>> construction:
>
> The JSDL schema has repeatable, hierarchical elements, with
> 0..n occurences of sections etc etc. It can include XML
> snippets from arbitrary name spaces. Just one example from
> many in the JSDL spec:
>
> "7.1 Attribute Extension
>
> Every JSDL element allows for additional attributes, as
> many as needed, provided these attributes have a namespace
> other than the normative namespaces defined by JSDL."
>
> There is a good API to write XML documents (DOM), and there
> are many tools to do so. Why should SAGA re-do that? None
> of these APIs/tools is simple - they cannot be, by design.
>
> Please check the JSDL spec: JSDL is simple for middleware
> level - but it is too complex (and I mean really) on the
> level SAGA is targeting at.
>
> So, providing an API to create JSDL is, IMHO, out of scope.
>
>
>> job_description::construct(string jsdl);
>> job_description::as_jsdl();
>>
>> job_service::create_job(string jsdl);
>> job::migrate(..., string jsdl, ...);
>>
>> where we clearly could stick to JSDL V1 for this conversion and the allowed
>> attribute set on our job descriptions.
>
> I honestly don't see the reason why we should deviate from
> our design principles (simple API, orient on use cases,
> don't do more than users ask for) just because JSDL exists.
>
> Its nice it exists, we support it (it can and should be used
> by SAGA _implementations_). But it has no more appeal on
> API level than has OGSA Base Notification, for example.
>
> So, my proposal is:
>
> - stick to the JD we have in SAGA (it works nicely with
> JSDL)
>
> - support upcoming JSDL extensions in due time, either by
> new keys on the job description (JSDL scheduling), or
> programmatically (i.e. JSDL parameter sweeps)
>
> - if there are use cases asking for it (only then),
> provide a jsdl_to_jd converter
>
> Well, I realise I am repeating myself - not a good sign. We
> probably should give the topic anoter week of rants, and
> then do a straw poll on the mailing list, or continue to
> fight on a phone call... :)
>
> Cheers, Andre.
>
>
>> Regards Hartmut
>>
>>> -----Original Message-----
>>> From: saga-rg-bounces at ogf.org
>>> [mailto:saga-rg-bounces at ogf.org] On Behalf Of Pascal Kleijer
>>> Sent: Sunday, February 18, 2007 8:51 PM
>>> To: SAGA RG
>>> Subject: Re: [SAGA-RG] Fwd (andre at merzky.net): Re: JSDL - SAGA
>>>
>>> Dear all,
>>>
>>> I have given me the week-end to get some reflection on the
>>> problem. As mentioned it is not a trivial issue. If things
>>> goes out of hand, we might as well rename the project CAGA of
>>> not go gaga :P
>>>
>>> The necessity to support JSDL 1.0.x is a necessity, since it
>>> is an OFG standard. I do not think JSDL 1.0 is overly complex
>>> right now. The Job Description should be made such that
>>> passing from one form to another is automatic (JD -> JSDL).
>>> The support of a property key for JSDL isn't bad; if used,
>>> all other properties can be potentially ignored. The JD would
>>> not have to parse it. The current specification is simple
>>> enough to not violate the "S" rule of SAGA.
>>>
>>> The problem we have is with all the up coming extensions. In
>>> my case, I clearly violated the SAGA API rules in order to
>>> make the system run with NAREGI. I tweaked the problem to
>>> solve my case. In the 80/20 rule, it probably means that the
>>> SAGA job submission won't be usable. I will have to write
>>> code that does not deal with SAGA and goes directly to the
>>> underwear. Now I am speaking as a Grid "expert" and not as a
>>> Grid newbie (which is the SAGA user base). If you want to use
>>> complex Grid programming and do complex things, SAGA might
>>> not be the right solution.
>>>
>>> The proposal made by Dan is not elegant in my opinion because
>>> it will force developer to know the underlying technology
>>> they want to use; it will also clutter the JD. The proposal
>>> from Thilo might work for OO languages because you can hide a
>>> lot behind patterns, but it won't mean a specific
>>> implementation will support such scheme.
>>>
>>> The passing through solution seems to be mildly seen by the
>>> some GAT people here. I would say no, if the error handling
>>> is explicit enough to return a message telling that this JSDL
>>> cannot be interpreted by the underlying system. I will not
>>> see this as a bad solution either.
>>>
>>> --
>>>
>>> Best regards,
>>> Pascal Kleijer
>>>
>>> ----------------------------------------------------------------
>>> HPC Marketing Promotion Division, NEC Corporation
>>> 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan.
>>> Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382
>>>
>>>
>>> Daniel S. Katz wrote:
>>>> Another related idea is for SAGA to have one class for job
>>>> descriptions with a quite loose ability to add "hints",
>>> such as in MPI
>>>> I/O. The hints would not be required and might not be used, but in
>>>> some cases would be used, depending on what SAGA is sitting
>>> on top of.
>>>> Dan
>>>>
>>>> Thilo Kielmann wrote (on 2/16/07 5:54 AM):
>>>>> Folks,
>>>>>
>>>>> I think we are having conflicting goals here.
>>>>> (Technical goals, not personal ones ;-)
>>>>>
>>>>> On one hand, we have the "S for simplicity" in SAGA, and
>>> we must keep it.
>>>>> On the other hand, we have the necessity to support JSDL,
>>> future JSDL
>>>>> extensions, or any other types of job decscriptions that
>>> people want to use.
>>>>> (And still might to be invented.)
>>>>>
>>>>> Assume, JSDL++ (whatever it will look like in near future) will
>>>>> become a widely adopted standard. (Or anything else,
>>> doesn't matter
>>>>> in the following.) Then, SAGA implementations will have to use
>>>>> JSDL++, and to form JSDL++ from SAGA job descriptions.
>>>>> Simultaneously, users are likely to use JSDL++ themselves,
>>> and may wnat to use JSDL++ to express their resource needs.
>>>>> At this point in time, SAGA will sit in the middle, and it may be
>>>>> very clumsy to first translate from JSDL++ to a SAGA job
>>> description,
>>>>> and then back to
>>>>> JSDL++ somewhere "down under" in the implementation.
>>>>>
>>>>> For the very purpose of SAGA as a universal and simple
>>> grid API, it has to:
>>>>> - be independent of job description standards (mostly simpler than
>>>>> them)
>>>>> - support job description standards
>>>>>
>>>>>
>>>>> My suggestion:
>>>>>
>>>>> SAGA should have one class of job descriptions, and the
>>> possibility
>>>>> to create subclasses for more specific job descriptions
>>> (like JSDL).
>>>>> Such subclasses could be defined as separate extension documents
>>>>> (just like gridcpr). (Or was it "resource descriptions"???)
>>>>>
>>>>> With this approach, users could still write programs that are
>>>>> independent of the underlying job submission machinery, having a
>>>>> simplified view on jobs and resource attributes etc. etc.
>>>>> At the same time, subclassed job/resource descriptions could be
>>>>> "passed through" transparently from the API to the
>>> implementeation,
>>>>> without being converted back and forth, a process that is
>>> very likely
>>>>> to loose some important details.
>>>>>
>>>>>
>>>>> Is this a route to go?
>>>>>
>>>>>
>>>>> Thilo
>>>>>
>>>>> On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
>>>>>
>>>>>> From: 'Andre Merzky' <andre at merzky.net> <mailto:andre at merzky.net>
>>>>>> To: SAGA RG <saga-rg at ogf.org> <mailto:saga-rg at ogf.org>
>>>>>> Subject: [SAGA-RG] Fwd (andre at merzky.net
>>> <mailto:andre at merzky.net>):
>>>>>> Re: JSDL - SAGA
>>>>>>
>>>>>> Hi group(s),
>>>>>>
>>>>>> a couple of us had a recent discussion (f2f and email) about JSDL
>>>>>> and SAGA. The question is: should we support JSDL fully on API
>>>>>> level? E.g., should we allow the application programmer to
>>>>>> specify/use JSDL documents for job creation?
>>>>>>
>>>>>> The reasons for doing that are compelling: JSDL is one of
>>> the most
>>>>>> acknowledged standards in OGF, and the number of backends
>>> supporting
>>>>>> JSDL is rapidly increasing it seems.
>>>>>> Supporting JSDL directly would allow to interface with
>>> other tools
>>>>>> using JSDL, and would allow to reuse JSDL documents where
>>> these are
>>>>>> already available.
>>>>>>
>>>>>> Well, I have however some problems with that approach, which are
>>>>>> outlined in the cited email below.
>>>>>>
>>>>>> Do you guys have any other thoughts, and what solution would you
>>>>>> prefer?
>>>>>>
>>>>>> Cheers, Andre.
>>>>>>
>>>>>>
>>>>>> ----- Forwarded message from 'Andre Merzky' <andre at merzky.net>
>>>>>> <mailto:andre at merzky.net> -----
>>>>>>
>>>>>>
>>>>>>> From: 'Andre Merzky' <andre at merzky.net> <mailto:andre at merzky.net>
>>>>>>> To: Shantenu Jha <sjha at cct.lsu.edu> <mailto:sjha at cct.lsu.edu>
>>>>>>> Cc: Hartmut Kaiser <hkaiser at cct.lsu.edu>
>>> <mailto:hkaiser at cct.lsu.edu>,
>>>>>>> 'Thilo Kielmann' <kielmann at cs.vu.nl> <mailto:kielmann at cs.vu.nl>,
>>>>>>> 'Andre Merzky' <andre at merzky.net> <mailto:andre at merzky.net>
>>>>>>> Subject: Re: JSDL - SAGA
>>>>>>>
>>>>>>> Hartmut and I discussed that somewhat last week. So he
>>> knows I am
>>>>>>> not wholehartedly for that option. SAGA is supposed to abstract
>>>>>>> the low level details, not to expose them.
>>>>>>>
>>>>>>> JSDL is going to define a number of extensions now.
>>> Some of these
>>>>>>> extensions are very useful for us, others not.
>>>>>>> Mostly, they will be more complex than JSDL itself.
>>>>>>>
>>>>>>> Are we going to support the extensions? Which? All/some?
>>>>>>> How to select? What error do we report on unsupported
>>> extensions?
>>>>>>> Do we mandate that extensions are supported by the backends?
>>>>>>> Which?
>>>>>>>
>>>>>>> Even w/o extensions: is the job description updated
>>> after an JSDL
>>>>>>> attrib is set? What about those attribs which are not
>>> JSDL keys?
>>>>>>> Assume an implementation which implements the existing SAGA job
>>>>>>> description keys: MUST it support complete JSDL now? What error
>>>>>>> whould it report?
>>>>>>>
>>>>>>> These are probably all solvable problems, and I do agree
>>> that there
>>>>>>> are advantages, i.e. the re-use of existing JSDL documents.
>>>>>>> Anyway, IMHO we should be careful, consider if we really have
>>>>>>> enough use cases etc. Also, a free function
>>>>>>> jsdl_to_job_description may do the trick, w/o
>>> complicating the job
>>>>>>> package itself.
>>>>>>>
>>>>>>> Cheers, Andre.
>>>>>>>
>>>>>>>
>>>>>> Another point I'd like to raise is: if HPC profile
>>> bekomes a widely
>>>>>> accepted OGF standard, do we support it directly, too? Or
>>>>>> OGSA-Workflow? Where to stop, and where is the 'S'
>>>>>> in that approach?
>>>>>>
>>>>>> Andre.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Quoting [Shantenu Jha]
>>>>>>>
>>>>>>>> What little I know, I think so to.
>>>>>>>>
>>>>>>>>
>>>>>>>> Quoting [Hartmut Kaiser]
>>>>>>>>
>>>>>>>>> Agree 100%
>>>>>>>>> The easiest way is probably just to add a attribute in the job
>>>>>>>>> description taking the whole JSDL.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Quoting [Thilo Kielmann]
>>>>>>>>
>>>>>>>>> Yes!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Quoting [Shantenu Jha]
>>>>>>>>>
>>>>>>>>>> Shouldn't we ensure that SAGA consumes JSDL w/o any
>>>>>>>>>> problem/changes?
>>>>>>>>>>
>>>>>>>>>>
>>>>>> --
>>>>>> "So much time, so little to do..." -- Garfield
>>>>>> --
>>>>>> saga-rg mailing list
>>>>>> saga-rg at ogf.org <mailto:saga-rg at ogf.org>
>>>>>> http://www.ogf.org/mailman/listinfo/saga-rg
>>>>>>
>>>>>
>>>>>
>>>>
>>> ----------------------------------------------------------------------
>>>> --
>>>>
>>>> --
>>>> saga-rg mailing list
>>>> saga-rg at ogf.org
>>>> http://www.ogf.org/mailman/listinfo/saga-rg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 4385 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.ogf.org/pipermail/saga-rg/attachments/20070220/f7ffe993/attachment-0003.bin
More information about the saga-rg
mailing list