[SAGA-RG] Fwd (andre at merzky.net): Re: JSDL - SAGA
Andre Merzky
andre at merzky.net
Mon Feb 19 16:10:11 CST 2007
Ha, actually, that is what I expected this discussion would
lead to :-P
You write:
> - JSDL is an OGF Standard now and there is no way of ignoring it in SAGA
and Pascal:
> The necessity to support JSDL 1.0.x is a necessity, since
> it is an OFG standard.
I think we need to step back a little. Yes, JSDL is an OGF
standard - but so is OGSA, OGSA Security, OGSA-DAIS.
Yes, we should support JSDL, and in particular be
implementable on top of JSDL. We also should support OGSA,
and in particular be implementable on top of OGSA. Same
with DAIS.
That does NOT mean that we need an API to write JSDL
documents, nor an API to write OGSA Base NOtification
documents, nor an API to write DAIS Collection Access XML
docs.
SAGA is NOT a general purpose API for existing OGF
standards. It is _designed_ to be incomplete.
Quoting [Hartmut Kaiser] (Feb 19 2007):
>
> Hi all,
>
> After thinking about this issue a bit more I would like to make a even more
> 'radical' suggestion.
>
> Let's recap:
> - JSDL is an OGF Standard now and there is no way of ignoring it in SAGA
> - the current SAGA JD is more constrained in terms of scope if compared to
> JSDL
> - there will be a lot of middleware supporting JSDL in the future
> - there wll be a lot of tooling support to create JSDL in the future
>
> A viable solution to this in SAGA could be (duck):
>
> - redo the SAGA job_service and job interfaces so that these work solely
> based on JSDL strings
Further, the saga job description has a (small) number of
keys which are not supported by JSDL, and will not be in the
near future. Those are scheduling related: Queue,
JobStartTime etc.
> - supply a SAGA JD component providing a _simple_ interface for JSDL
> construction:
The JSDL schema has repeatable, hierarchical elements, with
0..n occurences of sections etc etc. It can include XML
snippets from arbitrary name spaces. Just one example from
many in the JSDL spec:
"7.1 Attribute Extension
Every JSDL element allows for additional attributes, as
many as needed, provided these attributes have a namespace
other than the normative namespaces defined by JSDL."
There is a good API to write XML documents (DOM), and there
are many tools to do so. Why should SAGA re-do that? None
of these APIs/tools is simple - they cannot be, by design.
Please check the JSDL spec: JSDL is simple for middleware
level - but it is too complex (and I mean really) on the
level SAGA is targeting at.
So, providing an API to create JSDL is, IMHO, out of scope.
> job_description::construct(string jsdl);
> job_description::as_jsdl();
>
> job_service::create_job(string jsdl);
> job::migrate(..., string jsdl, ...);
>
> where we clearly could stick to JSDL V1 for this conversion and the allowed
> attribute set on our job descriptions.
I honestly don't see the reason why we should deviate from
our design principles (simple API, orient on use cases,
don't do more than users ask for) just because JSDL exists.
Its nice it exists, we support it (it can and should be used
by SAGA _implementations_). But it has no more appeal on
API level than has OGSA Base Notification, for example.
So, my proposal is:
- stick to the JD we have in SAGA (it works nicely with
JSDL)
- support upcoming JSDL extensions in due time, either by
new keys on the job description (JSDL scheduling), or
programmatically (i.e. JSDL parameter sweeps)
- if there are use cases asking for it (only then),
provide a jsdl_to_jd converter
Well, I realise I am repeating myself - not a good sign. We
probably should give the topic anoter week of rants, and
then do a straw poll on the mailing list, or continue to
fight on a phone call... :)
Cheers, Andre.
> Regards Hartmut
>
> > -----Original Message-----
> > From: saga-rg-bounces at ogf.org
> > [mailto:saga-rg-bounces at ogf.org] On Behalf Of Pascal Kleijer
> > Sent: Sunday, February 18, 2007 8:51 PM
> > To: SAGA RG
> > Subject: Re: [SAGA-RG] Fwd (andre at merzky.net): Re: JSDL - SAGA
> >
> > Dear all,
> >
> > I have given me the week-end to get some reflection on the
> > problem. As mentioned it is not a trivial issue. If things
> > goes out of hand, we might as well rename the project CAGA of
> > not go gaga :P
> >
> > The necessity to support JSDL 1.0.x is a necessity, since it
> > is an OFG standard. I do not think JSDL 1.0 is overly complex
> > right now. The Job Description should be made such that
> > passing from one form to another is automatic (JD -> JSDL).
> > The support of a property key for JSDL isn't bad; if used,
> > all other properties can be potentially ignored. The JD would
> > not have to parse it. The current specification is simple
> > enough to not violate the "S" rule of SAGA.
> >
> > The problem we have is with all the up coming extensions. In
> > my case, I clearly violated the SAGA API rules in order to
> > make the system run with NAREGI. I tweaked the problem to
> > solve my case. In the 80/20 rule, it probably means that the
> > SAGA job submission won't be usable. I will have to write
> > code that does not deal with SAGA and goes directly to the
> > underwear. Now I am speaking as a Grid "expert" and not as a
> > Grid newbie (which is the SAGA user base). If you want to use
> > complex Grid programming and do complex things, SAGA might
> > not be the right solution.
> >
> > The proposal made by Dan is not elegant in my opinion because
> > it will force developer to know the underlying technology
> > they want to use; it will also clutter the JD. The proposal
> > from Thilo might work for OO languages because you can hide a
> > lot behind patterns, but it won't mean a specific
> > implementation will support such scheme.
> >
> > The passing through solution seems to be mildly seen by the
> > some GAT people here. I would say no, if the error handling
> > is explicit enough to return a message telling that this JSDL
> > cannot be interpreted by the underlying system. I will not
> > see this as a bad solution either.
> >
> > --
> >
> > Best regards,
> > Pascal Kleijer
> >
> > ----------------------------------------------------------------
> > HPC Marketing Promotion Division, NEC Corporation
> > 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan.
> > Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382
> >
> >
> > Daniel S. Katz wrote:
> > > Another related idea is for SAGA to have one class for job
> > > descriptions with a quite loose ability to add "hints",
> > such as in MPI
> > > I/O. The hints would not be required and might not be used, but in
> > > some cases would be used, depending on what SAGA is sitting
> > on top of.
> > >
> > > Dan
> > >
> > > Thilo Kielmann wrote (on 2/16/07 5:54 AM):
> > >> Folks,
> > >>
> > >> I think we are having conflicting goals here.
> > >> (Technical goals, not personal ones ;-)
> > >>
> > >> On one hand, we have the "S for simplicity" in SAGA, and
> > we must keep it.
> > >>
> > >> On the other hand, we have the necessity to support JSDL,
> > future JSDL
> > >> extensions, or any other types of job decscriptions that
> > people want to use.
> > >> (And still might to be invented.)
> > >>
> > >> Assume, JSDL++ (whatever it will look like in near future) will
> > >> become a widely adopted standard. (Or anything else,
> > doesn't matter
> > >> in the following.) Then, SAGA implementations will have to use
> > >> JSDL++, and to form JSDL++ from SAGA job descriptions.
> > >> Simultaneously, users are likely to use JSDL++ themselves,
> > and may wnat to use JSDL++ to express their resource needs.
> > >> At this point in time, SAGA will sit in the middle, and it may be
> > >> very clumsy to first translate from JSDL++ to a SAGA job
> > description,
> > >> and then back to
> > >> JSDL++ somewhere "down under" in the implementation.
> > >>
> > >> For the very purpose of SAGA as a universal and simple
> > grid API, it has to:
> > >> - be independent of job description standards (mostly simpler than
> > >> them)
> > >> - support job description standards
> > >>
> > >>
> > >> My suggestion:
> > >>
> > >> SAGA should have one class of job descriptions, and the
> > possibility
> > >> to create subclasses for more specific job descriptions
> > (like JSDL).
> > >> Such subclasses could be defined as separate extension documents
> > >> (just like gridcpr). (Or was it "resource descriptions"???)
> > >>
> > >> With this approach, users could still write programs that are
> > >> independent of the underlying job submission machinery, having a
> > >> simplified view on jobs and resource attributes etc. etc.
> > >> At the same time, subclassed job/resource descriptions could be
> > >> "passed through" transparently from the API to the
> > implementeation,
> > >> without being converted back and forth, a process that is
> > very likely
> > >> to loose some important details.
> > >>
> > >>
> > >> Is this a route to go?
> > >>
> > >>
> > >> Thilo
> > >>
> > >> On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
> > >>
> > >>> From: 'Andre Merzky' <andre at merzky.net> <mailto:andre at merzky.net>
> > >>> To: SAGA RG <saga-rg at ogf.org> <mailto:saga-rg at ogf.org>
> > >>> Subject: [SAGA-RG] Fwd (andre at merzky.net
> > <mailto:andre at merzky.net>):
> > >>> Re: JSDL - SAGA
> > >>>
> > >>> Hi group(s),
> > >>>
> > >>> a couple of us had a recent discussion (f2f and email) about JSDL
> > >>> and SAGA. The question is: should we support JSDL fully on API
> > >>> level? E.g., should we allow the application programmer to
> > >>> specify/use JSDL documents for job creation?
> > >>>
> > >>> The reasons for doing that are compelling: JSDL is one of
> > the most
> > >>> acknowledged standards in OGF, and the number of backends
> > supporting
> > >>> JSDL is rapidly increasing it seems.
> > >>> Supporting JSDL directly would allow to interface with
> > other tools
> > >>> using JSDL, and would allow to reuse JSDL documents where
> > these are
> > >>> already available.
> > >>>
> > >>> Well, I have however some problems with that approach, which are
> > >>> outlined in the cited email below.
> > >>>
> > >>> Do you guys have any other thoughts, and what solution would you
> > >>> prefer?
> > >>>
> > >>> Cheers, Andre.
> > >>>
> > >>>
> > >>> ----- Forwarded message from 'Andre Merzky' <andre at merzky.net>
> > >>> <mailto:andre at merzky.net> -----
> > >>>
> > >>>
> > >>>> From: 'Andre Merzky' <andre at merzky.net> <mailto:andre at merzky.net>
> > >>>> To: Shantenu Jha <sjha at cct.lsu.edu> <mailto:sjha at cct.lsu.edu>
> > >>>> Cc: Hartmut Kaiser <hkaiser at cct.lsu.edu>
> > <mailto:hkaiser at cct.lsu.edu>,
> > >>>> 'Thilo Kielmann' <kielmann at cs.vu.nl> <mailto:kielmann at cs.vu.nl>,
> > >>>> 'Andre Merzky' <andre at merzky.net> <mailto:andre at merzky.net>
> > >>>> Subject: Re: JSDL - SAGA
> > >>>>
> > >>>> Hartmut and I discussed that somewhat last week. So he
> > knows I am
> > >>>> not wholehartedly for that option. SAGA is supposed to abstract
> > >>>> the low level details, not to expose them.
> > >>>>
> > >>>> JSDL is going to define a number of extensions now.
> > Some of these
> > >>>> extensions are very useful for us, others not.
> > >>>> Mostly, they will be more complex than JSDL itself.
> > >>>>
> > >>>> Are we going to support the extensions? Which? All/some?
> > >>>> How to select? What error do we report on unsupported
> > extensions?
> > >>>> Do we mandate that extensions are supported by the backends?
> > >>>> Which?
> > >>>>
> > >>>> Even w/o extensions: is the job description updated
> > after an JSDL
> > >>>> attrib is set? What about those attribs which are not
> > JSDL keys?
> > >>>> Assume an implementation which implements the existing SAGA job
> > >>>> description keys: MUST it support complete JSDL now? What error
> > >>>> whould it report?
> > >>>>
> > >>>> These are probably all solvable problems, and I do agree
> > that there
> > >>>> are advantages, i.e. the re-use of existing JSDL documents.
> > >>>> Anyway, IMHO we should be careful, consider if we really have
> > >>>> enough use cases etc. Also, a free function
> > >>>> jsdl_to_job_description may do the trick, w/o
> > complicating the job
> > >>>> package itself.
> > >>>>
> > >>>> Cheers, Andre.
> > >>>>
> > >>>>
> > >>> Another point I'd like to raise is: if HPC profile
> > bekomes a widely
> > >>> accepted OGF standard, do we support it directly, too? Or
> > >>> OGSA-Workflow? Where to stop, and where is the 'S'
> > >>> in that approach?
> > >>>
> > >>> Andre.
> > >>>
> > >>>
> > >>>
> > >>>> Quoting [Shantenu Jha]
> > >>>>
> > >>>>> What little I know, I think so to.
> > >>>>>
> > >>>>>
> > >>>>> Quoting [Hartmut Kaiser]
> > >>>>>
> > >>>>>> Agree 100%
> > >>>>>> The easiest way is probably just to add a attribute in the job
> > >>>>>> description taking the whole JSDL.
> > >>>>>>
> > >>>>>>
> > >>>>> Quoting [Thilo Kielmann]
> > >>>>>
> > >>>>>> Yes!
> > >>>>>>
> > >>>>>>
> > >>>>>> Quoting [Shantenu Jha]
> > >>>>>>
> > >>>>>>> Shouldn't we ensure that SAGA consumes JSDL w/o any
> > >>>>>>> problem/changes?
> > >>>>>>>
> > >>>>>>>
> > >>> --
> > >>> "So much time, so little to do..." -- Garfield
> > >>> --
> > >>> saga-rg mailing list
> > >>> saga-rg at ogf.org <mailto:saga-rg at ogf.org>
> > >>> http://www.ogf.org/mailman/listinfo/saga-rg
> > >>>
> > >>
> > >>
> > >>
> > >
> > >
> > ----------------------------------------------------------------------
> > > --
> > >
> > > --
> > > saga-rg mailing list
> > > saga-rg at ogf.org
> > > http://www.ogf.org/mailman/listinfo/saga-rg
--
"So much time, so little to do..." -- Garfield
More information about the saga-rg
mailing list