[jsdl-wg] CREAM JDL vs JSDL
Andrew Stephen McGough
asm at doc.ic.ac.uk
Thu May 12 02:37:19 CDT 2005
Dear Massimo,
Many thanks for your interest and your comparison of CREAM JDL with
JSDL. Before I address the issues that you raise I feel it best to first
set out the scope of what we feel the JSDL group is aiming for and
indeed where the group should progress to from this point on.
It is probably best to define JSDL 1.0 as a "minimum standard". There
are many Grid job submission systems (and consumers) already in
existence. The main problem that we have had in defining JSDL 1.0 is
that these different systems define what a job is and what attributes
you can specify about them vary dramatically. Systems such as Condor
allow for workflows through the Condor DAGMAN system with the ability to
specify to an intricate level of detail the type of resource preferred
for execution while other systems such as (Sun) Grid Engine simply allow
single shell scripts to be deployed onto a computational queue. In order
to come up with a specification that all potential consumers were
willing to consider we have had to scope back the functionality of JSDL
1.0. Hence this has become effectively a "minimum standard requirement"
for job submission. We don't have the authority (or desire) to force
(Sun) Grid Engine to support Condor DAGMAN workflows. Likewise we can't
force Condor to support the (Sun) Grid Engine ticketing system - which
I'm sure they would argue is far superior to the Condor rank &
requirements system.
With this said JSDL 1.0 is not the end of the story by any means. We see
it rather being a core set which further standardization can be built
on. Functionality which is not core to all JSDL consumers can be
standardized as "add-ons" to the core specification. This is how we feel
that we can support the high level functionality which is really needed
to make good use of the Grid.
If I can now move over to the points that you raised in your email. For
no particular reason I'd like to cover them in reverse order:
Support for matchmaking: I for one believe the current functionality in
JSDL for matching with resources is minimal - though having said this it
matches the minimum functionality that we can expect to find on current
job submission systems. At this point we need to convince potential JSDL
consumer vendors to adopt JSDL. If this can be achieved by a simple
mapping to their existing system then we have a good chance to get this
adoption. However, if they are required to add in vast amounts of
functionality then they are less likely to adopt.
With this said, I am working with a group at University College London
(UCL) who are using JSDL to submit to a Condor pool. Within this work
they are developing on extensions to JSDL to deal with extra
functionality core to Condor. One of the elements they are proposing is
a <condor:rank> element which is added into a <Resource> element. This
is perfectly consistent with the JSDL standard 1.0 and once we get JSDL
1.0 out of the door we will propose an extension to JSDL 1.0 to deal
with Condor systems. This I hope you could take an active role in to
make this extension specification usable both with Condor and CREAM.
Similarly for the idea of parameter sweep operations - they are looking
into extensions to provide this. Interestingly I was contacted the other
day by someone at the University of Edinburgh who wants to write an
extension to our locally developed JSDL consumer so that the JSDL
parametric sweep extensions can be used on a Globus cluster by
generating multiple Globus submissions to represent the single JSDL job
submission.
In terms of the Workflow: From my understanding of Condor DAGMAN which
is the core of the EGEE job submission system (it's a few years since
I've checked this one) a DAGMAN workflow is a file containing a number
of Classads along with some control flow instructions. We also envisage
a similar process for JSDL. However, the GGF only allows a group to be
chartered to complete one task - in our case to develop a Job submission
language. With this said I know that there are several people within the
group (including some of the chairs) who want to extend JSDL with
workflow. To achieve this we will need to develop a new group and then
perform these tasks. Hopefully this would be something you would be keen
to take part in.
I've just realized how much I've written! So I shall summaries as follows:
- View JSDL 1.0 as the "core" elements of the system. With the "you may
choose to support these features" coming in later versions of JSDL. This
by no means prevents you from using these higher level
functionalities at this stage, nor from taking an active role in
standardizing them.
- We are keen for your input. Hopefully you can now see how we see the
JSDL work progressing. We are by no means intending to wrap things up
with the release of JSDL 1.0 and indeed seek to provide the higher level
functionality once this is complete.
- Many features that both you and other groups seek can't be put into
the core of JSDL 1.0. Though hopefully with your help we can work with
these with the goal of obtaining standardization for them in the near
future.
I'd also like to invite you to talk to us, preferably in person, about
this as it is far too easy to misinterpret things within an email. We'd
be more than happy to talk with you at GGF14, or if you would prefer we
could join you for a teleconference/VRVS/AccessGrid discussion sometime
before then.
With best regards,
steve..
Massimo Sgaravatto - INFN Padova wrote:
>Dear all
>
>In the context of the Compute Resource Management Interfaces (CRM)
>initiative (the same mentioned by Oxana), the Job Description Language
>(JDL) used by the CREAM (Computing Resource Execution AND Management)
>service has been compared with JSDL.
>The CREAM JDL is a subset of the EGEE WMS JDL, that is the language used
>to specify characteristics and requirements for jobs submitted to the EGEE
>Workload Management System. This JDL is a high-level, user-oriented
>language based con Condor Classads for describing jobs and complex
>requests (aggregates of jobs). It is a fully extensible language, hence
>the user is allowed to use whatever attribute for the description of a
>request. However, only a certain set of attributes, referred as "supported
>attributes", is taken into account.
>
>The result of this evaluation is available at:
>
>http://www.pd.infn.it/grid/jra1/CREAMJDL-vs-JSDL.pdf
>
>A presentation summarizing these results is also available at:
>
>http://www.pd.infn.it/~sgaravat/Grid/CREAMJDL-vs-JSDL-EGEE3.pdf
>http://www.pd.infn.it/~sgaravat/Grid/CREAMJDL-vs-JSDL-EGEE3.ppt
>
>
>In short we found that for many functionality for which specific CREAM
>JDL attributes exist there aren't "default" JSDL directives that could be
>used.
>For most of them this is not surprising since they refer to
>features/functionality very specific to our architecture. In this case I
>guess JSDL extensions could be used.
>
>On the other hand (and here I agree with Oxana) there are some issues
>that in my opinion could/should probably be addressed by the default JSDL
>directives.
>In particular (and, again, I agree with Oxana):
>
>- the specification of workflows (in our case DAGs, job collections,
> parametric jobs)
>- the support for matchmaking, i.e. allowing specifying requirements (and
> preferences) on the target resource where the job has to be executed.
>
>
>Thanks for your attention
>
> Cheers, Massimo
>
>
>
>
>
> \\\|///
> \\ ~ ~ //
> (/ @ @ /)
> -------oOOo-(_)-oOOo----------------------------------
> Massimo Sgaravatto
> INFN Sezione di Padova
> Via Marzolo, 8
> 35131 Padova - Italy
> Tel: ++39 0498277047 Fax: ++39 0498277102
> oooO E-mail: massimo.sgaravatto at pd.infn.it
> ( ) Oooo Home page: http://www.pd.infn.it/~sgaravat
> --------\ (----( )----------------------------------
> \_) ) /
> (_/
>
>
>
>
>
>
>
--
--
------------------------------------------------------------------------
Dr A. Stephen McGough http://www.doc.ic.ac.uk/~asm
------------------------------------------------------------------------
Technical Coordinator, London e-Science Centre, Imperial College London,
Department of Computing, 180 Queen's Gate, London SW7 2BZ, UK
tel: +44 (0)207-594-8409 fax: +44 (0)207-581-8024
------------------------------------------------------------------------
Assistant Warden, West Wing, Beit Hall, Imperial College,
Prince Consort Road, London, SW7 2BB tel: +44 (0)207-594-9910
------------------------------------------------------------------------
More information about the jsdl-wg
mailing list