[ogsa-bes-wg] Re: [jsdl-wg] Questions and potential changes to JSDL, as seen from HPC Profile point-of-view

[Marvin Theimer]

Hi;

I'm going to reply to a bunch of the previous emails in this one.  As
several people, including Oxana and Karl and Michel and Donal, have
pointed out, the fully general problem of describing job requirements
and the resources available from various candidate execution systems is
awfully hard.  Indeed, it is arguably still a research problem and hence
not something for defining standards about.

The HPC profile work is about defining a standard for those simple
situations that are well-understood and common enough that a standard
would provide some tangible benefit in terms of allowing multiple
implementations of things like job schedulers and job scheduling client
libraries to interoperate with each other.  From that point-of-view, I
need to divide the world of JSDL and RSS solutions into at least two
parts: a (very simple) part that we understand well enough to
standardize now and a research part that requires further exploration
and should NOT be standardized now.  This is why I keep banging the drum
about structuring things as base cases and extensions, so that I can
define a very simple base case and a few simple extensions for purposes
of the HPC profile standards work now (i.e. this summer) while allowing
a graceful path to the future for all the really interesting and
important work yet to come.

So, what should be the base case and the few simple extensions that
should get standardized by the HPC profile working group now (meaning
this summer)?  I would love to hear peoples' opinions and feedback --
especially in terms of deltas on the straw man proposal that I posted in
my previous email(s). :-)

With thanks in advance,
Marvin.

-----Original Message-----
From: Donal K. Fellows [mailto:donal.k.fellows at manchester.ac.uk] 
Sent: Friday, June 09, 2006 7:53 AM
To: Oxana Smirnova
Cc: Alexander Papaspyrou; Michel Drescher; Karl Czajkowski; Marvin
Theimer; JSDL Working Group; ogsa-bes-wg at ggf.org; Ed Lassettre; Ming Xu
(WINDOWS)
Subject: Re: [ogsa-bes-wg] Re: [jsdl-wg] Questions and potential changes
to JSDL, as seen from HPC Profile point-of-view

Oxana Smirnova wrote:
> As a user, I strongly disagree. I *am* interested to have my jobs 
> executed as soon as possible, for sure. This means I want them to be 
> sent by a workload management system not just to any site that matches

> job requirements, but to the best site - e.g., with the fastest 
> processor, bigger memory, better bandwidth etc. I may also be
interested 
> in to send them to a cheapest site, or to a fastest site among the
cheap 
> ones. I may prefer to stay away from sites that use afs, and I may
need 
> to specify that I need inbound connectivity for a worker node. I
perhaps 
> only want to use sites in one specific country, for some licensing 
> reasons. There are so many levels of optimization that users need, one

> can write a book about it.

This is certainly a good indication that there cannot be a fixed set of
resource selection descriptions. Any truly workable solution to this
problem has to be really quite general. I've already designed (and
implemented) such a scheme as it happens. ;-)

> This is not a hypothetical case: I know many users that schedule jobs
by 
> hand to sites that in their experience are better, while the workload 
> management system can not tell this from the available information or 
> job description. This manual scheduling is orthogonal to the Grid
idea, 
> I dare say.

It's certainly reflecting the (too common) case where users and admins
are in different warring camps. :-(

> Instead, job specification should include very explicit 
> attributes, including potentially a preferred sysadmin name :-) As it
is 
>  pretty difficult to define a boundary between generic service levels 
> description and specific informmation for fine-tuning, I would say
it's 
> better to stay with one "language" that covers all.

I should note that this is properly part of the domain of the OGSA-RSS
working group, which I happen to chair. :-) I'm currently looking for a
co-chair (more than one would be cool!) so that I'm not completely
overworked. Volunteers will be able to partake in numerous benefits,
such as invitations to speak at OGSA F2F meetings and the chance to go
to the Chairs' Appreciation Night...

Donal.