[ogsa-bes-wg] Re: [jsdl-wg] Questions and potential changes to JSDL, as seen from HPC Profile point-of-view

[Marvin Theimer]

Hi;

 

One thing that I think might be useful is to enumerate use cases to try
to identify the simple, common use cases.  In particular, I suspect that
looking at what various existing production (as compared to research)
meta-schedulers support will be instructive since they already face the
task of trying to schedule against programmatically defined interfaces
for existing BES-like services (as compared to human clients, who can
eyeball a given service's resource descriptions/policies and then
perform scheduling decisions in wetware).

 

My first cut at looking at various meta-schedulers - including LSF and
Condor - is that the following types of resource descriptions get used:

*        Simple aggregate descriptions, such as the number of available
CPUs in a subsidiary scheduling service (e.g. a compute cluster), the
average CPU load, and the job queue length.

*        Named queues that jobs can be submitted/forwarded to.  LSF, in
particular, allows for the definition of a variety of queues that are
effectively globally visible and that a LSF (meta-)scheduler can forward
jobs to.  A concrete example is a "fan-in" scenario, in which a large
central compute cluster accepts "large jobs" from a set of distributed
workgroup clusters to which human users submit their various jobs.  A
"large job" queue is defined that all the workgroup cluster schedulers
are aware of and users can then submit all their jobs to their local
workgroup cluster. For large jobs they submit it to the "large job"
queue and the workgroup scheduler forwards jobs received for that queue
to the central compute cluster's job scheduler.

*        "Full" descriptions of subsidiary system compute resources.  In
this case the meta-scheduler receives "full" information about all the
compute resources that exist in all of the subsidiary scheduling
systems.  LSF supports this with a notion of "resource leasing", where a
compute cluster's LSF scheduler can lease some (or all) of its compute
nodes to a remote LSF scheduler.  In that case all the state information
that would normally go to the local LSF scheduler about the leased nodes
is also forwarded to the remote scheduler owning the lease.  Condor
supports something similar with its class-ads design.  A meta-scheduler
will receive class-ad descriptions for all the compute nodes that it may
do match-making for.  In this case, a "full" description consists of
whatever has been put into the class-ads by the individual compute nodes
participating in the system.

 

I would love to hear from other members of the community what their
characterization of common simple use cases is.  Also, it would be great
if people who could provide additional characterization of various
existing production meta-schedulers would post such information to the
mailing list (or point me to where that information has already been
posted if I'm unaware of it :-)).

 

Several things leap to my mind from looking at these usage examples:

*        A few relatively simple, standardized aggregate description
quantities can enable one of the most desirable common use cases, namely
spreading volumes of high-throughput jobs across multiple clusters in an
approximately load-balanced manner.

*        Condor's extensible class-ad design, of arbitrary name-value
pairs with some number of standardized names, has been fairly successful
and provides a lot of flexibility.  As an example, note that LSF's job
forwarding queues can be implemented as class-ad elements.  Note that
the open-ended nature of class-ads means that any installation can
define its own queues (with associated semantics) that are meaningful
within, for example, a particular organization.

*        To efficiently describe something like the leased compute nodes
of an LSF cluster or the class-ads for an entire compute cluster may
require introducing the notion of arrays of descriptions/class-ads.

*        The key to interoperability is to define a useful set of
standard elements that clients can specify in job submission requests
and that resource management services (including compute nodes and
schedulers) can advertise.  The interesting/key question is how small
this set can be while still enabling an interesting set of actual usage
scenarios.  It would be interesting to know what LSF exports when
leasing compute nodes to remote LSF schedulers and what the "commonly
used" set of class-ad terms is across representative sets of Condor
installations.  (I'm guessing that the JSDL working group has already
looked at questions like this and has some sense of what the answers
are?) 

 

I know that the JSDL WG is already discussing the topic of class-ad-like
approaches.  I guess I'm placing a vote in favor of looking at such a
design approach and adding the question of what a beginning "base" set
of standardized class-ad names should be.  This would be "one" approach
that might be workable without requiring that we first solve one or more
research problems.  If JSDL is structured to allow for multiple
approaches then it would allow for progress now while not excluding more
ambitious approaches of the kind Karl outlined in his email.

 

Marvin.

 

-----Original Message-----
From: owner-ogsa-bes-wg at ggf.org [mailto:owner-ogsa-bes-wg at ggf.org] On
Behalf Of Karl Czajkowski
Sent: Saturday, June 10, 2006 10:39 PM
To: Marvin Theimer
Cc: Michel Drescher; Donal K. Fellows; JSDL Working Group;
ogsa-bes-wg at ggf.org; Ed Lassettre; Ming Xu (WINDOWS)
Subject: Re: [ogsa-bes-wg] Re: [jsdl-wg] Questions and potential changes
to JSDL, as seen from HPC Profile point-of-view

 

Marvin:

 

I think one decision to make is whether BES services are homogeneous

or not.  I think Donal is advocating homogeneity. However, I do not

think this is the main source of complexity.  In either case, I agree

with you that JSDL ought to be usable as a core syntax for describing

the "resources available from a BES instance" as well as the

"resources required for an activity".  As you describe it, this is

sort of a "class ad" in the Condor sense of the word. The problem

comes from trying to advertise a resource that can handle multiple

jobs simultaneously.

 

The tricky part is that this is not just "nodes free", but must be

intersected with policies such as maximum job size.  Should there be a

vocabulary for listing the total free resources and the job sizing

policies directly?  Or should the advertisement list a set of jobs

that can be supported simultaneously, e.g. I publish 512 nodes as

quanity 4 128-node job availability slots?  The latter is easier to

match, but probably doesn't work in the simple case because of

combinatoric problem of grouping jobs which are not maximal.  How does

a user know that they can have quantity 8 64-node jobs or not?

 

Also, I am ignoring the very real problem of capturing per-user

policies. I do not think it is as simple as returning a customized

response for the authenticating client.  How is middleware supposed to

layer on top of BES here? How does a meta-scheduler know whether

quantity 8 64-node jobs can be accepted for one user?  For 8 distinct

users?  Does a (shared) meta-scheduler now need to make separate

queries for every client?  How does it understand the interference of

multiple user jobs?  I think there is really a need for a composite

availability view so such metaschedulers can reasonably think about a

tentative future, in which they try to subdivide and claim parts of

the BES resource for multiple jobs. Can this be handled with a

declarative advertisement, or does it require some transactional

dialogue?  The transactional approach seems too tightly coupled to me,

i.e. I should be able to compute a sensible candidate plan before I

start negotiating.

 

If we say all of this is too "researchy" for standardization, then I

am not sure what the standard will really support. Perhaps the best

approach is the first one I mentioned, where relatively raw data is

exposed on several extensible axes (subject to authorization checks):

overall resource pool descriptions, job sizing policies, user rights

information, etc.  The simple users may only receive a simple subset

of this information which requires minimal transformation to tell them

what they can submit. The middleware clients receive more elaborate

data (if trusted) and can do more elaborate transformation of the data

to help their planning.

 

The only alternative I can imagine, right now, would be a very

elaborate resource description language utilizing the JSDL "range

value" concept to expose some core policy limits, as well as a number

of extensions to express overall constraints which define the outer

bounds of the combinatoric solution space.  This DOES seem pretty

"researchy" to me... but maybe someone else sees a more appealing

middle ground?

 

 

karl

 

-- 

Karl Czajkowski

karlcz at univa.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.ogf.org/pipermail/jsdl-wg/attachments/20060615/09306a5d/attachment.htm