[ogsa-wg] Thoughts on extensions mechanisms for the HPC profile work

[Donal K. Fellows]

Marvin Theimer wrote:
> I mis-spoke when referring to resource matchmaking systems as being
> mostly about exact matches.  What I had in mind was exactly what you
> described and your description is much better than mine.  Many thanks
> for the correction and improved description!

Resource matching code is what I've been working on for a few years now.
It's really why I started engaging fully with GGF; so I could have fewer
resource description languages to have to work with. :-)

> I agree with you that specifying alternatives and prioritizations is
> difficult.  Personally, I'm not convinced that specifying specific
> scores is much easier (having had to use such a system once).  In
> general, I would argue that keeping things simple is most effective and
> that we should make sure that specifying simple common cases is both
> easy and efficient to do.  

I know scoring is hard too, though it's not too bad if you're keeping
the scoring of things separate from a filtering stage. The difficulty is
really if you try to understand the score values themselves; my
experience is that they're pretty arbitrary and often with quite large
magnitudes.

> The beauty of an evolutionary, extensions-based design is that we can
> start with simple approaches, layer more complex alternatives on top as
> desired, and then let user experience decide which extensions are
> actually useful.  Along those lines, I think Karl's suggestion of having
> a binary "ignore/must-support" flag represents a relatively simple
> extension, whereas anything beyond would represent a much more
> complicated extension.

I'd prefer a "mayIgnore" flag. :-) OK, the reason for this is that I
believe that if someone is asking for a feature they should get it or
get a definite early failure by default, and if they want to specify
that they have optional resource requirements they'll have to do extra
work for it. I suppose it's trying to make the default case (which I
always think of as "false" for boolean flags; I think that's a notion
that's programmed into the brains of many people other than me too) be
the common one.

On the other hand, I question the extent to which an optional resource
requirement has any real meaning anyway. The only times I can see a use
are for when you're really trying to capture some other sense of
resource by proxy, such as asking for certain operating systems with an
explicit executable path instead of asking for the abstract application
name that is installed at that location on those systems. But I feel
that people should not ask for such proxies for what they desire; they
should say what they really need (Blast, Gaussian, etc.) and let the
middleware take the strain. Since checking for optional resources makes
writing resource checkers much more complex and yet only gives you a
feature that I think shouldn't be used, what's the point of putting it
in? It just makes life harder, including for us as spec writers.

To elaborate on the last point, I'd like to say that many revisions of
JSDL had a complex language for resource composition, but we threw it
out. This was in part because we were having problems coming up with
good use-cases for it (we could construct silly examples, but nothing
that we'd actually want to use in practice; even the complex cases were
far better served by the introduction of some resource virtualization
than trying to use the several composition schemes we tried) but was
also due to the fact that it proved really hard to write a good spec of
what happens when a resource is optional. What do you do if a resource
is missing? How do you communicate to the application what resources
were actually allocated? Should there be some kind of preferencing
system? What is the semantics of the composition system? What does
negation/complement mean? How do you nail the schema down hard enough so
that stupidities don't slip through?

Throwing that whole lot out was one of the best things we did as a
working group. We'd still be at it now otherwise. :-) It did take
several major rounds of revisions to throw it all out though, and it's
arguable that we missed a little bit of it (i.e. JSDL's somewhat
peculiar outer structure). That's probably going to turn out useful
though; the world is funny that way...

Donal.