[gsa-rg] Use case document updated and pre GGF14 plans

[Ramin Yahyapour]

Hi Philipp,

> 
> Do the profiles reflect the user's viewpoint or that of scheduling
> system? Or: Are we talking about usage profiles (which I would call
> patterns) or provision profiles. This is not quite clear to me, and I
> think we should agree how to approach this issue.

I do not really get the question about user vs. scheduling system.
Maybe you can give me some more hints.

But here a my 2cts anyway:
The profiles describe in a more abstract way a simplified use-case scenario.
It is clear that there are more complex application scenarios for which
some kind od scheduling is needed and simpler ones.
It might easier for some "consumers" of a GSA that they could find there
use case without being overwhelmed by issues which do not apply in their
scenario.
For instance, if you are setting up a Grid for a HPC Grid where the different
HPC clusters or parallel computers do not have any sophisticated local
resource management system which can provide you with advance reservation features
or deadlines for a potential job submission, there is no need for a complex
Grid scheduling architecture that is dealing with negotiating advance reservations.
That means, these Grid scenario would be completely covered by Profile 1.

In another application example, you might participate in the LHC experiment and use
their Grid, you would like to include data management as part of the scheduling. Here you
need a more sophisticated profile in which you can at least coordinate the staging/
availability of data. Here you need services and information about the location
of data, possibilities and cost of data transfers, availability of storage.

Therefore, I do not see where we talk about user point of view vs. scheduling point
of view. We are talking about common profiles of use cases scenarios and the
corresponding requirements on the scheduling architecture. Of course we focus
only on the scheduling related part.


> 
>>
>>
>> 2) Single-Site Execution with Service Guarantees Profile
>> - similar to profile 1 this job does not require any co-allocation or
>> complex
>>   workflows.
>> - however, this time service guarantees in terms of start and end time
>> of a
>>   resource allocation is supported.
>> - the guarantee can be precise with exact start and end time of the
>> allocation
>>   or there are just quality information about latest start and/or end
>> time.
>>   In the latter case, we do not know exactly when a job is executed
>> but we have
>>   information when it will be finished at the latest.
>> - this profile supports more sophisticated local resource management
>> systems which
>>   can give you some kind of information about the service guarantees.
>> A typical example
>>   is a queuing system with a backfilling scheduling strategy. These
>> systems can provide
>>   you with a deadline when your job will be finished at the latest.
>> However, the job might
>>   be started much earlier depending on the execution behaviour of
>> other jobs on the system.
>> - It is clear that this profile requires some kind of
>> communication/negotiation between
>>   a higher-level scheduler and a lower-level scheduler, as the
>> information about the
>>   deadline depends on the specific job requests and can therefore not
>> deduced just
>>   from querying an information service.
>>
>> 3) Job Execution with Advance Reservation Profile
>> - similar to profile 1) this profile now includes the ability to make
>> a concise
>> reservation when a job resource allocation will take place.
>> - I am not sure whether this should make an independent profile or
>> just be part of
>> profile 2). I would suggest to make it an additional profile as the
>> ability to make
>> and advance reservation is actually quite different from having
>> information when your
>> job is executed. In the latter case it is an information that may just
>> be available
>> from the local RMS. However, in advance reservation I expect that a
>> higher-level scheduler
>> could request from a lower system to make a reservation for tomorrow
>> 10:00-12:00.
>> This usually requires different lower-level scheduler who can cope
>> with such reservations
>> beyond simple queuing.
> 
> Since a service guarantee like starting/ending time of a job requires
> advance reservation (ok, you could checkpoint/migrate/... and execute
> the job at the desired time, but ...), but other guarantees like
> "cheeper than X" does not, I would see the profile 3) as a special case
> of profile 2).

I tend to disagree.
I do not see that service guarantees always need advance reservation.
As I wrote in my example, for HPC systems many use some kind of scheduling
strategy like backfilling for which you have to provide a maximum length
of your job.  On these systems, the local scheduler can usually easily provide
you before/at submission time with the information when your new job will
be finished at the latest. However, you still do not know when your job
will actually run. It can (and usually will) be much earlier than the expected
deadline. Therefore, from a practical point of view many existing scheduling systems
from HPC could more or less easily provide you with such a service guarantee.

However, this does not mean that these systems can provide you with advance
reservation also. I associate with advance reservation the possibility that
I get a confirmed reservation that I get a resource allocation on a specific timeframe.
Say, I make a reservation for tomorrow 12:30 to 13:30 on your machine.
For many low-level scheduler this is somewhat difficult to cope with.
However, derivations of backfilling strategies could also deal with this, but this
is not yet common. One reason is that the utilization on a real parallel
machine might be highly affected if a large percentage of jobs would use
adavance reservation. At the end you have many "fix points" in your schedule
for which the scheduler cannot change anything based on the dynamic load situation.
So he can just try to schedule the remaining "free jobs" around these reserved
blocks.

Concluding, I tend more and more that these should be distinct profiles as this
covers better the existing background for many Grid systems.
But this is just my opinion.


>> 4) Co-Allocation Profile
>> - in this profile several resources are required and need to be
>> coordinated. In many
>>   use-cases these resources have to be available at the same time.
>> ( I am not sure whether we should limit a profile on parallel
>> co-allocation or whether
>> we should allow here also co-allocation of resources with different
>> allocation times, as
>> for instance needed for planning simple or complex workflows (staging
>> etc.).
> 
> I think we should cover co-allocation in general and give examples for
> the different cases you mentioned.

Okay, why not. But this would mean that if we would like that you cannot
present a Grid scheduling implementation for this profile without requiring
a complete solution for workflows also. Or am I wrong?

>> - To facilitate a Grid scheduling systems that can cope with this
>> co-allocation profile
>>   it will be necessary to obtain information about when resources are
>> actually free for an
>>   allocation as a higher-level scheduler will need to synchronize
>> between different Grid
>>   resources. That means, we need some information about the idle slots
>> on the resources from
>>   an information service or some negotiation protocol between
>> higher-level and lower-level scheduler
>>   in which we can query for individual offers. It will be the task of
>> a higher-level scheduler to
>>   make from these choice the actual scheduling when resources will be
>> co-allocated.
>> - this profile will definitely need advance reservation to assure the
>> exact times of the
>>   allocation on the different resources.
> 
> Isn't this like 3) plus additional functions on the higher scheduling
> level?

Yes, you are right. The support for co-allocation would only affect the higher-level
schedulers. Therefore, you may be right that this does not qualify for a complete new
profile. It will not generate much requirements on services or protocols.
Maybe just the need of combining several agreements into one single container agreement
(or several agreement requests/offers into one combined bundle).

>> x) Complex Workflow Profile
>> - the job can comprise a complex workflow for which a higher-level
>> scheduler should
>>   be able to make an advance planning for the static parts it can
>> foresee.
>> - the workflow poses temporal dependencies between different resource
>> allocation.
> 
> Isn't that the first "use case" already in the document?

Yes it is. However, looking on these use-cases I would see this as
one of the most complex ones. Therefore, I would propose to make it
a separate profile. In writing the profile section we could reference
this use-case.



>> Regarding the doc, I do not think that we need to spend too much text
>> or detail
>> on the profiles. A simple and brief collection in the doc might
>> already be quite
>> valuable for further discussion on the architectural conclusions we
>> derive for
>> these profiles.
> 
> The question is if we put the very general use cases into the profile
> section (or do I misunderstand your proposal?), they are much more
> detailed than the other profiles you included in this email. Any
> suggestion wrt that?

I would suggest to have two distinct parts of the document.
First, we have actual application use-cases from projects. (Maybe we will
get some more over time for future versions of this doc.)
Then, we "derive" from these quite "heterogenously" written
contributions, our profile in the second part of the doc.

We might even consider to have a simple table in the "requirements document"
later on, in which we mark for the different profiles what kind of services,
protocols, features must be available to cover this profile.

Profile 1: No Reservation, No Negotiation, Accounting, No Billing
Profile x: Reservation, Negotiation, Accounting, No Billing

Ramin


-- 
Dr.-Ing. Ramin Yahyapour         | mailto:Ramin.Yahyapour at uni-dortmund.de
Computer Engineering Institute   | phone: +49 (231) 755-2735
University of Dortmund           | mobil: +49 (179) 5261973
44221 Dortmund / Germany         | fax:   +49 (231) 755-3251