[gsa-rg] Use case document updated and pre GGF14 plans

[ph.wieder]

Hi Ramin,

Ramin Yahyapour wrote:
> Hi Philipp,
> 
> as a follow-up on our discussion from our last F2F meeting,
> here are some thoughts for the use-case doc:
> 
> 
>>- The existing use cases are quite difficult, we need to add something
>>like simple job submission.
>>- A new chapter with "Scheduling Profiles" is needed (NOTE: Are we
>>talking really about profiles in the sense of OGSA.?) A possible "Simple
>>job submission profile" would include an implementation example, the key
>>functions needed (only static info, simple cost function, ...).
> 
> 
> I would propose to add this "Profile" Section to the doc (which actually includes
> the more abstract use-cases we have in mind). And we keep the current use-cases
> as they are (as some kind of application examples).
This is a very good idea. I actually thought about the different foci of 
the use cases and I think your suggestion helps to make the distinction 
clear.

> Let us discuss a little about these profiles, shall we?
> Maybe I just start a basic list which I see from looking at the use-case
> contributions:
> 
> 1) Simple Job Submission Profile:
> ----------------------------------
> This profile would cover the simple case:
> - no complex workflow
> - no reservation requirements
> - no co-allocation of resources
> - the lower-level schedulers do not support any service level guarantees.
>   That means, no information about the actual execution time is known in advance.
> - a higher-level scheduler receiving this job requests only uses static and dynamic
>   information which are available in an information service.
> - Based on some heuristic or algorithm the higher-level scheduler selects a
>   lower-level scheduler and forwards the request to this remote system
> - there is no real Grid scheduling involved in this scenario
> - the higher-level scheduler works just as a broker/dispatcher which fowards
>   a job to remote sites.
> 
> - This profile covers some HPC scenarios in which the local sites have simple
>   queuing systems (e.g. FCFS w/o backfilling) without complex support for
>   interaction with higher-level Grid schedulers.
>   The higher-level scheduler just makes a selection based on the current situation:
>   like what Grid resources are available which fit the request,the current load,
>   queue length on the system, repsonse time on past requests etc.
Do the profiles reflect the user's viewpoint or that of scheduling 
system? Or: Are we talking about usage profiles (which I would call 
patterns) or provision profiles. This is not quite clear to me, and I 
think we should agree how to approach this issue.
> 
> 
> 2) Single-Site Execution with Service Guarantees Profile
> - similar to profile 1 this job does not require any co-allocation or complex
>   workflows.
> - however, this time service guarantees in terms of start and end time of a
>   resource allocation is supported.
> - the guarantee can be precise with exact start and end time of the allocation
>   or there are just quality information about latest start and/or end time.
>   In the latter case, we do not know exactly when a job is executed but we have
>   information when it will be finished at the latest.
> - this profile supports more sophisticated local resource management systems which
>   can give you some kind of information about the service guarantees. A typical example
>   is a queuing system with a backfilling scheduling strategy. These systems can provide
>   you with a deadline when your job will be finished at the latest. However, the job might
>   be started much earlier depending on the execution behaviour of other jobs on the system.
> - It is clear that this profile requires some kind of communication/negotiation between
>   a higher-level scheduler and a lower-level scheduler, as the information about the
>   deadline depends on the specific job requests and can therefore not deduced just
>   from querying an information service.
> 
> 3) Job Execution with Advance Reservation Profile
> - similar to profile 1) this profile now includes the ability to make a concise
> reservation when a job resource allocation will take place.
> - I am not sure whether this should make an independent profile or just be part of
> profile 2). I would suggest to make it an additional profile as the ability to make
> and advance reservation is actually quite different from having information when your
> job is executed. In the latter case it is an information that may just be available
> from the local RMS. However, in advance reservation I expect that a higher-level scheduler
> could request from a lower system to make a reservation for tomorrow 10:00-12:00.
> This usually requires different lower-level scheduler who can cope with such reservations
> beyond simple queuing.
Since a service guarantee like starting/ending time of a job requires 
advance reservation (ok, you could checkpoint/migrate/... and execute 
the job at the desired time, but ...), but other guarantees like 
"cheeper than X" does not, I would see the profile 3) as a special case 
of profile 2).

> 
> 4) Co-Allocation Profile
> - in this profile several resources are required and need to be coordinated. In many
>   use-cases these resources have to be available at the same time.
> ( I am not sure whether we should limit a profile on parallel co-allocation or whether
> we should allow here also co-allocation of resources with different allocation times, as
> for instance needed for planning simple or complex workflows (staging etc.).
I think we should cover co-allocation in general and give examples for 
the different cases you mentioned.

> - To facilitate a Grid scheduling systems that can cope with this co-allocation profile
>   it will be necessary to obtain information about when resources are actually free for an
>   allocation as a higher-level scheduler will need to synchronize between different Grid
>   resources. That means, we need some information about the idle slots on the resources from
>   an information service or some negotiation protocol between higher-level and lower-level scheduler
>   in which we can query for individual offers. It will be the task of a higher-level scheduler to
>   make from these choice the actual scheduling when resources will be co-allocated.
> - this profile will definitely need advance reservation to assure the exact times of the
>   allocation on the different resources.
Isn't this like 3) plus additional functions on the higher scheduling level?
> 
> 
> As I already wrote to quite extend above. I just sketch some potential ideas
> of other profiles:
> 
> x)  Job/Data Staging Profile
> - simple profile in which the availability/transfer of data is taking into account.
> - That means, a scheduler must consider the time needed to make given data available on the
>   target resource.
> - this could be seen as a simple workflow in which we have a planned/coordinated staging of data
>   as part of the scheduling request.
> 
> x) Complex Workflow Profile
> - the job can comprise a complex workflow for which a higher-level scheduler should
>   be able to make an advance planning for the static parts it can foresee.
> - the workflow poses temporal dependencies between different resource allocation.
Isn't that the first "use case" already in the document?

> 
> x) Economic/Market-Based Grid Profile
> - For making the above profiles reasonable simple and therefore implementable,
> I would suggest to omit any economic models from the profiles above.
> That is, we do not negotiate for the price. Nevertheless, the resource allocation
> may cost something and the budget has to be taken into account. But this is
> handled like other simple constraints, e.g. a maximum budget for a resource allocation.
> 
> - I propose that we add an additional more sophisticated profiles in which the negotiation
> on resource offers between lower- and higher-level schedulers comes into play.
> In these negotiation models we can add cost/utility/preference-functions (whatever
> someone has in mind).
Agreed.
> 
> 
> As said at the beginning this is just my try to derive profiles from
> the given use-cases. Open issues: For instance, I do not yet know
> whether we should derive additional profiles specifically for scheduling in a
> coordinated fashion for data and network. Of course, these are just resources as any
> other resource too. However, in my opinion the scheduling for these resources has
> some specialities, e.g. data can be replicated and therefore made available at
> different locations.
> The management of network if this should be taken into account, is usually
> dependend on the selection of the other resources for a job (You will probably
> never query for where there is a network link with at most 2 GBit/s without telling
> the endpoints.)
I have to think about that a bit more. Maybe we solve this while writing 
the other stuff.
> 
> Regarding the doc, I do not think that we need to spend too much text or detail
> on the profiles. A simple and brief collection in the doc might already be quite
> valuable for further discussion on the architectural conclusions we derive for
> these profiles.
The question is if we put the very general use cases into the profile 
section (or do I misunderstand your proposal?), they are much more 
detailed than the other profiles you included in this email. Any 
suggestion wrt that?

Best regards, Philipp.
> 
> Ramin
>