[gsa-rg] Use case document updated and pre GGF14 plans

[Ramin Yahyapour]

Hi Philipp,

as a follow-up on our discussion from our last F2F meeting,
here are some thoughts for the use-case doc:

> - The existing use cases are quite difficult, we need to add something
> like simple job submission.
> - A new chapter with "Scheduling Profiles" is needed (NOTE: Are we
> talking really about profiles in the sense of OGSA.?) A possible "Simple
> job submission profile" would include an implementation example, the key
> functions needed (only static info, simple cost function, ...).

I would propose to add this "Profile" Section to the doc (which actually includes
the more abstract use-cases we have in mind). And we keep the current use-cases
as they are (as some kind of application examples).

Let us discuss a little about these profiles, shall we?
Maybe I just start a basic list which I see from looking at the use-case
contributions:

1) Simple Job Submission Profile:
----------------------------------
This profile would cover the simple case:
- no complex workflow
- no reservation requirements
- no co-allocation of resources
- the lower-level schedulers do not support any service level guarantees.
  That means, no information about the actual execution time is known in advance.
- a higher-level scheduler receiving this job requests only uses static and dynamic
  information which are available in an information service.
- Based on some heuristic or algorithm the higher-level scheduler selects a
  lower-level scheduler and forwards the request to this remote system
- there is no real Grid scheduling involved in this scenario
- the higher-level scheduler works just as a broker/dispatcher which fowards
  a job to remote sites.

- This profile covers some HPC scenarios in which the local sites have simple
  queuing systems (e.g. FCFS w/o backfilling) without complex support for
  interaction with higher-level Grid schedulers.
  The higher-level scheduler just makes a selection based on the current situation:
  like what Grid resources are available which fit the request,the current load,
  queue length on the system, repsonse time on past requests etc.


2) Single-Site Execution with Service Guarantees Profile
- similar to profile 1 this job does not require any co-allocation or complex
  workflows.
- however, this time service guarantees in terms of start and end time of a
  resource allocation is supported.
- the guarantee can be precise with exact start and end time of the allocation
  or there are just quality information about latest start and/or end time.
  In the latter case, we do not know exactly when a job is executed but we have
  information when it will be finished at the latest.
- this profile supports more sophisticated local resource management systems which
  can give you some kind of information about the service guarantees. A typical example
  is a queuing system with a backfilling scheduling strategy. These systems can provide
  you with a deadline when your job will be finished at the latest. However, the job might
  be started much earlier depending on the execution behaviour of other jobs on the system.
- It is clear that this profile requires some kind of communication/negotiation between
  a higher-level scheduler and a lower-level scheduler, as the information about the
  deadline depends on the specific job requests and can therefore not deduced just
  from querying an information service.

3) Job Execution with Advance Reservation Profile
- similar to profile 1) this profile now includes the ability to make a concise
reservation when a job resource allocation will take place.
- I am not sure whether this should make an independent profile or just be part of
profile 2). I would suggest to make it an additional profile as the ability to make
and advance reservation is actually quite different from having information when your
job is executed. In the latter case it is an information that may just be available
from the local RMS. However, in advance reservation I expect that a higher-level scheduler
could request from a lower system to make a reservation for tomorrow 10:00-12:00.
This usually requires different lower-level scheduler who can cope with such reservations
beyond simple queuing.

4) Co-Allocation Profile
- in this profile several resources are required and need to be coordinated. In many
  use-cases these resources have to be available at the same time.
( I am not sure whether we should limit a profile on parallel co-allocation or whether
we should allow here also co-allocation of resources with different allocation times, as
for instance needed for planning simple or complex workflows (staging etc.).
- To facilitate a Grid scheduling systems that can cope with this co-allocation profile
  it will be necessary to obtain information about when resources are actually free for an
  allocation as a higher-level scheduler will need to synchronize between different Grid
  resources. That means, we need some information about the idle slots on the resources from
  an information service or some negotiation protocol between higher-level and lower-level scheduler
  in which we can query for individual offers. It will be the task of a higher-level scheduler to
  make from these choice the actual scheduling when resources will be co-allocated.
- this profile will definitely need advance reservation to assure the exact times of the
  allocation on the different resources.


As I already wrote to quite extend above. I just sketch some potential ideas
of other profiles:

x)  Job/Data Staging Profile
- simple profile in which the availability/transfer of data is taking into account.
- That means, a scheduler must consider the time needed to make given data available on the
  target resource.
- this could be seen as a simple workflow in which we have a planned/coordinated staging of data
  as part of the scheduling request.

x) Complex Workflow Profile
- the job can comprise a complex workflow for which a higher-level scheduler should
  be able to make an advance planning for the static parts it can foresee.
- the workflow poses temporal dependencies between different resource allocation.

x) Economic/Market-Based Grid Profile
- For making the above profiles reasonable simple and therefore implementable,
I would suggest to omit any economic models from the profiles above.
That is, we do not negotiate for the price. Nevertheless, the resource allocation
may cost something and the budget has to be taken into account. But this is
handled like other simple constraints, e.g. a maximum budget for a resource allocation.

- I propose that we add an additional more sophisticated profiles in which the negotiation
on resource offers between lower- and higher-level schedulers comes into play.
In these negotiation models we can add cost/utility/preference-functions (whatever
someone has in mind).


As said at the beginning this is just my try to derive profiles from
the given use-cases. Open issues: For instance, I do not yet know
whether we should derive additional profiles specifically for scheduling in a
coordinated fashion for data and network. Of course, these are just resources as any
other resource too. However, in my opinion the scheduling for these resources has
some specialities, e.g. data can be replicated and therefore made available at
different locations.
The management of network if this should be taken into account, is usually
dependend on the selection of the other resources for a job (You will probably
never query for where there is a network link with at most 2 GBit/s without telling
the endpoints.)

Regarding the doc, I do not think that we need to spend too much text or detail
on the profiles. A simple and brief collection in the doc might already be quite
valuable for further discussion on the architectural conclusions we derive for
these profiles.

Ramin

-- 
Dr.-Ing. Ramin Yahyapour         | mailto:Ramin.Yahyapour at uni-dortmund.de
Computer Engineering Institute   | phone: +49 (231) 755-2735
University of Dortmund           | mobil: +49 (179) 5261973
44221 Dortmund / Germany         | fax:   +49 (231) 755-3251