[saga-rg] tasks and jobs...
G.E.POUND at soton.ac.uk
G.E.POUND at soton.ac.uk
Tue Feb 7 08:33:34 CST 2006
Andre,
The benefits of the application of the TaskContainer semantics to job
submission are compelling. Whilst the separation of JobService.submitJob()
into JobService.create() and Job.run() methods adds a little complexity,
the possibility of optimised bulk operations may justify this.
The role of the JobService changes a little, it may be used to create Job
objects without submitting them to the resource manager. Will
JobService.runJob() invoke the Job object that has been submitted to the
resource manager?
---
I am yet to be convinced that the 'Task' and 'JobManagement' namespaces
should be linked. Whilst the semantics of these namespaces are similar they
are designed for different purposes; a simple model of asynchronous method
calls in the client, and the submission of batch jobs to a remote resource.
The advantage of keeping these two namespaces separate is to avoid an
unnecessary dependency between these two different areas of the API. For
example; if in the future additional methods are required to support
asynchronous method calls these would be reflected in the JobManagement
package. [Or if the Task namespace were altered to support language
specific features, see below]
The advantage of linking the namespaces would be to allow all classes
implementing TaskContainer to handle Job objects.
---
In Java the basis for multithreading support is sub-classing the
java.lang.Thread class (or implementing the java.lang.Runnable interface).
In practice SAGA implementations could return inner classes that may be
called asynchronously (by either of these approaches) for each API method.
This is essentially the whole story (prior to Java 5.0), and the whole
approach could be encapsulated within the SAGA.Task model.
The problem with encapsulation within the SAGA.Task model is that the
fine-grain control over the threads (when sub-classing java.lang.Thread) is
lost. Furthermore when using Java 5.0 it would not be possible to leverage
the high-level concurrency utilities now available
(java.util.concurrent.*). For example the task scheduling framework would
be appropriate to control the execution of the threads.
For more details see:
http://java.sun.com/docs/books/tutorial/essential/threads/index.html
In C# the model is nicer than in Java. Methods may be invoked asynchronously
by creating a 'delegate' for that method using Thread.ThreadStart.
Evaluation of the delegates in different threads can be managed at a
high-level via the thread pool.
When working with these languages it there will be some functionality that
it will be important not to preclude. In Java <5.0 there may be little
above and beyond that which is available in SAGA.Task of value to most
users, and encapsulation may be the correct approach. However the
high-level support for concurrency available in Java 5.0 and C# is
certainly important. I am unsure whether access to this functionality is
best achieved by by-passing, or altering SAGA.Task namespace in the
language bindings.
Graeme
Quoting Andre Merzky <andre at merzky.net>:
> Hi Graeme,
>
> Quoting [G.E.POUND at soton.ac.uk] (Feb 06 2006):
> >
> > Andre,
> >
> > I have a couple of reservations about this action that you may be able
> to
> > answer.
> >
> > I had been hoping to avoid implementing the 'Task' namespace in the
> java
> > bindings and encourage developers to use the language's support to
> > threading to allow asynchronous method calls in the client code.
>
> I understood that much from your last comments, but my
> limited knowledge does not allow me to give a qualified
> answer I'm afraid. Hmm, maybe we can work this out together
> :-)
>
> Could you post some code examples to demonstrate how a
> asynchronous seek (for example) would be coded in a java
> application?
>
> The C++ code would be:
>
> saga::file f (url);
> saga::task t = f.seek <saga::task> (off, whence, &pos);
>
> t.run ();
> t.wait ();
>
>
> Next question is, how would a java application manage many
> tasks - i.e. is there something similar to a
> saga::task_container ?
>
> saga::task_container tc;
>
> tc.add (task_1);
> tc.add (task_2);
> tc.add (task_3);
>
> tc.run ();
> tc.wait ();
>
>
> > I am therefore concerned about creating a dependency between the
> > 'JobManagement' namespace and the 'Task' namespace.
>
> I think, for the java bindings that would mean that job
> implements the task interface. Apart from the run you
> mention below, the semantic of the task interface is
> actually included in job already, more or less, only the
> methods and states are differently named (that is one
> motivation for the proposal really).
>
>
> > The submission of remote jobs is naturally asynchronous, and there are
> > natural semantic parallels to the asynchronous model described by the
> > 'Task' namespace. However from my reading of the API I understood these
> two
> > models to be independent in purpose; creating a dependency could hinder
> the
> > natural description (and development) of these two areas of the API.
>
> Both models are not really different on purpose. Would you
> see an advantage of having them truly separate?
>
> >
> > The idea of a job container for the management of a large number of
> remote
> > jobs is useful. However the TaskContainter does not appear to be wholly
> > compatible; the run() method to start the asynchronous operations is
> > unnecessary for jobs that have been submitted to a remote resource.
>
> You are right. Well, that might be somewhat subtle, but in
> the example code below, the submit_job() call is accompanied
> by c new reate_job(). That creates a job which needs to be
> run(), which would make it compatiple to the task model.
>
> saga::job j1 = job_server.create_job (jd);
> // job state is 'pending'
>
> j1.run ();
> // job state is 'running' or so - 'not pending'
>
> j1.wait ();
> // job state is Done or Failed
>
>
>
> saga::job j2 = job_server.submit_job (jd);
> // job state is 'running' or so - 'not pending'
>
> j1.wait ();
> // job state is Done or Failed
>
>
> That is very similar to the semantics we have for tasks...
>
>
> One reason for this proposal is additionally that we want to
> approach the bulk operations soon. Consider a parameter
> sweep, where 100.000 jobs are to be run.
>
> for ( i = 0; i < 100.000; i++ )
> {
> jobs[i] = job_server.submit (jd[i]); // SUBMIT
> }
>
> for ( i = 0; i < 100.000; i++ )
> {
> jobs[i].wait ();
> }
>
> As for each submission, you are very likely to have at least
> one remote operation (they are independend), that will take
> a lot of time.
>
> Compare that to:
>
> task_container tc;
>
> for ( i = 0; i < 100.000; i++ )
> {
> tc.add (job_server.create (jd[i])); // CREATE
> }
>
> tc.run ();
> tc.wait ();
>
> It is rather straigh forward to optimize the task container
> for bulk job submission, or bulk operations in general (we
> hope).
>
>
> What I am not sure is, what would that look like in native
> java? Are there similar mechanisms?
>
>
> Looking forward to your comments,
>
> Andre.
>
>
> > Graeme
> >
> >
> > Quoting Andre Merzky <andre at merzky.net>:
> >
> > > Hi,
> > >
> > > I just had a discussion with Thilo about the topic, as he
> > > and me obviously talked somewhat orthogonal to each other...
> > >
> > > Well, now we have the same opinion, kind of, and I have
> > > barely any bruises... Anyway, I want to summarize our point
> > > here, as I probably was not really clear in my initial post.
> > >
> > > Sorry if re-iteration of the topic bores you...
> > >
> > > So, we have tasks, which represent async operations, with a
> > > couple of states attached, and the ability to call run(),
> > > wait() and cancel() on these. And we can collect them in
> > > containers, and wait() on many of these tasks conveniently.
> > >
> > > And then we have jobs, which represent remote executables,
> > > with a couple of states attached, and the ability to call
> > > run (== create them), wait() and cancel(). And some more
> > > methods. And we can't collect them in containers right now,
> > > but would like to.
> > >
> > > You see the similarities, right? Its even more obvious in
> > > code:
> > >
> > > Tasks:
> > > --------------------------------------------
> > > task_container tc;
> > > task t = file.copy <saga::task> (...);
> > > t.run ( );
> > > t.wait (1.0);
> > >
> > > tc.add (t);
> > > tc.wait ( );
> > > --------------------------------------------
> > >
> > > Jobs:
> > > --------------------------------------------
> > > job_container jc;
> > > job j = job_server.submit (job_descr);
> > > j.wait (1.0);
> > >
> > > jc.add (j);
> > > jc.wait ( );
> > > --------------------------------------------
> > >
> > > slightly changed:
> > > --------------------------------------------
> > > job_container jc;
> > > ! job j = job_server.create (job_descr);
> > > + j.run ( );
> > > j.wait (1.0);
> > >
> > > jc.add (j);
> > > jc.wait ( );
> > > --------------------------------------------
> > >
> > > The similarities are obvious I think. Now, if job would
> > > IMPLEMENT the task interface (or inherit from task), we
> > > would unify both classes, and hence:
> > >
> > > - simplify jobs (leave only those methods which are
> > > specific to jobs, like migrate, signal, ...
> > >
> > > - allow to out jobs into task containers, efficiently
> > > handling large amounts of jobs and other tasks
> > >
> > > - have the API and used paradigms more uniform.
> > >
> > > Also, if later tasks get suspendable, as Gregor rightly
> > > suggested, we can move more methods to tasks, w/o breaking
> > > the paradigms.
> > >
> > > In terms of state, following mappings would be appropriate:
> > >
> > > job::Pending -> task::Pending
> > > job::Done -> task::Done
> > > job::Failed -> task::Failed
> > > job::??? -> task::Cancelled
> > >
> > > job::Queued,Running,Pre/Poststaging,... -> task::Running
> > >
> > > So, no adjustements to the statet models are needed AFAICS,
> > > apart from Cancelled
> > > (Does it make sense on jobs?
> > > Should job::stop be job::cancel?
> > > Should tasks::cancel be task::job?)
> > >
> > > Hope that clearifies things. I think Gregor was on target
> > > with his remarks, and Hartmut signalled consent as well.
> > > And I think I convinced Thilo (Andre rubs his bruises).
> > >
> > > Unless there is any opposition, I'll go ahead and document
> > > that in the strawman then, ok?
> > >
> > > Cheers, Andre.
> > >
> > >
> > > Quoting [Thilo Kielmann] (Feb 04 2006):
> > > > Date: Sat, 4 Feb 2006 22:29:41 +0100
> > > > From: Thilo Kielmann <kielmann at cs.vu.nl>
> > > > To: saga-rg at ggf.org
> > > > Subject: Re: [saga-rg] tasks and jobs...
> > > >
> > > > > Wouldn't it be useful to have jobs implementing the task
> > > > > interface?
> > > >
> > > > Certainly, no.
> > > > Jobs and Tasks are two different things, and they are this on
> purpose.
> > > >
> > > > However, Tasks always have been the mechanism for asynchronous
> > > operation,
> > > > which is kind-of obsoleted by having asynchronous ops directly.
> > > >
> > > > If you want to work on the "S" of SAGA: why not unify both Tasks
> and
> > > Jobs
> > > > into a better "Job" notion, and do local asynchronous operations
> via
> > > > async, local calls?
> > > >
> > > > Thilo
> > > --
> > > "So much time, so little to do..." -- Garfield
> > >
> > >
> > >
> > >
> >
> >
>
>
>
> --
> "So much time, so little to do..." -- Garfield
>
>
>
>
More information about the saga-rg
mailing list