[ogsa-dmi-wg] Fwd: dmi question

[Shahbaz Memon]

here is David's email with my response.

---------- Forwarded message ----------
From: Shahbaz Memon <m.memon at fz-juelich.de>
Date: Thu, Oct 22, 2009 at 2:21 PM
Subject: Re: dmi question
To: david.meredith at stfc.ac.uk


Hi Dave,

My apologies for delay.

> Quick question: If I remember correctly, at the last DMI phone meeting you mentioned that you had been working on an approach to define data transfers using a DMI-like approach that catered for multiple data sources and sinks? If true, do you have any more details or schema examples in preparation for next weeks DMI phone meeting ?

I have gone through the schemas on google code. For initiating many
transfers, GetDataTransferInstance would yield some requirements for
managing them in one go. From schema point of view it would be more
optimal to reuse the elements of existing DMI spec.

With reference to last telecon, I had some thoughts to avoid major
changes in DMI spec. I am not sure if it is trivial or within the DMI
scope. But still my comments are as follows;

- to use same DataEPR element from DMI as compare to mjsdl:Source or
mjsdl:Targets: We can reuse them as they are already in the spec,
except the absence of transfer requirements. I am not sure whether its
feasible to have individual or group transferrequirements. It seems
that you want to apply transferreqs for  set of transfers, may be we
can also put an optional transferreqs for individual transfer ones (in
order to provide flexibility).

- jobResourcekey is a single identifier returned from the
getdatatransfer operation, but it is pointing to multiple jobs.
I.M.H.O If we return ref of each transfer, so that each one can be
traceable and monitored, rather than a composite id.  My proposition
is that we can return all the transfer-ids and a composite-id (group
or transaction-id) together to distinguish group of transfers.
Generally, this composite-id can also be queried to fetch the
bytes-transferred , upon which server returns total bytes-transferred
of their children.

- Now question comes after the client has already sent the request and
intend to monitor transfers (individual / composite). In PGI we did
the same work, we came to the conclusion of removing Job interface and
re-using the factory interface for managing multiple jobs. Its
completely fine here too, because of the overhead of managing multiple
web service instances generated within a single request. In PGI, we
have introduced the "changestatus" method which takes set of job eprs
and transition them to certain state e.g (Pending->Ready) or
(Ready->Start) phase. Now the question comes in DMI, if we are taking
group transfers as one atomic job, then to manage its state is not a
issue. The complexity lies when the group contains multiple individual
transfers, and to track or upgrade each of the status will be tricky.

Normally, the users of Grid job submission interface always expects
that job returns a single status, though there are multiple
stage-ins/outs (with parallel different state transitions) are
attached. In dataminx, the current schema depicts the use of jsdls.
Specifically, in our case, DMI is a data transfer interface, i.e. more
orientation towards the data features. And, end users are not taking
transfers as grid jobs, though they consider them as data jobs where
each transfer (with certain properties) can be steered or monitored.
In grid/computing jobs, users are not so much concerned with the
attached source->target transfers, rather they are interested in job
per se.

May be we can discuss them more in our upcoming telecon. We can also
see what more can be done to make DMI a simple and clean interface and
progress the document uploaded by Mario.

Best Regards,

Shahbaz Memon




On Thu, Oct 22, 2009 at 1:21 PM,  <david.meredith at stfc.ac.uk> wrote:
> Hi Shabaz,
>
> Quick question: If I remember correctly, at the last DMI phone meeting you mentioned that you had been working on an approach to define data transfers using a DMI-like approach that catered for multiple data sources and sinks? If true, do you have any more details or schema examples in preparation for next weeks DMI phone meeting ?
>
> To date, I haven't really had much time to think about this, but this should change soon (I am currently in the process of handing off my current projects so that I can concentrate on DataMINX/DMI/transfers - scheduled to start Dec 1st).
>
> Kind regards,
> Dave
>
>
> ------------------------
> David Meredith
> STFC eScience Centre
> Daresbury Laboratory
> Warrington Cheshire
> WA4 4AD
> Tel: 01925 603762(Direct Line)
> email: david.meredith at stfc.ac.uk
>
> --
> Scanned by iCritical.
>
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
>